The Terrifying Problem Of AI Control - Stuart Russell | Modern Wisdom Podcast 364

Stuart Russell is a Professor of Computer Science at the University of California and an author. Programming machines to do what we want them to is a challenge. The consequences of getting this wrong become very grave if that machine is superintelligent with essentially limitless resources and no regard for humanity's wellbeing. Stuart literally wrote the textbook on Artificial Intelligence which is now used in hundreds of countries, so hopefully he's got an answer to perhaps the most important question of this century. Expect to learn how artificial intelligence systems have already manipulated your preferences to make you more predictable, why social media companies genuinely don't know what their own algorithms are doing, why our reliance on machines can be a weakness, Stuart's better solution for giving machines goals, what the future of artificial intelligence holds and much more... Sponsors: Get 20% discount on the highest quality CBD Products from Pure Sport at https://puresportcbd.com/modernwisdom (use code: MW20) Get perfect teeth 70% cheaper than other invisible aligners from DW Aligners at http://dwaligners.co.uk/modernwisdom Extra Stuff: Buy Human Compatible - https://amzn.to/3jh2lX5 Get my free Reading List of 100 books to read before you die → https://chriswillx.com/books/ To support me on Patreon (thank you): https://www.patreon.com/modernwisdom #artificialintelligence #controlproblem #computerscience - 00:00 Intro 00:33 King Midas & AI 06:07 Super-intelligent AI 11:48 Language Challenges 21:42 How AI Could Go Wrong 46:17 Social Media Algorithms 1:03:14 Becoming Enfeebled by Machines 1:20:44 Maintaining Control of AI Growth 1:42:23 Impacts of Stuart’s Work 1:48:01 Where to Find Stuart - Listen to all episodes online. Search "Modern Wisdom" on any Podcast App or click here: Apple Podcasts: https://apple.co/2MNqIgw Spotify: https://spoti.fi/2LSimPn Stitcher: https://www.stitcher.com/podcast/modern-wisdom - Get in touch in the comments below or head to... Instagram: https://www.instagram.com/chriswillx Twitter: https://www.twitter.com/chriswillx Email: https://chriswillx.com/contact/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Well, you have to build machines that know that they don't know what the objective is and act accordingly. To the extent that the machine does know the objective, it can take actions as long as those actions don't mess with parts of the world that the algorithm isn't sure about. You'd need the machine to actually ask permission and it would have an incentive to do that. So it knows that it doesn't know what the objective is, but it knows that its mission is to further human objectives.

• 0:29Chris Williamson?

  Stuart Russell, welcome to the show.

• 0:31Speaker 00

  Thank you. Nice to be here.

• 0:33Chris Williamson?

  What do King Midas and artificial intelligence have in common?

• 0:39Speaker 00

  Good question. So King Midas is famous in two ways, right? So he had the golden touch, so people think of him as kind of a lodestone for getting rich.

• 0:54Speaker 00

  But the moral of the story with King Midas is he said, I want everything I touch to turn to gold. And he got exactly what he wanted. So the gods granted his wish and then he finds out that he can't eat because his food turns to gold. And he can't drink because his wine turns to gold and then his family turns to gold.

• 1:13Speaker 00

  So he dies in misery and starvation. And this tale is basically a description of what happens or what might happen with super intelligent AI, where the super intelligent AI plays the role of the gods. And we are King Midas and we tell the AI, this is what we want and we make a mistake. Right? And then the AI is pursuing this objective.

• 1:44Speaker 00

  And it turns out to be the wrong one. And then we have created a conflict. We've basically created a chess match between us and the machines where the machines are pursuing some objective that turns out to be in conflict with what we really want.

• 2:02Speaker 00

  And that's basically the story of how things go south with super intelligent AI. And if you look at what Alan Turing said, 1951, he was on the radio.

• 2:17Speaker 00

  BBC Radio 3, the third programme. And he said, basically, we should have to expect the machines to take control.

• 2:30Speaker 00

  End of story. And I think this is what he had in mind, that that they would be pursuing objectives and we would have no way to stop them or interfere with them because they are more capable than us, so they control the world.

• 2:45Chris Williamson?

  That's the challenge, the fact that it's not just the objective is misaligned, but it's that the power deploying that misalignment is so vast that there's no stopping it once it's set away.

• 2:57Speaker 00

  Yeah, and you know, if you're if you're a gorilla or a chimpanzee or whatever, you know, you thought, you know, your ancestors thought that they, you know, they were the pinnacle of evolution and then they accidentally made humans and then they lost control. They have no, no control over their own future at all because we're here and we're smarter than they are and end of subject.

• 3:21Chris Williamson?

  Yeah, it's rare that the person that's supposed to be or the the agent that's supposed to be in charge is actually less capable or less powerful or less intelligent than the agent that they're commanding to do their bidding.

• 3:36Speaker 00

  Yes, yeah, we don't have any good models for how this relationship would work. So even if we do solve the control problem, there are various issues that we'll still have to face. So for example, how do we retain anything resembling the kind of intellectual vigor of civilization when our own mental efforts are just puny compared to to those of the machines that, you know, we're supposed to control. So, you know, and in some science

• 4:17Speaker 00

  fiction books, for example, Ian Banks's culture novels, which I highly recommend to your listeners.

• 4:28Speaker 00

  He struggles with this because, you know, they've got super powerful AI and everything is hunky dory. Um, the AI systems always do stuff that's beneficial for humans. But in in a way, they end up treating humans like children. And there's always sort of this delicate balance which parents have, right? Um, when do I stop doing up my kids' shoelaces? And make them do their own shoelaces, right? And and it's, except that with parents and children, the children are not supposed to be the ones who are in control of the parents. Sometimes they are, but not supposed to be.

• 5:13Speaker 00

  Um, and, you know, we just don't have a model for that where the children are commanding the parents, but the parents treating the children like children and saying, okay, well, I think it's time for Johnny to, you know, learn to do his own shoelaces. So I'm going to hold off on helping Johnny today. Um, you know, I just, I just don't exactly know how it's going to work. Um, and how humans are going to continue to have the incentive to slog through twenty years of education and so on. Um, to to learn things that the machines can already do much better.

• 5:55Chris Williamson?

  That's thankfully not a problem that we need to deal with just yet. I suppose the fact that we don't have an imminent, well, I suppose we don't know if it's going to be a hard take off. So it might be imminent, it might be tomorrow. Uh, but everything suggests that it's not. Have you got any conception around how long it will be before we do face a super intelligent AGI?

• 6:16Speaker 00

  Uh, well, usually I say I will not answer that question.

• 6:24Speaker 00

  And I was at a, I was at a world economic forum meeting, which was officially off the record under Chatham House rules. And I was, somebody asked me that question. So I said, well, you know, off the record.

• 6:38Chris Williamson?

  There's a number. What is?

• 6:40Speaker 00

  I said, I said, off the record, you know, within the lifetime of my children.

• 6:47Chris Williamson?

  Okay.

• 6:47Speaker 00

  And, you know, that's a flexible number because medical advances might make their lives very long. And then 20 minutes later, it's on the Daily Telegraph front page.

• 6:59Chris Williamson?

  When was this?

• 7:00Speaker 00

  Uh, probably 2015, I think. Uh, January 2015.

• 7:10Speaker 00

  Professor predicts sociopathic robots will take over the world within a generation. Uh, that was what the headline said. So, so I, I'm, you know, even though I tried to be cautious, I still got screwed. So I think, um, so I'm actually more conservative. I, I don't think it's going to happen very soon. I don't believe that just scaling up the methods we have is going to create superintelligence. You know, simply put, you make the machines faster, you get the wrong answer more quickly.

• 7:45Chris Williamson?

  What's the bottleneck that we're facing at the moment then? Is it hardware? Is it algorithms?

• 7:52Speaker 00

  It's definitely not hardware. I think we probably have more than enough hardware to create superintelligent AI already.

• 8:02Speaker 00

  I think it's, well, algorithms, but it's basic conceptual gaps in how we're approaching the problem.

• 8:13Speaker 00

  You know, our, our current deep learning systems, in a real sense, don't know anything.

• 8:23Speaker 00

  And so, uh, it's very hard for them to accumulate knowledge over time.

• 8:31Chris Williamson?

  What does that mean? That they don't know anything?

• 8:34Speaker 00

  So they can learn, they can learn a sort of an input-output function.

• 8:43Speaker 00

  And in the process of doing that, they can acquire internal representations that facilitate the representation of that function and so on. But, you know, they don't, they don't learn, let's say, Newton's laws of physics

• 9:05Speaker 00

  and then become able to apply those laws to other problems, right? They have to sort of retrain from scratch, or almost from scratch, on new problems. And if you think about the way science works, right, which is the best example of human accumulation of knowledge, right? We know that it wasn't simply the accumulation of raw sensory data and then training a giant network on vast quantities of raw sensory data, because all the people who had that sensory data are dead. Right? So whatever they learned, whatever they accumulated, had to be expressed in the form of knowledge, right?

• 9:48Speaker 00

  Which subsequent generations could then take on board and use to do the next phase.

• 9:55Speaker 00

  And so a cumulative learning approach based on the acquisition of knowledge, which can be used for multiple purposes, we don't really know how to do that, at least in the deep learning framework. In the classical AI framework, where we had explicit representations of knowledge using logic or probability theory, that could be done. And I still think it can be done in those paradigms. I just think we need to put a lot more effort and maybe to be a bit more flexible. I think one thing we've learned from, from deep learning is that there is advantage in being, as it were, sloppy.

• 10:40Speaker 00

  So that we...

• 10:43Speaker 00

  We don't have to think, okay, how can I learn Newton's laws? Well, I'm gonna put F and M and A together in some order and I'll eventually find the right law, right? That assumes that you have F and M and A already as precisely defined concepts. But in the learning process, you can be more sloppy, right? You don't have to have exactly the precise definition of mass and exactly the precise definition of force. You can have a kind of a sloppy definition. Well, it's, you know, something, there's something going on about how big and heavy the thingy is, right? And there's something going on about how hard am I pushing it? And there's something going on about how, you know, how is it moving?

• 11:26Speaker 00

  And gradually those things gel. And so you can simultaneously learn the knowledge and the concepts that go into the formulation of the knowledge. And so I think that that idea is something that we could bring back to the classical tradition.

• 11:47Speaker 00

  And improve it.

• 11:49Chris Williamson?

  What are some of the challenges around language?

• 11:53Speaker 00

  So this is a second big area where I think

• 11:57Speaker 00

  we need breakthroughs. So the language models that we have right now, GPT-3 and so on, which everyone is very excited about.

• 12:10Speaker 00

  Their job is basically to predict the next word. And they become very, very good at predicting the next word. And then they can,

• 12:22Speaker 00

  having predicted the next word, they can then add that next word. And that's how they generate text, right? So you can just keep repeatedly applying it and it will start spitting out things that look like sentences and so on.

• 12:36Speaker 00

  But what they're really doing is

• 12:41Speaker 00

  Predicting the next word based on all the previous words that were in the text sequence, right? And it's a little bit like astronomy in the time of Ptolemy. So Ptolemaic astronomy was what happened before we had any idea that the planets were massive objects moving under the force of gravity. So we plotted out the apparent motion of the planets through the heavens. And we basically described their shape. And if you look at the shape, if you sort of plot it out over the course of a night,

• 13:23Speaker 00

  it's this, you know, mostly sort of somewhat circular looking arcs, but with wiggles and spirals and so on added in, you know, over long time scales because of the relative motion of the planets around the sun. And so Ptolemaic astronomy just consisted of describing those big, complicated, wiggly, circly, spirally shapes. And then you could gradually extrapolate them, right? So once you understood the pattern of this big, wiggly, spirally shape, you could then extend it, you know, okay, I got the shape. Now I'm just going to, you know, keep drawing it, you know, and say, okay, well, next week the planet should be here. And you'd be right.

• 14:08Speaker 00

  And so that's sort of what's going on with these text prediction algorithms, right? That they're taking all the previous words, which is by analogy to the positions of the planets, and then saying, okay, I get the shape. I'm going to predict what it's going to be in two words, three words, four words into the future. But there's no sense of why, right? Why is the word on the page? The word is not on the page because the previous thousand words were on the page. All right, that's not the real cause of why that word is there. The real cause of why that word is there is because there's a world outside this text

• 14:53Speaker 00

  and someone is trying to say something about it, right? And that's sort of what I call the physics of text.

• 15:05Chris Williamson?

  There's no knowing beyond the simple output. So this is, I guess, is this similar to a philosophical zombie in a way? That you're able to output a thing that looks like it's a simulacrum of intelligence within a sort of narrow band. But there's nothing going on deeper below the surface.

• 15:25Speaker 00

  Uh, that, that's one way. I'm, I'm not, I'm not here talking about. Is there real conscious understanding?

• 15:32Chris Williamson?

  Yeah, we haven't got there yet.

• 15:34Speaker 00

  But just, does the, does this causal model of why the text is there? Do, does that approximate reality in any way? No. Right? The reality of text is that people are trying to say things. Um, and they're trying to say things about a world that they live in and are acting in. So, uh, you know, the simplest model would be they're just trying to say true things. But actually they're trying to get something done in that world and, and part of getting something done is saying true things. And sometimes it's saying false things and sometimes it's asking questions.

• 16:15Speaker 00

  Um, but this, you know, you can see this connection, you know, why the real world matters.

• 16:23Speaker 00

  Because statistically,

• 16:26Speaker 00

  the fact that there's a real world and we're all talking about the same real world means that every document in the world is correlated with every other statistically. So if one document says J K Rowling wrote Harry Potter, right? And another document, you know, written in Russian, you know, a year later says, you know, the author of Harry Potter is, you know, what do you expect the next word to be? Well, you expect it to be J K Rowling because they're talking about the world and. Even if this is a new way of saying it in a new language,

• 17:07Speaker 00

  it's correlated through the this common hidden variable, if you like, which is the real world. And none of that is

• 17:18Speaker 00

  is there in current deep learning models of language. So I think they're fundamentally flawed.

• 17:28Speaker 00

  And this is one of the reasons why they take trillions of words of text. I mean, they read

• 17:36Speaker 00

  about as much as all human beings in history have read. Right? Uh, you know, and and so and and they're still, you know, they still make stupid mistakes. They still kind of lose the plot, right? One of the things you see is that because they have a you know, they're predicting the next word based on a relatively limited memory of the previous text, they kind of lose the plot. So as they go on, they'll start either repeating themselves or uh, contradicting them what they said earlier on in in in the text and so on. So, you know, having said that, they do exhibit quite impressive kind of short term recall and question answering.

• 18:21Speaker 00

  And and a certain amount of generalization is going on. Right? Because you can you can see that because you can you can ask them questions or you can tell them things using a name or a place that they've never ever seen before. And then you can ask them questions about that name or that place. And they'll answer it correctly. So there's there's some generalization going on, right? They they've learned a general pattern and then they're able to instantiate the general pattern with particular people or places or whatever it might be. And so,

• 18:57Speaker 00

  you know, that that's a sign that learning has happened. But generally, we don't understand what's going on beyond that. And so we don't know when they're just spouting gibberish. Right? You think it's answering your question and actually it's just spouting complete gibberish. You don't know.

• 19:14Chris Williamson?

  I suppose the challenge here is that the main way that we communicate is through language. So. If you're not a computer programmer and you wanted to have a conversation with a super intelligent AGI home assistant, you would need to tell it what you mean. It would not only need to be able to understand the words that came out of your mouth, but our language, our use of language is imprecise also. So it also needs to be able to work out what you meant to mean, not just what you said. Then it needs to interpret it, then it needs to be able to convert that into something that it can do within itself, and then it needs to enact that. So yeah, I mean.

• 19:51Speaker 00

  Yeah, so I mean, we, we built systems that could do that even in the 60s and 70s. You know, and, and they sort of work the way you would expect. They, they understood the structure of the language, they, what's called parsing the sentences, so figuring out what's the subject, what's the verb, what's the object. And then converting that into an internal logical representation. And then doing the reasoning necessary to answer the question or to add the fact to the system's internal knowledge base. And then generating answers to questions and converting the answers back into language. You know, so that, that process, you know, we've, we've known how to do, it's just been

• 20:36Speaker 00

  very difficult to make it robust because the variety of ways of saying things is enormous. You know, we speak in ways that aren't grammatical, but are still perfectly comprehensible. You know, we do things like lie, right? The last thing you want to do is, is for the system to believe everything everyone says because then, then it's very easy to manipulate. So it has to understand that what we're, you know, what's coming out of our mouths is, as, as Wittenstein put it, is a move in a game. Right? It's not gospel truth, it's an action that we are taking. And the action might be to try to fool you or to try to make you do something that you wouldn't otherwise do or whatever.

• 21:19Speaker 00

  So that, you know, that level is completely not there, right? You can, GPT-3 takes all text as gospel, gospel truth or whatever, right? It doesn't distinguish between fiction and, and fact, between propaganda and truth and so on. It's all just text.

• 21:41Chris Williamson?

  What are some of the big ways that we could get artificial intelligence wrong?

• 21:48Speaker 00

  So I think the

• 21:51Speaker 00

  the current approach to AI, which has been there really since the beginning. And in the book, Human Compatible, I call it the standard model, which is a word that people use in physics to refer to, you know, all laws of physics that we pretty much agree are right.

• 22:08Speaker 00

  So in AI, the standard model has been to build machines that

• 22:15Speaker 00

  that behave rationally. And this notion of rational comes from philosophy in economics, that you you take actions that you can be expected to achieve your objectives. And that, you know, that goes back to Aristotle and other places.

• 22:34Speaker 00

  So we took that model and we created AI systems that fit that model. Now, with people, we have our own objectives. So we can be rational with respect to our objectives. Of course, machine doesn't have objectives intrinsically. So we put the objective in and it seems like a perfectly reasonable plan, right? I get in my automated taxi, I say, take me to the airport, right? That becomes the taxi's objective and then it takes me to the airport. It figures out a plan to do it and does it, right? And pretty much all AI systems have been built on this basis, that

• 23:15Speaker 00

  one of the inputs that's required to the algorithm is the objective. You know, if it's a game playing program, you know, the objective is checkmate or whatever the whatever it might be. If it's a

• 23:30Speaker 00

  a route planning algorithm, then it's the destination. If it's a reinforcement learning algorithm, then it's the reward and punishment definition.

• 23:40Speaker 00

  And so on. So and this is, this is a pretty common paradigm, not just AI, but, you know, the control theory systems that fly our airplanes, right, they they minimize a cost function. So the engineer specifies a cost function which penalizes deviations from the desired trajectory, you know, and then the algorithms, you know, will optimize a given cost function.

• 24:08Speaker 00

  And, uh, okay, so what's the problem with that? The problem is, as I mentioned earlier, you brought up King Midas. We don't know how to specify the objective completely incorrectly. And so for artificially defined problems like chess, it kind of, you know, chess comes with a definition of checkmate. So it's, you know, it's sort of fooling us into thinking that this is an easy problem to specify the objective. But take the, you know, the automated taxi, the self-driving car. Right? Is the destination the objective? Well, better not be because, you know, then it might drive you there at 200 miles an hour.

• 24:49Speaker 00

  And, you know, you, you, you come home with 50 speeding tickets and, uh, if you weren't dead, you know. So obviously, safety is also part of the objective, right? Okay, well, fine, safety, but then how do you trade off safety and getting to the destination? Right? Um, if you prioritize safety above everything else, then you're never going to leave the garage. Because just going out onto the road incurs some risk. Well, okay, so now we have to put in some trade off between safety and making progress. And then you've got, you know, obeying the laws, then you've got not pissing off all the other drivers on the road. Uh, you know, then you've got not shaking up the passenger, right? By starting and stopping too much.

• 25:33Speaker 00

  Uh, and it go, the list goes on and on and on. And the self-driving car companies are now facing this problem. And they have whole committees, you know, and they have meetings all the time trying to figure out. Okay, we get the, you know, the, the latest data from our cars in the field and all the mistakes they made. And, you know, tweak all the objectives to get them to behave better and so on. So even for that problem, it's really hard. And if you had something like, you know, curing cancer or, uh, you know, fixing the carbon dioxide levels, right? You can see how things go wrong, right? You, you want to cure cancer really fast? Sounds good. Okay, great, well, then we'll, we'll induce tumors in the entire human population.

• 26:18Speaker 00

  Um, so that we can run millions of experiments in parallel on different ways of curing them. Um, you just don't, you don't want to be in that situation, right?

• 26:33Speaker 00

  And so the the answer, it seems to me, is

• 26:42Speaker 00

  we have to get rid of the standard model.

• 26:46Speaker 00

  Right. And so here am I, you know, I wrote a textbook based on the standard model. In fact, it's sort of in many ways, it it made the standard model, the standard model, and it, you know, and here am I saying, actually, sorry, chaps, we got it all wrong. And we're gonna have to, I'm gonna have to rebuild the whole field.

• 27:11Speaker 00

  And, you know, so you've got to get rid of this assumption that the human is going to supply the complete fixed objective.

• 27:24Chris Williamson?

  It's too complex, or it'd be too arduous, I'm going to guess, to be able to program in plugging the little holes in the bottom of the boat for each one of the ways that the machine could slightly go off course. So you've got safety. Okay, we write the safety algorithm. Okay, we've got speed, we write the speed algorithm. I'm going to guess that the goal would be to get a more robust, sort of scalable, general solution to this that would be able to find a problem, a solution to all potential problems, that would be able to optimize the outcome across all potential challenges.

• 28:02Speaker 00

  Yeah, sort of. I mean, it's if you, so, I mean, basically, what you have to build machines that know that they don't know what the objective is

• 28:13Speaker 00

  and act accordingly.

• 28:16Speaker 00

  So what does act accordingly mean? Uh, well, to the extent that the machine does know the objective, it can take actions as long as those actions don't mess with parts of the world that the algorithm isn't sure about your preferences. Right. You know, so if if you if you have a machine that's going to try to restore carbon dioxide levels in the atmosphere to their pre-industrial concentrations, right, so that's a really good objective. Right. Well, you know, it wouldn't be a good objective if the solution was, you know, get rid of half the oxygen.

• 28:57Speaker 00

  Alright, because then we would all slowly asphyxiate. So that would be really bad. Don't do that. Okay, what if it means turning the oceans into sulfuric acid?

• 29:10Speaker 00

  Yep, okay, no, don't do that. Alright. So you'd need the machine to actually ask permission. Right? And it would have an incentive to do that. So it knows that it doesn't know what the objective is, but it knows that its mission is to further human objectives, whatever they are. So it has an incentive to ask, to ask permission, to defer if we say, "Stop, you know, that isn't what I meant." Right? It has an incentive to obey, because it wants to avoid doing whatever it is that violates our objectives. And so you get these new kinds of behaviors. A system that believes that it has the objective,

• 29:56Speaker 00

  right, becomes a kind of religious fanatic. Right? It pays no attention when we say, "You know, stop, you're destroying the world." It's, "I'm sorry, I've got the objective. You know, whatever you're saying is wrong, because I got the objective and I'm pursuing it." Right? We don't, you know, we don't want machines like that. So in this new model, it seems much more difficult, and in a way, it is much more difficult, to satisfy an objective that you don't know. Right? But it produces these behaviors, you know, asking questions, asking permission, deferring, you know, and in the extreme case, allowing yourself to be switched off. Right? If the machine might do something really catastrophic,

• 30:43Speaker 00

  then we would want to switch it off. Now, a machine that believes that it has the correct objective is going to prevent you from switching it off, because... That would be failing, right? It wouldn't achieve its objective if it gets switched off. The machine that knows that it doesn't know what the objective is, actually wants you to switch it off. Right? Because it doesn't want to do anything sufficiently bad that you'd want to switch it off. So it wants, it has a positive incentive to allow itself to be switched off. And so this new model, um, I don't think it's perfect, but it's, it's a huge step

• 31:25Speaker 00

  beyond the way we've been thinking about AI for the last 70 years. And I think it's the key, you know, it's the core of a solution uh, that will allow us, you know, not to end up like King Midas.

• 31:40Chris Williamson?

  What's not perfect about it?

• 31:43Speaker 00

  Um.

• 31:46Speaker 00

  I think the, the biggest problem that I'm wrestling with right now is the fact that human objectives are actually, shall we say, plastic or malleable.

• 32:03Speaker 00

  Right? Um, and you can tell that because, you know, we don't have them when we're born, right? When we're born, we have pretty simple objectives.

• 32:12Speaker 00

  And uh, you know, so it's something about our culture, maturation, etc., that creates adults who have

• 32:25Speaker 00

  to some extent, fairly definite preferences about, about the future. So the way I think about it is not asking you to write them down. Right? Because that's really, that's really hopeless. Um. But if you, if I could show you sort of two movies of the future, right? Future A, future B. Um. And you could sort of watch those and then reset yourself and watch the other one and reset yourself. And then say, which one, which one do I prefer? Right? I think that's a a reasonable back of the envelope description of, of what we're talking about, right? The, you know, everything you care about in the future.

• 33:05Speaker 00

  If the movie, if you couldn't quite tell whether you liked A or B because there's some detail missing, then you can get some more detail on, on those parts and, um, you know, a, a, a future where. The oceans are turned into sulfuric acid and we all die of oxygen deprivation. Uh, it's pretty clear that's not the future we prefer. Um, so, uh, the, the issue with plasticity and malleability is that, um, although I might say I like future A today, right, tomorrow I'm a new person and I might like future B instead,

• 33:49Speaker 00

  but it's kind of too late because now you've stuck me into future A. Um, and so the first problem there is, well, who do you believe, right? Do you, you know, you're making a decision now.

• 34:04Speaker 00

  Should I respect the preferences of the person now? Or should I anticipate how you're going to change in future and respect your future self? And, and I don't, you know, philosophers haven't really given us a good answer to that question. So that's, that's one part, right? Sort of deep philosophical issue. Um, the, the more problematic part is that if our preferences can be changed, then the machine could satisfy our preferences by changing them rather than by satisfying them.

• 34:44Speaker 00

  So it could, it could find out ways to change our preferences so that we'd be happy with whatever it was going to do, right? Rather than it figuring out how to make us happy with the preferences that we have. Um, you know, and you could say, well, yeah, politicians do that and advertisers do that, right? They, uh, but we don't think of that as a good thing. And it could be, you know, with, with machines, it could be a much more extreme version of that. And so, um, So I think of what's in the book as kind of version zero of the theory and version one would have to deal with this with this aspect. You know, there are there are other difficult questions to answer. Like,

• 35:28Speaker 00

  you know, obviously, machines are making decisions not on behalf of one person, but on behalf of everybody.

• 35:36Speaker 00

  And how exactly do you trade off the preferences of individuals, you know, who all have different futures that they prefer? And that's not a new question.

• 35:48Speaker 00

  You know, it's thousands of years old. And, you know, I I think that that's I I feel that's a manageable question. You know, and crudely speaking, the answer is you you add them up.

• 36:03Speaker 00

  Right?

• 36:05Speaker 00

  And that's the what's called the utilitarian approach. And

• 36:10Speaker 00

  we associate names like Bentham and Mill

• 36:14Speaker 00

  with that idea. And more recently, Harsanyi, who's was a Berkeley economics professor, who won the Nobel Prize and put a lot of utilitarianism onto a an axiomatic footing.

• 36:31Speaker 00

  You know, so he so what what it's interesting, actually, to to understand what that means, because a lot of people have a you know, sort of emotional dislike of utilitarianism, partly because the word utilitarian, you know, that sort of refers to sort of, you know, gray plastic furniture and, you know, and council flats.

• 36:52Chris Williamson?

  Branding problem, yeah.

• 36:53Speaker 00

  Yeah, it's a branding problem, exactly. It got sort of mixed up with with the with the wrong word. And you know, people complain about it, you know, not being sufficiently egalitarian.

• 37:10Speaker 00

  And not, you know, people assume that it refers to money, like, you know, maximizing the amount of money in the wealth in them. Nothing to do with that.

• 37:21Speaker 00

  But the kinds of axioms that Harsanyi proposed, when you actually think about them, they they probably, you know, most people would accept them as quite reasonable. So, for example,

• 37:37Speaker 00

  Suppose you've got two futures, you know, future A and future B. And future B is exactly the same as future A, except one person is happier in future B than they were in future A. Everyone else is exactly as happy as they were before. He said, "Well, it seems reasonable that you'd say future B is better than future A." Right? And so he has a couple of axioms like that. And from those axioms, you can derive the utilitarian solution, which is basically add 'em up, right? Find whichever policy maximizes the sum total of human happiness.

• 38:21Speaker 00

  And, you know, so I think there are various difficulties involved, right? So when you say the sum total of human happiness, are you including all the people who haven't yet been born? And if so, what about actions that affect who gets born? Right? And that sounds like, you know, that sounds pretty weird. But actually, you know, the Chinese government with their one-child policy, right? They wiped out 500 million people with that policy. Right? So they, quote, "killed" more people than anyone in history has ever killed.

• 39:06Speaker 00

  You know, way worse than the Holocaust, way worse than Stalin.

• 39:12Chris Williamson?

  By preventing them from being able to have existed.

• 39:15Speaker 00

  The sense of preventing them from being existing, it was huge. Right? Was it the right? You know, was that a moral decision? Was that the correct decision? Really hard to say. Right? I mean, the reason they did it was because they were afraid that they would have mass starvation if they had too much population growth. And they had to experience what mass starvation was like. So, you know, it's arguable that it was a reasonable thing to do, but...

• 39:45Speaker 00

  It did lead to a lot of people not existing.

• 39:47Chris Williamson?

  Presumably going for

• 39:49Speaker 00

  Right, it's really hard.

• 39:51Chris Williamson?

  Presumably going for just raw utilitarianism has a ton of awful externalities as well, though. Like, the most happiness for the most people. Okay, well, there's two variables we can play about with that. We could just make tons and tons and tons of people. There you go. Okay, well, there we go. We got... Everyone's not that happy, but there's a lot of people and it's actually managed to make up for it. Or...

• 40:12Speaker 00

  Yeah, yeah, I mean, this, yeah, this is, uh... So Derek Parfit, who's a British philosopher, has a book called Reasons and Persons. And this is one of the arguments in the book, and he calls it the repugnant conclusion, which is that we should make basically infinitely many people who have, you know, a barely acceptable existence. And if you, if you watch the Avengers, you know, one of the Marvel, right? The one where Thanos is collecting the Stones of Power or whatever they call it.

• 40:52Speaker 00

  He's proposing one side of that philosophical argument, which is that he should get rid of half the people in the universe, and then the rest would be more than twice as happy.

• 41:04Chris Williamson?

  Yeah, dangerous, dangerous using Thanos as the basis for your philosophical justifications, isn't it?

• 41:09Speaker 00

  Yeah, so you have to, um...

• 41:14Speaker 00

  You have to get these things right.

• 41:16Chris Williamson?

  Before you give him the big glove.

• 41:19Speaker 00

  Before you give him the big glove. And that's, that's the same question we face with AI. But it's not as if there's a, you know, it's not as if there's a, an obviously better solution, right? So, so the alternative to utilitarianism is sometimes called the deontological or the rights-based approach, where you simply write down a bunch of rules, saying, can't do this, can't do this, can't do this, can't do this, right? Have to do that, have to do that.

• 41:48Speaker 00

  And...

• 41:50Speaker 00

  The utilitarian can can quite easily accommodate a lot of those rules, right? So if you say, you know, you can't kill people. Well, utilitarians, everyone say, well, of course you can't kill people because, you know, the person who gets killed, that's not what they want the future to be like. And so, uh, so the utilitarian solution would avoid murder. And and Mill goes on for pages and pages and pages about, well, of course, moral rules are, you know, I'm not throwing out moral rules.

• 42:31Speaker 00

  I'm just saying that if you actually go back to first principles, they follow from utilitarianism. Uh, but we don't have the time and energy to go back to first principles all the time. So we write down a bunch of moral rules. And I think there are, there are more complicated arguments about uh, avoiding strategic complications when you, when we're making decisions in society. It's much easier if there are moral rules rather than thinking all the time, okay, well, if I do this, and then they might do that, and I might do this, and they might do that, and I might do this. Right? So sort of playing this complicated chess game with, with, with eight billion people all the time.

• 43:12Speaker 00

  Um, it's just easier if there are rules that everyone knows exist and will be respected. Um, but the interesting place is, is what happens? Um, you know, in the, in the corner cases, right? Do, do we say, no, the rule, no matter what the utilitarian calculation is, the rule is absolute? And I think the answer is no. You can start out with some easy rules, right? This, you know, the rule says you can't eat, uh, you'll have to eat fish on Friday. Right. Well, you know, is that an absolute rule?

• 43:58Speaker 00

  Well, I don't know. I mean, i- if there were no fish and my child is starving and that, you know, the only, only thing for them to eat is some, some meat, I'll give them some meat. Right? Um, so we clearly see that,

• 44:16Speaker 00

  you know, rules are an approximation and when, when we're in difficult corner cases, we fall back to first principles.

• 44:26Speaker 00

  Right? And, and so I don't see that there's, there's the degree of conflict um, between utilitarian and deontological approaches that, that some people see. And what, you know, i- it's one of the typical arguments, you know, might in utilitarianism, might, against utilitarianism, sorry, would, would say something like, well, you know, with your organs, I could save five people's lives. Right? You know, your kidneys, your lungs, maybe your heart. So I'm entitled to just go around ripping the organs out of people to, to save other people's lives. Right? Well, of course, that's not what utilitarianism would suggest, because if that were

• 45:10Speaker 00

  how we behave, life would be intolerable for everybody on earth. Right? We'd be constantly looking around over our shoulders and grabbing our kidneys and, you know. So it's just, um, you know, so, so the, the utilitarian uh, solution, sometimes called the rule utilitarianism, is that it's useful to have these rules about behavior, not just to consider the individual act, but to consider what if that act were allowed? Right? What if there were a rule that you could always do that act? Then it would be terrible. So, um, so I think you can reconcile a lot of these, a lot of these debates. But the, the examples that we've.

• 45:54Speaker 00

  That we've already touched on, you know, the the fact that our preference changes, the fact that we have to consider people who don't yet exist or might not exist. You know, the, these are important unsolved questions, no matter what philosophical place you come from.

• 46:14Chris Williamson?

  It might sound like very far future predictions, but the user being manipulated to make their preferences easier by the machine is actually something that's already happened. Can you take us through what social media content algorithms have done?

• 46:35Speaker 00

  Sure, yeah, so the social media content algorithms, right, they decide what you read and what you watch. And they do that for literally billions of people for hours every day. Right? So in that sense, they have more control over human cognitive input than any dictator in history has ever had, more than Stalin, more than Kim Il-sung, more than Hitler. Right? They have massive power over human beings. And they're completely unregulated.

• 47:20Speaker 00

  And people are reasonably concerned about what effect they're having. And so what they do is basically they, they're set an objective because they're good standard model machine learning algorithms. And so they're set an objective, let's say maximize click through, right? The, the probability that you're going to click on the next thing. So imagine like this is YouTube. You know, you watch a video and low and behold, another video pops up. Right? And am I going to watch the next video that it sends me to watch? Or am I going to, you know, close the window?

• 47:57Speaker 00

  And so click through or, you know, engagement or various other metrics. These are the things that the algorithm is trying to optimize. And

• 48:09Speaker 00

  I suspect originally the companies thought, "Well, this is good because it's good for us. If they click on things, we make money." "And it's good for people because the algorithm will learn to send people stuff they're interested in."

• 48:23Chris Williamson?

  If they click on it, it's because they wanted to click on it?

• 48:26Speaker 00

  Yeah, right. And there's no point sending them stuff that they don't like. It's just cluttering up their inputs, so to speak.

• 48:33Speaker 00

  But, you know, I think the algorithms had other ideas. And the way that an algorithm maximizes click-through in the long run is not just by learning what you want. Right? Because you are not a fixed thing.

• 48:55Speaker 00

  And so you can get more long-run click-throughs if you change the person into someone who's more predictable. Right? Who's, for example, addicted to a certain kind of violent pornography. Right? And so YouTube can make you into that person by, you know, gradually sending you the gateway drugs and then more and more extreme content, whatever direction. So the algorithm doesn't know that you're a human being or you have a brain. Right? As far as it's concerned, you're just a string of clicks.

• 49:38Speaker 00

  Right? Content click, content click, content click. Right? And, but it wants to turn you into a string of clicks that in the long run, there's more clicks and less, less non-clicks. And so it learns to change people into more extreme, more predictable mainly, but it turns out probably more extreme versions of themselves. So if you indicate that you're interested in climate science, it might try to turn you into an eco-terrorist, you know, you know, articles full of outrage and so on.

• 50:20Speaker 00

  If you're interested in cars, it might try to turn you into someone who just watches endless reruns of Top Gear.

• 50:32Chris Williamson?

  Why is the person that's extreme more predictable?

• 50:36Speaker 00

  Well, I think that's an empirical hypothesis on my part, right? That if you're more extreme, you have a higher emotional response to content that affirms your current views of the world. And so, in politics, we call it red meat, right? The kind of content that gets the base riled up about, you know, whatever it is they're riled up about, whether it's the environment, or immigrants flooding our shores, or whatever it might be, right? You know, once you get the sense that someone might be a little bit upset about too many

• 51:20Speaker 00

  immigrants, then you send them stuff about all the bad things that immigrants do, and, you know, videos of people climbing over walls, and sneaking into beaches, and all the rest of the stuff, you know? Human propagandists have known this forever, but historically, human propagandists could only produce one message. Whereas the content algorithms can produce, in theory, one propaganda stream for each human being, specially tailored to them, and the algorithm knows how you engage with

• 52:05Speaker 00

  every single piece of content, right? Your typical, you know, Hitler's propagandist sitting in Berlin had absolutely no idea, on a moment-to-moment basis, how people were reacting to the stuff that they were broadcasting, right? They could see it in the aggregate, over longer periods of time, that certain kinds of content was effective in the aggregate, but they don't have anything like the degree of control that these algorithms have. And, you know, one of the strange things is that we actually have very little insight...

• 52:45Speaker 00

  Into what the algorithms are actually doing. So what I've described to you seems to be a logical consequence of how the algorithms operate and what they're trying to maximize. But I don't have hard empirical evidence that this is really what's happening to people. Because the platforms are pretty opaque.

• 53:07Chris Williamson?

  But they're opaque to themselves.

• 53:10Speaker 00

  They're opaque to themselves. So you know, Facebook's own oversight board doesn't have access to the algorithms and the data to see what's going on.

• 53:20Chris Williamson?

  Who does?

• 53:23Speaker 00

  I think the engineers, but their job is to maximize click through, right?

• 53:32Speaker 00

  So pretty much there isn't anyone who doesn't already have a vested interest in this, who has access to what's happening. And that, I think, is something that we're trying to fix.

• 53:48Speaker 00

  Both at the government level. So there's this new organization called the Global Partnership on AI, which is, you know, it could just be, you know, yet another do-gooder talking shop, but it actually has government representatives sitting on it. So it can make direct policy recommendations to governments. And in some sense, it has the force of governments behind it when it's talking to the Facebooks and Googles of the world. So we're in the process of seeing if we can develop agreements between governments and the platforms for a certain

• 54:33Speaker 00

  type of transparency. So it doesn't mean, you know, looking at whatever, you know, looking at what Chris is watching on YouTube.

• 54:41Chris Williamson?

  You do not want to do that. You do not want to do that at all.

• 54:44Speaker 00

  It means, you know, being able to find out, you know, how much terrorist content

• 54:54Speaker 00

  is being pumped out, where is it coming from, who is it going to?

• 54:59Chris Williamson?

  Slightly more sort of aggregated stuff like typical data scientists do.

• 55:02Speaker 00

  Yep. Yep. So and possibly being able to do some kinds of experiments, like, you know, if. If the recommendation algorithm works this way, you know, what effects do we see on users compared to an algorithm that works in a different way? So to me, that's the really interesting question.

• 55:24Speaker 00

  Is, you know, how do the recommendation algorithms work and what effect do they have on people?

• 55:32Speaker 00

  And if we find that they really are manipulating people, right, that they're, there's sort of a consistent drift,

• 55:42Speaker 00

  that a person who starts in a particular place will get driven in some direction that they might not have wanted to be driven in, then that's really a problem and we have to think about different algorithms. And so in AI, we often distinguish between reinforcement learning algorithms, which are trying to maximize a long-term sum of rewards. So in this case, the long-term rate of clicks on the content stream is what the algorithm is trying to maximize. Those kinds of algorithms, by definition, will manipulate.

• 56:22Speaker 00

  Because the, the action that they can take is to choose a particular piece of content to send you. And then the state of the world that they are trying to change is your brain. And so they, they will learn to do it, right? A supervised learning algorithm is one that's trying to get it right, right now.

• 56:49Speaker 00

  Right? So they are trying to predict whether or not you're going to click on a given piece of content.

• 56:59Speaker 00

  Right? So a supervised learning algorithm that learns a good model of what you will and won't click on could be used to decide what to send you in a way that's not based on reinforcement learning and long-term maximization. But simply, okay, given a model of what you're likely to click on, we'll send you something

• 57:21Speaker 00

  that's consistent with that model.

• 57:24Speaker 00

  Right. In that case, I think you could imagine that it would work in such a way that it wouldn't move you, it wouldn't cause you to change your preferences. But if it was done right, it could sort of leave you roughly where you are.

• 57:43Chris Williamson?

  Are you familiar with the term "audience capture"? Do you know what this means from a creator, an online creator's perspective?

• 57:52Speaker 00

  I can imagine, but not as a technical term.

• 57:54Chris Williamson?

  Yeah, well, it's not a technical term, but it's basically when you have a particular creator online who finds a message, narrative rhetoric, that resonates with the audience. And what you see is that this particular creator becomes captured and they start to feed their own audience a message that they know is going to be increasingly more well-liked. And for the most part, this actually does look like a slide toward one side of the one particular direction or the other, at least politically it does, but with anything it does too, that people inevitably sort of niche down and then they bring their audience along with it. So the fascinating thing here, I mean, first off, it's unbelievable that

• 58:36Chris Williamson?

  these algorithms that are simply there to try and maximize time on site, or click-throughs, or watch time, or whatever, that they have managed to find a way, things that we programmed managed to find a way to program us for it to be able to do its job better. I mean, that, when I read that in your book, I, it's insane. Like, that's one of the most terrifying things, and it's happening right, it's happened. Like, everybody that's listening to this has had something occur with regards to their preferences, their worldview, whatever it might be, something has slid in one way or another. You may be right, it may not be toward the extremes. I would say anecdotally, based on what I see in the world, increasing sort of levels

• 59:18Chris Williamson?

  of partisanship, no matter what it is, whether it be sports, politics, race relations, anything. People are moving toward the extremes. Why is this happening? Oh, well, you know, it's people getting into echo chambers and they're only being shown stuff like that. And also the fact that the algorithms are actually trying to make them more predictable. But on top of that as well, there's another layer, which is the creation of the content itself that comes in from the creators and they have...

• 59:50Chris Williamson?

  Their own levels of manipulation, which has occurred from their feed, then they kind of second order that into what do I want to create? What have I seen that's successful? What does my audience seem to resonate with from me? So you have layers and layers of manipulation going on here.

• 1:00:07Speaker 00

  Yeah, yeah, and I think, you know, in some ways, the creators are are being manipulated by the system. You know, I think every journalist now is thinking, okay, I have to get something that's clickbait. I have to write an article that can have a headline that is sufficiently attractive that it'll get clicked on. You know, and it is almost the point where the, you know, the headline and the article are completely divorced from each other. And and you can see this now in the comments, right? The people writing the comments at the end of the article will say, oh, I'm really, I'm really pissed off. This is just clickbait. You know, the article really doesn't say anything about the thing you said you were going to say.

• 1:00:49Speaker 00

  So on, so so this. And it was not as if this has never been going on. And obviously, you can't. You can't ban people from writing interesting articles. Or, you know, I often think about, you know, the novel and it says on the back, I couldn't put it down, right? What should we ban? Novels that, you know, that's addictive, you know, you can't have that, right? No, but I think it wasn't too bad before because the feedback loop was very slow. And there wasn't this, you know, targeting of individuals by algorithms who are, you know, so you think about the

• 1:01:33Speaker 00

  the number of learning opportunities for the algorithm, right? I mean, it's billions every day for the YouTube selection algorithm, right? So it's the.

• 1:01:45Speaker 00

  The amount, the consistency, the frequency and the customization of the learning opportunities for manipulation, so much greater, I mean, it's, you know, millions or billions of times greater and more systematic. And that, that systematic element, so it reminds me, there's this, I don't know if it's apocryphal, but there's a, um, there's a story about this psychology lecturer and he's been teaching the students about subliminal, uh, effects. And the, you know, the students decide to play a trick on him, which is, you know, every, every time he's on the left-hand side of the room, they pay attention, they're really interested.

• 1:02:28Speaker 00

  And every time he walks onto the right-hand side of the room, they all really bored, you know, start checking out the email and so on. And, uh, and by the end of the lecture, he's glued against the left-hand, right? And he has no idea that he's being manipulated. Um, but because of the fact that this was like systematic and, uh, you know, and sufficiently frequent, it's, has a very, very strong effect. Uh, you know, and I think that, that the difference here is that it's because it's algorithmic, um, and it's tied into this very high-frequency interaction, uh, that people have with social media, it, it has a huge effect.

• 1:03:09Speaker 00

  Um, and it has, I think, a pretty rapid effect as well.

• 1:03:14Chris Williamson?

  What are some of the concerns you had? You mentioned it earlier on about is it enfeebled, becoming too enfeebled to the machines?

• 1:03:21Speaker 00

  Uh, yeah, so this is, I think, one of, one of two major concerns, you know, if we manage to create superhuman AI and to control it.

• 1:03:37Speaker 00

  One concern is the misuse concern, right? So I call it the Dr. Evil problem, right? Dr. Evil doesn't want to use the provably safe, controllable AI. He wants to make his own AI that's going to take over the world. And you can imagine that gets out of control and then bad things happen. The enfeeblement problem is sort of overuse, right? That we, because we have available to us AI systems that can run our civilization for us, we lose the incentive to know how to run it ourselves.

• 1:04:20Speaker 00

  And that problem, you know, it's really, it's, it's a really hard problem to figure out how to prevent it.

• 1:04:33Chris Williamson?

  Because inevitably, the AI would have to make the human do something that probably in the moment, the human didn't want to do. The AI would actually be programming itself to be less useful than it could be in order to give us a sort of hormesis stressor dose that allows us to stay useful.

• 1:04:51Speaker 00

  Yeah, I mean, it's, so when I say overuse, I mean, I literally mean that, right? That we would use AI too much for our own good. And so E.M. Forster has a story, you know, so he usually wrote, you know, late Victorian, early Edwardian social, you know, British upper class, social issue kinds of novels, but he wrote a science fiction story called The Machine Stops, which I also recommend to your audience. And it's quite amazing, right?

• 1:05:33Speaker 00

  It, it has the internet, email, MOOCs.

• 1:05:38Chris Williamson?

  When was it written?

• 1:05:39Speaker 00

  1909. It has iPads, video conferencing, you know, people are obese because they never get out of their chairs, they're stuck on the computer all the time. They start to eschew face-to-face contact because they are.

• 1:06:04Speaker 00

  Basically glued to the screen and lose contact with the physical environment altogether. You know, all the kinds of things that people complain about now, he wrote about. And the machine, right, so it's not just the internet, it's a whole, it's called the machine, that looks after everyone's physical needs. And so, so they just spend their time glued to the screen. They don't know how the machine works anymore. And they don't need to, because the machine runs itself. And then, of course, it stops running itself, things go south.

• 1:06:47Speaker 00

  But, you know, I did a little back-of-the-envelope calculation. So it turns out that about 100 billion humans have lived.

• 1:06:57Speaker 00

  And our civilization, right, is passed on by teaching the next generation of humans everything we know, and then maybe they'll learn a bit more. Right, and if that process fails, right, you know, if it goes into reverse, that the next generation knows less, right, then you could imagine things unraveling. And, you know, so the total amount of time spent just passing on civilization is about a trillion person-years of effort.

• 1:07:36Speaker 00

  And for that to end would be the biggest tragedy that one could possibly imagine. Right, but if we have no incentive to learn it all, because finally, right, instead of having to put it into the heads of the next generation of humans, we could just put it into the heads of the machine, and then they take care of it. Right, and if you've seen WALL-E, right, that's exactly what happens. Right, and they even show what happens to the generations over time, right? They become stupider and fatter and...

• 1:08:13Speaker 00

  Unless, you know, they can't run their own civilization anymore. So the machines should say, because this is such an important thing, right? It's of value to us and to our descendants that we are capable, that we are knowledgeable, we know how to do things. That we have autonomy, intellectual vigor. Those are really important values. But, so the machines should say, okay, we are going to stand back and let these humans, they have, we have to let the humans tie their own shoelaces.

• 1:08:49Speaker 00

  Otherwise, they'll never learn.

• 1:08:52Speaker 00

  But we, right, we are short sighted, lazy, greedy people, and we might say, no, you have to tie our shoelaces. We'll keep doing that. And then we, then we lose that autonomy and that intellectual vigor. So this is a cultural problem, right? It's a problem with us. The technology might be saying, no, no, no, no, no, but we're overriding it.

• 1:09:19Chris Williamson?

  All of these problems, all of these problems are problems with us. The problems are the fact that our goals are plastic, the fact that our language is imprecise, the fact that we are sometimes rational, sometimes irrational, that we don't have an omnipotent view, that we can see what we want when we're going to want it. Also, these challenges around the fact that sometimes we want something in the moment that we're not going to want in the future. And that we're going to complain at the algorithm and say, well, no, no, machine, you're supposed to be here to do my bidding, and now you're telling me that I've got to walk to the shop to get the milk. I want you to get the milk. Well, sire, I don't know why he's middle aged, middle aged peasant now. Well, sire, you must go to the milk, get the milk yourself. You know that it is good for your calves and your bone density.

• 1:10:00Chris Williamson?

  So, like, I mean, this, we haven't even touched on how this even begins to be converted into computer code, which is, I imagine, a whole other complete minefield of difficulty to be able to actually get what we're talking about. This is purely sort of within the realm of philosophy. What are, what are some of the challenges that we have here when you have a all powerful super being that can do whatever you want?

• 1:10:28Speaker 00

  Yeah, I mean, it's inevitable in a sense, because what we, what we've always wanted is technology that's beneficial to us.

• 1:10:39Speaker 00

  And

• 1:10:42Speaker 00

  A lot of the time, we say, oh, well, here's a technological idea. You know, I hope it's going to be beneficial to us, like, you know, the motor car, for example, right?

• 1:10:53Speaker 00

  And then it turns out not to be. Or at least arguably, although it conveyed lots of benefits, it might have ended up being our destruction.

• 1:11:05Chris Williamson?

  It's one of the Bostrom's black balls, right? Or gray balls, I suppose, out of the urn.

• 1:11:10Speaker 00

  It's, yeah, so it's responsible largely for for the destruction of the climate. And so unless we get out of this, it will have been a really bad idea to do it. But almost by definition, with something as powerful as AI or as superintelligent AI,

• 1:11:36Speaker 00

  you need to know, it's either going to be very beneficial or very not beneficial. Right? It's not going to be like margarine versus butter or something like that, right? And so

• 1:11:50Speaker 00

  so we have to ask ourselves, okay, what does beneficial mean?

• 1:11:57Speaker 00

  Right? If we're going to prove a theorem that such and such, you know, developing such a technology is actually going to be beneficial.

• 1:12:07Speaker 00

  Then inevitably, it comes back to what are humans and what do we want? And how do we work? And so,

• 1:12:18Speaker 00

  yeah, so that that kind of surprised me when I started along this path of trying to solve the control problem.

• 1:12:28Speaker 00

  You know, I had ideas for algorithms that that would do it and so on, but I didn't realize the extent to which it would push me into these these humanistic questions. And that's been fascinating and it's it's a little bit of a minefield for you know, for a technologist to stray into these areas because they they're they operate in different ways. In many ways, they're much more vicious than the technological fields, because in technology, it's sort of, it's

• 1:13:04Speaker 00

  There's us humans and then it works or it doesn't, right? Or it's true, it's a true scientific theory or isn't. So there's this third party called nature out there. But in the humanistic areas, there isn't a third party, right? It's just, you know, one school of thought, another school of thought.

• 1:13:24Chris Williamson?

  It's just debates all the way down.

• 1:13:26Speaker 00

  Fighting it out for supremacy. And it takes a while to adjust to that.

• 1:13:33Speaker 00

  But the questions are, you know, super important and really fascinating. So I've enjoyed it a lot. So coming back to the question of the algorithms, right?

• 1:13:52Speaker 00

  You know, one way to think about it is that's perhaps a little bit less daunting is to look back at what's happened in AI with respect to, shall we say, ordinary uncertainty.

• 1:14:12Speaker 00

  So in the early days of AI, we mostly worked on problems where the rules were known and fixed and deterministic, like the rules of chess. Or, you know, finding a path through a map or something like that, right? We know what the map is. We know that if you turn left, you go left. So we can have logical rules. We could use these deterministic symbolic techniques to solve the problem. And then we found that as we moved into the real world,

• 1:14:45Speaker 00

  uncertainty becomes really important, right? So, you know, if you're controlling the Mars rover and you have to, you know, you give it some command to, you know, to go 70 meters in a particular direction because it takes 10 minutes for the commands to go backwards and forwards.

• 1:15:06Speaker 00

  Is it going to get there? Well, you don't know. It might get stuck or it might, you know, deviate a little bit or, you know, one wheel will start spinning and it won't make any progress. So who knows? So real world problems, you always have to handle the uncertainty in your knowledge of.

• 1:15:23Speaker 00

  Um, the physics of the world and in your even just what your senses are telling you, right, that their senses themselves are imperfect and noisy and incomplete. So uncertainty became a core consideration in AI around nineteen, early nineteen eighties, I would say. And so that period from, you know, late eighties to uh early two thousands was really the period where probability was, you know, the dominant paradigm for AI research. But in all that time, it does not seem to have occurred to anyone, except for a few,

• 1:16:04Speaker 00

  you know, uh, I think very bright people, that there is also uncertainty in the objective. So we have all these problem formulations for decision making under uncertainty, but they assume that you know the objective exactly and perfectly. And at least looking back at that now, it's just like, well, that's bonkers, right? It's just as bonkers as assuming that you know the physics of the world exactly and perfectly, or that your senses give you exact and perfect access to the state of the world at all times, right? Because, you know, we had already seen many examples of objective failure, right?

• 1:16:46Speaker 00

  Where we specified the wrong objective and the machine did something completely, you know, that we thought was completely bonkers, because in fact it was doing exactly what we told it to do, we just didn't realize what we had told it. And my favorite example is um one from uh from simulated evolution. And um, you know, so simulated evolution, you you you define a fitness landscape, right? So which simulated creatures are considered to be more fit or less fit, and therefore they get to reproduce and mutate, and gradually you can evolve creatures that are really, really good at whatever it is you want them to be good at.

• 1:17:28Speaker 00

  So um, so the objective was, uh, well, what they wanted was to evolve creatures that could run really fast. So they specified the objective as the maximum velocity of the center of mass of the creature.

• 1:17:50Speaker 00

  And that sounds like a perfectly reasonable definition. So what evolved enormously tall trees, like a hundred miles high, that would then fall over?

• 1:18:04Speaker 00

  And in fall, they went really, really fast. So they won the competition, right?

• 1:18:11Speaker 00

  They turned out to be the solution to that problem.

• 1:18:14Chris Williamson?

  Someone thought they were going to get some supercharged nitro cheetahs or leopards or something.

• 1:18:19Speaker 00

  Exactly.

• 1:18:20Chris Williamson?

  Instead, you end up with trees reaching up into the stratosphere and then falling all over the place.

• 1:18:25Speaker 00

  Yeah. So, you know, I thought that's a great example, you know, and of course, that's only, you know, that's a simulation in the lab. So people for ho, ho, ho. And I mean, okay, let me fix the problem. But of course, in the real world, right, in in your climate engineering system, or your, you know, your economic governor, or whatever it might be, right, that you can't just go ho, ho, ho.

• 1:18:47Chris Williamson?

  We'll fix it. It will press the reset button. Brian Christian told me about a problem, robots playing football, and they'd put a very small utility function in for gaining control of the ball. Possession is an instrumental goal towards scoring that you can't score if you don't have the ball. And what the robot found was that it could actually maximize its utility function by going up to the ball and vibrating its paddle a hundred times a second up against the ball, which is far easier than actually trying to score.

• 1:19:22Speaker 00

  Yep.

• 1:19:22Chris Williamson?

  It ended up thinking it had done really great. And the guys just had these sort of seizure-ing robots all over the pitch, vibrating up against the ball.

• 1:19:32Speaker 00

  Yeah, exactly. Right. Which is sort of what happens with little, little kids sometimes, right? They want to get the ball, and they want to get the ball, and they don't know what to do with it once they've got it.

• 1:19:42Chris Williamson?

  Yeah, yeah.

• 1:19:43Speaker 00

  But anyway, the, yeah, I mean, that, some of these problems have technical solutions. That, that particular one, there's a, there's a theorem for what kind of supplemental rewards you, you can provide that are intended to be helpful.

• 1:20:01Speaker 00

  But we'll end up not changing the final behavior. So it'll make it easier to learn the final behavior, but it will ensure that the final behavior is still optimal according to the original objective. So you can fix that aspect of the problem. But if you leave something out from the objective, you left it out. And that's equivalent to saying it's worth zero.

• 1:20:25Speaker 00

  Right? So anything you leave out of the objective is like saying it's a value zero to humans. And that's, you know, that's a big problem. So you almost always want to say, well, I told you some of the things I care about, but there's other stuff I haven't told you and you should be aware of that.

• 1:20:44Chris Williamson?

  It feels to me like there's sort of two cars in this race. On one side, we have technological development that has lots of facets, hardware, algorithms, so on and so forth. And then on the other side, you have the control problem. You have getting the alignment right. It has to be that the alignment problem gets across the finish line before the technology does,

• 1:21:09Chris Williamson?

  or else we are essentially rolling the dice and hoping that we've got it right by some sort of fluke. And I imagine that there are far more ways of getting it wrong than there are of getting it right.

• 1:21:20Speaker 00

  Yeah, well, I mean, getting it wrong is actually the default. Right? If we just continue pushing on the AI technology in the standard model, we'll get it wrong.

• 1:21:33Chris Williamson?

  Are you, so is there any chance that if you continued with the standard model, that it could be right? Or would you give it such a low chance that it's negligible?

• 1:21:41Speaker 00

  I think it's negligible. And arguably, what's happening with social media is an example of getting it wrong. You know, and other people have pointed out that, you know, we don't need to wait for AI. Corporations that are maximizing, you know, quarterly discounted profit stream are also examples of machines pursuing incorrectly defined objectives that are destroying the world. And, you know.

• 1:22:11Speaker 00

  If you look at the climate issue from that point of view, I find it sort of enlightening, right? We have been outwitted by this AI system called the fossil fuel industry. It happens to have human components, but the way corporations are designed, right? They are machines with human components. And actually, the individual preferences of the humans in those machines don't really matter very much, because the machine is designed to maximize profit. And they outwitted the human race. They developed, you know, for more than 50 years, they've been running global propaganda

• 1:22:54Speaker 00

  and subversion campaign to keep, you know, to enable them to keep selling fossil fuels. And they won. Right? We can all say we're right, or we know that we shouldn't be doing this, and we know that, we know the causes of climate change, right? But we lost.

• 1:23:21Chris Williamson?

  A lot fewer implications of that than an all-knowing, all-powerful artificial intelligence, though. So although the implications are still grave, if the climate problems get worse, it's not the same. And again, the control problem simply has to get across the line. If you're essentially adamant that currently, if you scale up the competence, not probably not the power, I suppose, of the computation that we have, it's bad. It's a bad situation. But obviously, you have-

• 1:23:59Speaker 00

  The difference is the irreversibility, right? I mean, climate change probably isn't going to make us extinct, you know, unless there's some real chaos theory catastrophe that happens.

• 1:24:13Speaker 00

  And eventually, we'll all be so fed up that we actually retake control from the fossil fuel industry. And that's sort of happening.

• 1:24:22Speaker 00

  But yeah, with AI, it could be irreversible, this loss of control.

• 1:24:29Speaker 00

  And, you know, if, if, if I'm right, that examples like social media are showing that we are already seeing the negative consequences of incorrectly defined objectives and, and, you know, even relatively weak machine learning algorithms that are, that are pursuing them. Um, then we should pay attention, right? These are the canaries in the coal mine. We should be saying, okay, we, we need to slow down and we need to,

• 1:25:05Speaker 00

  you know, look at this different paradigm. And, you know, the, the standard model is sort of just one corner, right? It's the corner where the objective is completely and perfectly known. At least that's the, that's the corner where it's appropriate to use the standard model. Right? And there's this, all the rest of the building, we haven't even looked at yet. Right? Where there's uncertainty about what the objective is and the system can uh behave accordingly. Um, and we've just, you know, just in the last few years have we had any algorithm that can solve uh this new category of problem.

• 1:25:49Speaker 00

  And it does, I mean, so the algorithms exist, right? They're, they're very simple and they work in very uh restricted and simple um instances of the problem, but they, they show the right properties, right? That they, they defer, they ask permission. Um, they understand what the human is trying to teach them about, about human preferences. And um, you know, it's, it seems our job in what's called, for want of a better word, the AI safety community. Our job is to build out that technology.

• 1:26:31Speaker 00

  To create all the new algorithms and theoretical frameworks and demonstration systems and so on to convince the rest of the AI community that this is the right way to do AI.

• 1:26:45Chris Williamson?

  Because we can't do the other thing in the race, we can't slow down the technological progress because trying to neuter one particular agent or actor or country or nation state or even one group of nations doesn't guarantee that some other group is not going to. China saying, right, we'll stop doesn't mean that America will say, we're just going to keep going or vice versa.

• 1:27:09Speaker 00

  Yeah, well, it's just, I mean, the the potential upside that people are seeing is so huge, right? I mean,

• 1:27:21Speaker 00

  when I when I say it would be the biggest event in human history, I mean it.

• 1:27:28Chris Williamson?

  Why?

• 1:27:29Speaker 00

  Right, it's not because

• 1:27:34Speaker 00

  you know,

• 1:27:37Speaker 00

  our advantage as humans, our whole civilization is based on a certain level of intelligence that we have, right? So we, our brains are the source of the the intelligence fuel that that makes our civilization go around. If we have access to a lot more all of a sudden, right, that is a step change

• 1:28:01Speaker 00

  in our civilization.

• 1:28:04Speaker 00

  And, you know, on the upside, it might enable us to solve problems that have been very resistant, like disease, poverty, conflict.

• 1:28:21Speaker 00

  And on the downside, it might just be the last thing we do.

• 1:28:28Speaker 00

  So.

• 1:28:30Chris Williamson?

  If you could, if you had a god's eye view, would you put a pause on technological development for 100 years outside of the control problem, for 1,000 years, for 50,000 years? Because we've spoken about the dangers of killing people that haven't yet been born. And when you're talking about civilizational potential, the observable universe, you know, galactic-sized empires, von Neumann probes, making everything is, you know, you're talking trillions and trillions of human lives, even if you go from the utilitarian approach. You have an unlimited amount of utility and happiness that could be given. And because we're unable at the moment to slow down technology, potentially within

• 1:29:14Chris Williamson?

  the next 100 years, all of that could be snuffed out.

• 1:29:18Speaker 00

  Yeah, it's interesting. In many of the works, I think in Bostrom's and in Max Technok's and others, the argument is based on, you know, these quintillions of future humans who might, you know, be able to colonize the universe and so on. That's never been a motivation for me, right? I mean, if I have a picture, it's just of a small village with a church and people playing cricket on the village green. And that's what I do. I don't want that to disappear. I don't want the civilization that we have to be gone because it's the only thing that

• 1:30:02Speaker 00

  has value. And I try not to think about what I'd do if I was God, as you say.

• 1:30:12Chris Williamson?

  Not good for the ego.

• 1:30:16Speaker 00

  Well, it's just...

• 1:30:23Speaker 00

  I mean, obviously, I don't have, no, no one, I think, is going to be able to switch off um scientific progress. You know, it, it, there are precedents. The biologists switched off progress on direct modification of the human genome in ways that are what they call heritable modifications, germ line editing. They switched that off. They said, you know, in nineteen seventy-five, or gradually from nineteen seventy-five onwards, they, they decided that that was not something they wanted to do.

• 1:31:08Speaker 00

  Which is interesting, because a large part of the history of genetics and that whole branch of biology, the improvement of the human stock was actually one of the major objectives. You know, and eugenics before the Second World War, thought of itself as a noble, a noble mission. Now, you could argue about, about that, but what the biologist is to say, you know what? We could do this, but we're not. Right? That was a big step, and, you know, is it possible for that to happen in AI?

• 1:31:53Speaker 00

  I think it's much more difficult. Because in biology, right, we are continuing to understand, right, so the developmental biology, right? So how does a given DNA sequence produce an organism, right? And what goes wrong? And, you know, is it a problem with genes or a problem with the, the development environment of the organism or what? And if you understand all those questions, then presumably you could, you could then

• 1:32:35Speaker 00

  say, okay, now I know how to modify the human genome so we can avoid all those problems. So the, the scientific knowledge is moving ahead.

• 1:32:48Speaker 00

  But the decision is, we're not going to use that knowledge for that kind of thing. And you can draw that boundary pretty easily because, you know, we're talking about physical, you know, physical procedures involving actual human beings and so on. And that's been regulated for many decades already. And so with AI,

• 1:33:14Speaker 00

  once you understand how to do something, it's pretty much done.

• 1:33:20Speaker 00

  Right there.

• 1:33:25Speaker 00

  Mathematics and code are just two sides of the same coin.

• 1:33:30Speaker 00

  And, uh, you know, mathematical ideas, you can't

• 1:33:38Speaker 00

  go around looking at everyone's whiteboard and saying, okay, I see you've got, you know, sigma for x equals one. Okay, stop right there, right? Yeah, that's, that's one too many Greek symbols. You've got to stop writing, right? You know, so we,

• 1:33:55Speaker 00

  because the question of, you know, decision making and learning and so on, these are fundamental questions. We can't stop research on them. So I have to assume

• 1:34:09Speaker 00

  that the scientific understanding of how to do it is just going to continue to grow.

• 1:34:19Speaker 00

  If, if it was the case, which some people seem to think that

• 1:34:24Speaker 00

  to go from a scientific understanding to a deployed system would require, you know, some massive gigawatt installation, you know, with billions of GPUs and so on and so forth, then perhaps you could regulate at that point.

• 1:34:38Chris Williamson?

  Because there would be a physical limitation that would be quite easy to enact. Okay, you can't have this many, this much power, this many. I'm going to guess that you feel otherwise, that you don't need that much hardware to run something that could be quite dangerous?

• 1:34:51Speaker 00

  Correct. Yeah. I, I think we already have enough power, as I said. And, uh,

• 1:35:00Speaker 00

  you know, it's, it's very hard to do meaningful calculations, but in just in terms of raw.

• 1:35:07Speaker 00

  Numerical operations per second.

• 1:35:11Speaker 00

  A Google TPU pod, which is the tensor processing unit, you know, even three or four years ago,

• 1:35:21Speaker 00

  was operating at a higher rate than the possible theoretical maximum capacity of the human brain. Right, so ballpark figure for the human brain is ten to seventeen operations per second. But I don't think any neuroscientists believe that we're doing anything like that much, right? I mean, they would probably ballpark it at ten to the twelve or ten to thirteen or something like that. But if you if you grant every possibility, it's ten to seventeen, where, you know, TPU pod, which is, you know, sort of wardrobe sized thing is a ten to the seventeen, you know, the biggest supercomputer is a ten to the eighteen.

• 1:36:04Speaker 00

  So,

• 1:36:06Speaker 00

  uh, you know, I think we have way more than enough power to build a super intelligent machine. So I just don't, I don't think that trying to cut it off at the, you know, large scale hardware installation level is going to be feasible either. Any more, you know, if you remember the old

• 1:36:25Speaker 00

  Apple ads for the G five.

• 1:36:30Speaker 00

  So the US had these export, put export controls on anything that was more than one gigaflop. Right, which sounds ridiculous now, but they put export, you know, because they didn't want those falling into the hands of the Russians or the Chinese. So the, you know, so A- Apple produced this ad, you know, with their little G five, like this little cube, and they had all these tanks like surrounding this little G five and saying, you know, this, this little G five is now, you know, the, the US government has, has

• 1:36:58Chris Williamson?

  Too hot to handle. Yeah, exactly.

• 1:37:01Speaker 00

  Right, so it's, you know, they used it as advertising material. So it's,

• 1:37:06Speaker 00

  you know, it, it's just unlikely that you could,

• 1:37:11Speaker 00

  that you, you could prevent the creation of super intelligent AI just by regulating hardware installations.

• 1:37:22Speaker 00

  So,

• 1:37:24Speaker 00

  so I do think of it as, as you say, a race.

• 1:37:28Speaker 00

  I think, you know, we may see.

• 1:37:34Speaker 00

  Catastrophes that are more obvious and unequivocal than what's happening in social media.

• 1:37:45Speaker 00

  And, you know, so that could happen on a small scale in self-driving cars. You know, I, I thought when the first Tesla completely failed to see a huge white truck and crash straight into it at full speed, I thought, you know, that kind of accident should at least say, okay, maybe these AI systems are not as good as we thought they were, but didn't seem to have much impact. And, you know, we've, we've killed several more people pretty much the same way.

• 1:38:18Speaker 00

  So it would have to be, I think, something, something pretty major would have to happen.

• 1:38:25Chris Williamson?

  Well, you say that, but I've been thinking this over the last 16 months, that COVID should have been the biggest wake-up call for synbio, for natural pandemics, for engineered pandemics, for research into that, any, anything over that side of the aisle, for whatever it is, BSL three or BSL four labs, they should all be on the moon, they should all be, you know, on the bottom of the ocean, we should be air gapped from them. And no one's talking about that. Rob Reid, Rob Reid's talking about it and that's it. Like, there's no one, no one's bothered. They're not.

• 1:38:59Speaker 00

  I think, I, I don't know. I, I have heard some biologists talking about reevaluating.

• 1:39:07Chris Williamson?

  After this global pandemic, maybe, maybe we should have a meeting, you know, have a cup of tea or something. But I just think that humans, because life's so comfortable at the moment for us, mostly, and because we have attached our sense of wellbeing to the progress of technology, I think everybody is praying at the altar of that currently. And the presumption is always more technology is good, there may be some hiccups along the way, but mostly we'll be able to fix it. And if we can't, we'll make a technology that can fix it, that will be enabled by the technology that was wrong. But there is a, as you say, a step change. You know, it's not just a change in degree, it's a change of kind. When we reach this particular level, recursive self-improvement, blah, blah, blah, game over.

• 1:39:53Speaker 00

  Yeah, well, I think arguably the same thing is going on in biology, whether it's genome line modification or, you know, synthesis of pathogens.

• 1:40:08Speaker 00

  You know, and interestingly, you know, for DNA synthesis, there is a response. It's not widely publicized, but all the manufacturers of DNA synthesis equipment are now building in circuitry, which is non-public, right? Which is a black box circuitry, which is checking the sequences to make sure you're not synthesizing any disease organism. Yep. And there, I think there's even a notification requirement. So if you try, someone will be knocking on your door very quickly.

• 1:40:48Speaker 00

  You know, so that's, that's, I think, a very sensible precaution.

• 1:40:56Speaker 00

  And, you know, and even so, right, that there's a movement within synthetic biology, sort of libertarian movement, the garage movement, saying, we should be able to synthesize whatever we want. It's our right, you know, it's scientific freedom. I'm sorry, I think you're nuts, right? There is, there is no, you know, scientific freedom is, is a value, but it doesn't trump all other values, including continued human existence.

• 1:41:27Speaker 00

  So, you know, and eventually, I think, AI, you know, computer science and AI have got to accept the same thing, right? We, we've now reached a point.

• 1:41:43Speaker 00

  Where we're able to have a really significant effect on society, for good or bad. And so that's time to grow up, right? It's time to accept that you have responsibilities and that society has a right to control what effect you have.

• 1:42:03Chris Williamson?

  That's a whole other conversation there when we think about the current push towards libertarianism on the internet. Web 3.0 decentralized, you can't stop me, you can't stop my money. I can be wherever I want, do whatever I want with whoever I want. I think that there's going to be a serious sort of cultural conflict there. You, one of the reasons that you wrote Human Compatible was as a wake up call to people within the sort of AI research safety community. We're approaching, what, two years, almost exactly two years since it was published. What has the effect of the book and your subsequent work press been? Has it had anything close to the impact that you intended? Has it had any other sorts of impacts?

• 1:42:50Speaker 00

  That's a very good question and honestly, I don't know the answer.

• 1:42:57Speaker 00

  There's certainly more academic interest in these topics.

• 1:43:04Speaker 00

  You know, the number, we have a center at Berkeley and the number of people coming to the workshop has been increasing rapidly.

• 1:43:15Speaker 00

  You know, workshops that we hold at, at the main conferences are growing really fast, hundreds and hundreds of people.

• 1:43:24Speaker 00

  I often, you know, I get emails from all kinds of AI researchers who say, you know, okay, I, I agree with everything you're saying in the book. You know, how, how can I redirect my research to be helpful in this?

• 1:43:42Speaker 00

  I believe, based on various sort of grapevine dribs and drabs, that the questions of control have filtered up to the highest levels of government in various countries, that this is now one of the risks that's considered, you know, when people take stock of, okay, what do we need to pay attention to over the horizon? This is one of the risks.

• 1:44:18Speaker 00

  But, you know, on the other hand, if I was just a, you know, mid-level AI manager in a technology corporation, you know, would I change what I'm doing? Well, I think my recommendation is always, look at the objective that you're specifying for your algorithms. Look at the actions that your algorithm could take and ask, could any of the effects of your algorithm go outside the scope that you're thinking of for your objective? And have you thought about all the possible consequences?

• 1:45:02Speaker 00

  And whether they would be desirable? That should be something that we do as a matter of course. And I would say, you know, the new EU AI regulations actually instantiate some of that. So that's reasonably good. So I, but there is no, you know, I can go and buy TensorFlow. I can, you know, I can go download reinforcement learning libraries, you know, there's tons of software in the standard model. But there's almost nothing in the new model.

• 1:45:46Speaker 00

  So there's just tons of research. Literally every chapter of the textbook has to be rewritten from scratch. And you can't rewrite it until someone's done all the work to create the new algorithms and the new theorems, and even to figure out the right way to formulate the problems.

• 1:46:10Speaker 00

  So that's something that I have to do, and my students have to do, and the other groups around the world that are working on this. And the sooner we get that done, the better, right? These will be better AI systems. It's not, you could think of them as, well, okay, you over there doing those traditional AI systems, you're bad, bad, bad, bad, bad, you know, and you need to fix your, fix your approach. But that's never been very effective as a way to get people to change their behavior, right? What you want them to get up in the morning and say, I'm going to build a really good AI system today. But what that means is one that is beneficial to human beings, right? And, and just like an engineer, a civil engineer

• 1:46:55Speaker 00

  who designs bridges, he gets up every morning and says, I'm going to build the best bridge I can, right? And what is best bridge means what's come to mean bridges that don't fall down.

• 1:47:07Speaker 00

  Right? And I was reading the memoir of the guy who ran the Russian bioweapons program. And it was clear that he got up every morning and said, I'm going to do the best science I can today. What did best mean for him? It means, you know, making anthrax more fatal, you know, and, and making infectious diseases more infectious, right? That's what best meant for him. So you can affect what people do by affecting what they think of as good science or good engineering. And it's, to me, it stands to reason that it's not good engineering if it does things that make you very, very unhappy.

• 1:47:47Speaker 00

  So we need to change the way people think about what is good AI and then give them the tools to make it.

• 1:47:56Chris Williamson?

  And time's running out.

• 1:47:58Speaker 00

  Yep.

• 1:48:00Chris Williamson?

  Stuart Russell, ladies and gentlemen, thank you very much for joining me today. If people want to check out your stuff, where should they go?

• 1:48:06Speaker 00

  So the book Human Compatible is available, including as an audiobook with a very plummy accent, not mine, but it's pretty good. My webpage, you can Google me, my webpage has all the publications. The Centre for Human Compatible AI has a bunch of resources. And then there are other groups, such as the Future of Humanity Institute at Oxford, Centre for the Study of Existential Risks at Cambridge, Future of Life Institute at MIT and Harvard. So there's a bunch of groups around the world and we're, you know, we're working as hard

• 1:48:51Speaker 00

  as we can to train students and get them out into all the universities and then teach the next generation how to go forward.

• 1:48:58Chris Williamson?

  Good luck, human.

• 1:49:00Speaker 00

  Thank you very much. Very nice talking to you, Chris.