Pod Insights a16z Podcast Controlling AI About this episode AI can do a lot of specific tasks as well as, or even better than, humans can — for example, it can more accurately classify images, more efficiently process mail, and more logically manipulate a Go board. While we have made a lot of advances in task-specific AI, how far are we from artificial general intelligence (AGI), that is AI that matches general human intelligence and capabilities?
In this podcast, a16z operating partner Frank Chen interviews Stuart Russell, Founder of the Center for Human-Compatible Artificial Intelligence (CHAI) at UC Berkeley. They outline the conceptual breakthroughs, like natural language understanding, still required for AGI. But more importantly, they explain how and why we should design AI systems to ensure that we can control AI, and eventually AGI, when it’s smarter than we are. The conversation starts by explaining what Hollywood's Skynet gets wrong and ends with why AI is better as "the perfect Butler, than the genie in the lamp."
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 Hi and welcome to the A16Z Podcast. I'm Doss and in this episode, Frank Chen interviews UC Berkeley Professor of Computer Science Stuart Russell. Russell literally wrote the textbook for artificial intelligence that has been used to educate an entire generation of AI researchers. More recently, he's written a follow-up, Human Compatible: Artificial Intelligence and the Problem of Control. Their conversation covers everything from AI misclassification and bias problems, to the questions of control incompetence in these systems, to a potentially new and better way to design AI. But first, Russell begins by answering, where are we really when it comes to artificial general intelligence, or AGI, beyond the scary picture of Skynet?
0:42 Speaker 01 Well, the Skynet metaphor is one that that people often bring up, and I think generally speaking Hollywood has got it wrong. They always portray the risk as an intelligent machine that somehow becomes conscious, and it's the consciousness that causes the machine to hate people and want to kill us all. And this is just a, this is just a mistake. The problem is not consciousness, it's really competence. And if you said, oh, by the way, you know, your laptop's now conscious, it doesn't change the rules of C++ right? The software still runs exactly the way it was always going to run when you didn't think it was conscious.
1:19 Frank Chen? So on the one hand, we have people like Elon Musk saying artificial general intelligence, like that's a real possibility. It may be sooner than a lot of people think. And on the other, you've got people like Andrew Ng who are saying like, look, we're so far away from AGI. All of these questions seem premature, and I'm not going to worry about the downstream effects of super intelligent systems until I worry about overpopulation on Mars. So what's your take on?
1:45 Speaker 01 Yeah, so he in fact upgraded that to overpopulation on Alpha Centauri.
1:51 Speaker 01 So let's first of all, talk about timelines and predictions for achieving human level or superhuman AI.
1:59 Speaker 01 So Elon actually is reflecting advice that he's received from AI experts. So some of the people, for example, at OpenAI think that five years is a reasonable timeline. And that the necessary steps mainly involve much bigger machines and much more data.
2:18 Frank Chen? So no conceptual or computer science breakthroughs, just more compute, more storage, and we're there.
2:23 Speaker 01 Yeah, so I really don't believe that.
2:26 Frank Chen? Yeah.
2:26 Speaker 01 Crudely speaking, right, the bigger, faster the computer, the faster you get the wrong answer. But I believe that we have several major conceptual breakthroughs that still have to happen. We don't have anything resembling real understanding of natural language, which would be essential for systems to then acquire the whole of human knowledge. We don't have the capability to flexibly plan and make decisions over long time scales. So we're very impressed by AlphaGo or AlphaZero's ability to think 60 or 100 moves ahead. That's superhuman. But if you apply that to a physical robot, whose decision cycle is a millisecond,
3:10 Speaker 01 that gets you a tenth of a second into the future, which is not very useful if what you're trying to do is not just lay the table for dinner, but do it anywhere.
3:19 Frank Chen? Right.
3:20 Speaker 01 All right, in any house, in any country in the world, figure it out. Laying the table for dinner is several million or tens of millions of motor-controlled decisions. And at the moment, the only way you can generate behavior on those time scales is actually to have canned subroutines that humans have defined. Pick up a fork, okay, I can train picking up a fork, but I've defined picking up a fork as a thing. So machines right now are reliant on us to supply that hierarchical structure of behavior. When we figure out how they can invent that for themselves as they go along and invent new kinds of things to do that we've never thought of, that will be a huge step towards real AI.
4:00 Frank Chen? As we march towards general intelligence, this literal ability to think outside the box, right, will be one of the hallmarks, I think, we look for.
4:09 Speaker 01 If you think about what we're doing now, we are, we're trying to write down human objectives. It's just that we tend, because we have very stupid systems, they only operate in these very limited contexts, like a Go board. And on the Go board, you know, A natural objective is win the game. If AlphaGo was really smart, even if you said, "Win the game." Well, you know, I can tell you, here's what chess players do. When they're trying to win the game. They go outside the game. And a more intelligent AlphaGo would realize, okay, well, I'm playing against some other entity. What is it? Where is it? Right? There must be some other part of the universe besides my own processor and this Go board.
4:51 Speaker 01 And then it figures out how to break out of its little, little world and start communicating. Maybe it starts drawing patterns on the Go board with Go pieces to try and figure out, you know, what, what visual language it can use to communicate with these other entities. Now, how long do these kinds of breakthroughs take? Well, you know, if you look back at nuclear energy, for the early part of the 20th century, when we knew that nuclear energy existed. So, you know, from E equals mc squared in 1905, we could measure the, the mass differences between different atoms. We knew what their components were. And they also knew that, you know, radium could emit vast quantities of energy over a very long period. So they, they knew that there was this massive store of energy.
5:36 Speaker 01 But mainstream physicists were adamant that it was impossible to ever release it. And to harness it in some. Right. So there was a famous speech that Lord Rutherford gave. And he was, you know, the, the man who split the atoms. It's like the leading nuclear physicist of his time. And that was September 11th, 1933. And he said that the possibility of extracting energy by the transmutation of atoms is moonshine. The question was, you know, is there any prospect in the next 25 or 30 years? So he said, no, it's impossible. And then the next morning, Leo Zilard actually read a report of that in The Times and went for a walk and invented the nuclear chain reaction based on neutrons, which people hadn't thought of before. And that was a conceptual breakthrough. Right. You went from impossible to now it's just an engineering challenge.
6:21 Speaker 01 So we need more than one breakthrough, right? You know, it takes time to sort of ingest each new breakthrough and then, and then build on that to get to the next one. So the average AI researcher thinks that we will achieve superhuman AI sometime around the middle of this century. So my personal belief is, is actually more conservative. One point is, we don't know how long it's going to take to solve the problem of control. If you ask the typical AI researcher, "Okay, and how are we going to control machines that are more intelligent than us?" They'll say, "Beats me." So you've got this multi-hundred-billion-dollar research enterprise with tens of thousands
7:02 Speaker 01 of brilliant scientists, all pushing towards a long-term goal where they have absolutely no idea what to do if they get there. So coming back to Andrew Ng's prediction, the analogy just doesn't work. If you said, "Okay, the entire scientific establishment on Earth is pushing towards a migration of the human race to Mars, and they haven't thought about what we're going to breathe when we get there," you'd say, "Well, that's clearly insane."
7:29 Frank Chen? Yeah, and that's why you're arguing we need to solve the control problem now, or at least the right design approaches to solving this control problem.
7:37 Speaker 01 So it's clear that the current formulation, the standard model of AI, as "build machines that optimize fixed objectives," is wrong. We've known this principle for thousands of years, that be careful what you wish for. You know, King Midas wished for everything he touched to turn to gold. That was the objective he gave to the machine, which happened to be the gods, and the gods gave him his objective. And then that was his food and his drink, and his family all turned to gold, and then he dies in misery. And so we've known this for thousands of years, and yet we built the field of AI around this definition of machines that carry out plans to achieve objectives that we put into
8:22 Speaker 01 them. And it only works if and only if we are able to completely, perfectly specify the objective. So the guidance is, "Don't put fixed objectives into machines, but build machines in a way that acknowledges the uncertainty about what the true objective is." For example, take a very simple machine learning task, learning to label objects in images. So what should the objective be? Well, you go and talk to a roomful of computer vision people, they will say, "Labeling accuracy." And that's actually the metric used for all these competitions. In fact, this is the wrong metric.
9:05 Frank Chen? How so?
9:06 Speaker 01 Because different kinds of misclassifications have different costs in the real world.
9:12 Speaker 01 Misclassifying one type of Yorkshire terrier as a different type of Yorkshire terrier is not that serious. Classifying a person as a gorilla is really serious. And Google found that out when their computer vision system did exactly that. And it probably cost them billions in goodwill and public relations. And that opened up, actually, a whole series of people observing the ways that these online systems were basically misbehaving in the way they classified people. If you do a search on Google Images for CEO, I think it was one of the women's magazines pointed out
9:54 Speaker 01 that the first female CEO appears on the 12th row of photographs and turns out to be Barbie.
10:01 Frank Chen? So if accuracy isn't the right metric, what are the design paths that you're suggesting we optimize for?
10:07 Speaker 01 If you're going to have that image labeling system take action in the real world, and posting a label on the web is an action in the real world, then you have to ask, what's the cost of misclassification? And when you think, okay, so ImageNet has 20,000 categories, and so there are 400 million, so 20,000 squared, different ways of misclassifying one object as another. So now you've got 400 million unknown costs. Obviously, you can't specify a joint distribution of 400 million numbers, one by one. It's far too big. So you might have some general guidelines that misclassifying one type of flower as another is not very expensive.
10:49 Speaker 01 Misclassifying a person as an inanimate object, those are going to be more expensive. But generally speaking, you have to operate under uncertainty about what the costs are. And then, how does the algorithm work? One of the things it should do, actually, is refuse to classify certain photographs, saying, I'm not sure enough about what the cost of misclassification might be. So I'm not going to classify.
11:14 Frank Chen? So that's definitely a divergence from state of the art today. State of the art today is you're going to assign some class to it. That's a dog, or Yorkshire terrier, or a pedestrian, or a tree. And then the algorithms can say, like, I'm really sure, or I'm not really sure. And then a human decides. You're saying something different. Which is, I don't understand the cost of uncertainty. So therefore, I'm not even going to give you a classification or a confidence interval on the classification. Like, I shouldn't. It's irresponsible for me to do.
11:43 Speaker 01 So I could give confidence intervals and probabilities, but that wouldn't be what image labeling systems typically do. They're expected to plump for one label. And the argument would be if you don't know the costs of plumping for one label or another, then you probably shouldn't be plumping. Right? And I read that Google Photos won't label gorillas anymore. So you can give it a picture that's perfectly obviously a gorilla, and it'll say, "I'm not sure what I'm seeing here."
12:13 Frank Chen? And so how do we make progress on designing systems that can factor in this context? Sort of understanding the uncertainty, characterizing the uncertainty?
12:22 Speaker 01 So there's sort of two parts to it. One is, how does the machine behave, given that it's going to have radical levels of uncertainty about many aspects of our preference structure? And then the second question is, how does it learn more about our preference structure? As soon as the robot believes that it has absolute certainty about the objective, it no longer has a reason to ask permission. And in fact, if it believes that the human is even slightly irrational, which of course we are, then it would resist any attempt by the human to interfere or to switch it off. Because the only consequence of human interference
13:03 Speaker 01 in that case would be a lower degree of achievement of the objective. So you get this behavior where a machine with a fixed objective will disable its own off switch to prevent interference with what it is sure is the correct way to go forward.
13:20 Frank Chen? Yeah. And so we'd want a very high threshold on confidence that it's understood what my real preference or desire is.
13:30 Speaker 01 Well, I actually think it's, in general, not going to be possible for the machine to have high confidence that it's understood your entire preference structure. It may understand aspects of it. And if it can satisfy those aspects... Without messing with the other parts of the world that it doesn't know what you want. Then that's good, but you know, there are always going to be things that it never occurs to you to to write down.
13:54 Frank Chen? So I can see how this design approach would lead to much safer systems, right? Because you have to factor in the uncertainty. I can also imagine sort of a practitioner today, sitting in their seat going, wow, like, that is so complex. I don't know how to make progress. So what do you say to somebody who's now thinking, wow, I thought my problem was X hard, but it's really 10x or 100x or 1000x hard?
14:17 Speaker 01 So interestingly, the the safe behaviors fall out as solutions of of a mathematical game. With a robot and a human, in some sense, they're cooperating because they both have the same objective, which is whatever it is the human wants. It's just that the robot doesn't know what that is. So if you formulate that as a mathematical game and you solve it, then the solution exhibits these desirable characteristics that you want, namely deferring to the human, allowing yourself to be switched off, asking permission, only doing minimally invasive things. We've seen, for example, in the context of self-driving cars,
14:59 Speaker 01 that when you formulate things this way, the car actually invents for itself protocols for behaving in traffic that are quite helpful. For example, one of the constant problems with self-driving cars is how they behave at four-way stop signs. Because they're never quite sure who's going to go first and they don't want to, you know, they don't want to cause an accident.
15:22 Frank Chen? They're optimized for safety, so they'll end up stuck at that four-way intersection.
15:25 Speaker 01 So they're stuck and everyone, everyone ends up pulling around them. And and it's actually, you know, will probably cause accidents rather than reducing accidents. So what the algorithm figured out was that if it got to the stop sign and it was unclear who should go first, it would back up a little bit. And that's a way of signaling to the other driver that it has no intention of going first and therefore they should go. And that falls out as a solution of of this game theoretic design for the problem.
15:54 Frank Chen? Let's go to another area where machine learning is often being used. I'm about to make a loan, right, to an individual. And so they take in all this data, they figure out your creditworthiness and they say like loan or not, right? How would sort of game theory inside loan decision making be different than traditional methods?
16:12 Speaker 01 So what happens with traditional methods is that they make decisions based on past data. And a lot of that past data reflects biases that are inherent in the way society works. So if you just look at historical data, you might end up making decisions that discriminate, in effect, against groups that have previously been discriminated against. Because that prior discrimination resulted in lower loan performance. And so you end up actually just perpetuating the negative consequences of social biases. So loan underwriting in particular has to be inspectable. And regulators have to be able to verify that you're making decisions on criteria
16:57 Speaker 01 that neither mention race or that are not proxies for race. So the principles of those regulations need to be expanded to a lot of other areas. For example, data seems to be suggesting that the job ads that people see online are extremely biased by race. If you're just trying to fit historical data and maximize predictive accuracy, you're missing out these other objectives about fairness at the individual level and the social level. So economists call this the problem of externality. And so pollution is a classic example, right? A company can make more money by just dumping pollution into rivers and oceans and the atmosphere,
17:40 Speaker 01 rather than treating it or changing its processes to generate less pollution. So it's imposing costs on everybody else. The way you fix that is by fines or tax penalties.
17:51 Frank Chen? You create a price for something that doesn't have a price, right?
17:55 Speaker 01 Right. Now, the difficulty, and this is also true with the way social media content selection algorithms have worked, it's, I think, very hard to put a price on this. And so...
18:08 Speaker 01 You know, the regulators dealing with loan underwriting have not put a price on it. They put a rule on it.
18:13 Frank Chen? Right.
18:14 Speaker 01 Saying you cannot do things that way.
18:16 Frank Chen? So let's take, you know, making a recommendation at an e-commerce site for here's a product that you might like. Like, how would we do that differently by baking game theory?
18:26 Speaker 01 So the primary issue with recommendations is understanding user preferences. One of the problems I remember with a company that sends you a coupon to buy a vacuum cleaner and you buy a vacuum cleaner. Great. So now it knows you really like vacuum cleaners. It keeps sending you coupons for vacuum cleaners. But of course, you just bought a vacuum cleaner. So you've no interest in getting another vacuum cleaner.
18:48 Frank Chen? You're not collecting vacuum cleaners.
18:50 Speaker 01 Right. You know, so just this distinction between consumable things and non-consumable things is really important when you want to make recommendations. And I think you you need to come to the problem for an individual user with a reasonably rich prior set of beliefs about what that what that user might like, based on demographic characteristics. How do you then adapt that and update it with respect to the decisions that the user makes about, you know, what products to look at, which coupons they cash in, which ones they don't, and so on. And one of the things that you might see falling out would be that the the recommendation system, it might actually ask you a question.
19:35 Speaker 01 I've noticed that you've showed no interest in all of these kinds of projects. Are you in fact a vegetarian?
19:41 Frank Chen? As you look back at your own career in this space, are you surprised that the field is where it is?
19:46 Speaker 01 You know, 10 years ago, I would have been surprised to see that speech recognition is now just a commodity that everyone is using on their cell phones across the entire world.
19:59 Frank Chen? Yeah. When I was in undergrad, they said, we definitely have to solve the Turing test before we're going to get speaker independent natural language.
20:05 Speaker 01 And, you know, I worked on self-driving cars in the, in the early 90s. And it was pretty clear that the perception capabilities were the real bottleneck. The system would detect about 99% of the other cars. So every 100th car, you just wouldn't see it. So these are things that are coming true, right? They, they were sort of holy grails. It's interesting that even though they achieved superhuman performance on these testbed data sets, there are still these adversarial examples that show that actually it's not seeing things the same way that humans are seeing them.
20:40 Frank Chen? Definitely making different mistakes than we make.
20:42 Speaker 01 And so it's, it's fragile in ways that we don't understand. For example, OpenAI has a system with simulated humanoid robots that learn to play soccer. One learns to be the goalkeeper, the other one learns to take penalties. And it looks great. You know, so this was a big success. And he did, he basically said, okay, can we get adversarial behavior from the goalkeeper? So the goalkeeper basically falls down on the ground immediately, waggles its leg in the air and the, the penalty taker, the one who's kicking the ball, just completely falls apart, right?
21:17 Frank Chen? I don't know how to respond to that.
21:18 Speaker 01 And never actually gets around to kicking the ball at all. I don't know whether he's laughing, so, ah, you can't kick the ball or what. But, you know, so it's not that just because we have superhuman performance on some nicely curated data set, that we actually have superhuman vision or superhuman motor control learning.
21:35 Frank Chen? Are you optimistic about the direction for the field?
21:38 Speaker 01 So one reason I'm optimistic is that as we see more and more of these failures of the standard model, people will say, oh, well, you know, clearly we need to build these systems this other way because that's sort of, you know, gives us guarantees that it will. It won't do anything rash, it'll ask permission, it will adapt to the user gradually, and it'll only start taking bigger steps when it's reasonably sure that that's what the user wants. I think there are reasons for pessimism as well in misuse for surveillance misinformation. I mean, there's more awareness of it, but there's nothing concrete being done about that with a few honorable exceptions, like San Francisco's ban on face recognition in
22:20 Speaker 01 public spaces, and California's ban actually on impersonation of humans by AI systems.
22:27 Frank Chen? The deep fakes.
22:28 Speaker 01 Not just deep fakes, but for example, robo calls.
22:32 Frank Chen? Right, I'm pretending to schedule a haircut appointment, and I didn't self identify as an AI.
22:37 Speaker 01 Right, so Google has now said that they're going to have their machines self identify as an AI. You know, so it's a relatively simple thing to comply with, it doesn't have any great economic cost, but I think it's a really important step that should be rolled out globally. That principle of not impersonating human beings is a fundamentally important principle. Another really important principle is don't make machines that decide to kill people.
23:03 Frank Chen? Ban on offensive weapons.
23:06 Speaker 01 Sounds pretty straightforward, but again, although there's much greater awareness of this, there are no concrete steps being taken, and countries are now moving ahead with this technology. So I just last week found out that a Turkish defense company is selling an autonomous quadcopter with a kilogram of explosive that uses face recognition and tracking of humans, and is sold as an anti-personnel weapon. So we made a movie called Slaughterbots to illustrate this concept. We've had more than 75 million views.
23:48 Frank Chen? So to bring this home to people who are sitting at their desks working on machine learning systems, if you could give them a piece of advice on what they should be doing, like what should they be doing differently having heard this podcast that they might not have been thinking?
24:01 Speaker 01 So for some applications, it probably isn't going to change very much. One of my favorite applications of machine learning and computer vision is the Japanese cucumber farmer.
24:11 Frank Chen? Oh, right.
24:12 Speaker 01 Who downloaded some software and trained a system to pick out bad cucumbers from his.
24:19 Frank Chen? And sorts them into the grades. The Japanese are very fastidious about the grades of produce, and he did it so inexpensively.
24:25 Speaker 01 Yeah, so that's a nice example. And it's not clear to me that there's any particular way you might change that because it's.
24:33 Frank Chen? No game theory really needed for it.
24:35 Speaker 01 It's a very, I mean, in some sense, it's a system that has a very, very limited scope of action, which is just to, you know, sort cucumbers. The sorting is not public, and there's no danger that he's going to label a cucumber as a person or anything like that. But in general, you want to think about, first of all, what is the effect of the system that I'm building on the world, right? And it's not just that it accurately classifies cucumbers or photographs. It's that, you know, of course, people will buy the cucumbers or people will see the photographs. And what effect does it have? And so often, when you're defining these objectives for a machine learning algorithm, they're going to leave out effects
25:20 Speaker 01 that the algorithm, that the resulting algorithm is going to have on the real world. And so can you fold those other effects back into the objective rather than just optimizing some narrow subset, like click through, for example, which could have extremely bad external effects? So the model, if you sort of want to kind of anthropomorphize model, you would rather have the perfect butler than the genie in the lamp.
25:51 Frank Chen? The all-powerful, kind of unpredictable genie.
25:53 Speaker 01 Right, and very literal minded about this is the objective.
25:56 Frank Chen? Right, awesome. Well, Stuart, thanks so much for joining us on the E16Z podcast.
26:01 Speaker 01 Okay, thank you, Frank. It's been a delight.
Previous On Pharma Trends and Big Company Innovation Next The Career Changing Magic of Strategic Compartmentalization Page updated September 30, 2026 · transcript and summary are machine-generated