Is A.I. the Problem? Or Are We?

If you talk to many of the people working on the cutting edge of artificial intelligence research, you’ll hear that we are on the cusp of a technology that will be far more transformative than simply computers and the internet, one that could bring about a new industrial revolution and usher in a utopia — or perhaps pose the greatest threat in our species’s history. Others, of course, will tell you those folks are nuts. One of my projects this year is to get a better handle on this debate. A. I., after all, isn’t some force only future human beings will face. It’s here now, deciding what advertisements are served to us online, how bail is set after we commit crimes and whether our jobs will exist in a couple of years. It is both shaped by and reshaping politics, economics and society. It’s worth understanding. Brian Christian’s recent book “The Alignment Problem” is the best book on the key technical and moral questions of A. I. that I’ve read. At its center is the term from which the book gets its name. “Alignment problem” originated in economics as a way to describe the fact that the systems and incentives we create often fail to align with our goals. And that’s a central worry with A. I., too: that we will create something to help us that will instead harm us, in part because we didn’t understand how it really worked or what we had actually asked it to do. So this conversation is about the various alignment problems associated with A. I. We discuss what machine learning is and how it works, how governments and corporations are using it right now, what it has taught us about human learning, the ethics of how humans should treat sentient robots, the all-important question of how A. I. developers plan to make profits, what kinds of regulatory structures are possible when we’re dealing with algorithms we don’t really understand, the way A. I. reflects and then supercharges the inequities that exist in our society, the saddest Super Mario Bros. game I’ve ever heard of, why the problem of automation isn’t so much job loss as dignity loss and much more. Mentioned: “Human-level control through deep reinforcement learning” “Some Moral and Technical Consequences of Automation” by Norbert Wiener Recommendations: What to Expect When You're Expecting Robots by Julie Shah and Laura Major Finite and Infinite Games by James P. Carse How to Do Nothing by Jenny Odell If you enjoyed this episode, check out my conversation with Alison Gopnik on what we can all learn from studying the minds of children. You can find transcripts (posted midday) and more episodes of "The Ezra Klein Show" at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Thoughts? Guest suggestions? Email us at [email protected]. “The Ezra Klein Show” is produced by Annie Galvin, Jeff Geld and Rogé Karma; fact-checking by Michelle Harris; original music by Isaac Jones; mixing by Jeff Geld; audience strategy by Shannon Busta. Special thanks to Kristin Lin.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Ezra Klein?

  I'm Ezra Klein and this is the Ezra Klein Show.

• 0:17Ezra Klein?

  So before we get started, a bit of housekeeping promotional kind of thing. There is a subreddit, a page on Reddit for this show. I'm not involved in it. We don't control it or moderate it or anything, but there are a couple thousand people who like talking about the show over on Reddit. It's at reddit.com/r/ezercline. And they've asked me to do and ask me anything next Friday, the 11th. And I thought I would tell people this is happening in case they want to go participate in it. It'll be sometime in the morning Pacific time for me, but if you want to head over there, there's going to be an opportunity to ask questions and I'll come for a couple hours and answer them. So that is at reddit.com/r/ezercline.

• 0:56Ezra Klein?

  So one of the strange things about living in the Bay Area is you'll be going about your business, talking to people about politics and parenthood and how nobody can afford a house. And all of a sudden you'll run into someone who works in artificial intelligence and you'll realize the world and what is important in it and what is coming in it looks entirely different to them. Like they are living across a chasm in expectations from you. That to them, we are on the cusp of a technology that will be more transformational than simply computers and the internet. They'll be on the level of an industrial revolution. It maybe could create a utopia, could create a dystopia, could end humanity entirely. And you can dismiss it. Some people do. Sometimes I do. I definitely have the impulse to.

• 1:41Ezra Klein?

  But these are smart people. I know them. And one thing that gives me pause is they are inside something I am outside of. They are seeing inside a technological revolution that is closed to most of us. They're seeing how fast a technology is moving when, you know, I mean, I try to follow it and I have no idea. And so one of the things I've resolved to do this year is get a better handle on what I think of artificial intelligence, what I think it's going to do to the economy, to society, to our politics, what I think the political approach to it should be. And that means first and foremost, just understanding what it is, what it's doing right now. There's an important distinction to make here. AI is not just Skynet. It doesn't mean sentience. It doesn't mean superintelligence.

• 2:25Ezra Klein?

  To be honest, it's not clear what it means. There's an old joke in computer science circles that artificial intelligence is anything a computer can't do yet. So like before, computers can beat humans in chess. Well, that would be artificial intelligence. Afterwards, well, that's just machine learning. Chess is a game. It has rules. You can just tell the computer the rules. So there's always an expanding frontier over which, well, that is true intelligence and this is just machine learning. But broadly, we're talking about machines that can learn and act more autonomously. And this is not a far off thing. We're using them now for all kinds of things, from what ads Facebook serves you, to where your bail is set after getting arrested for a crime. It is affecting your life, my life now. It is reshaping our economy and society now. And it's moving really,

• 3:09Ezra Klein?

  really fast. And even a lot of the people running these systems, they don't fully understand what they're doing. And that's to say nothing of the politicians and regulators who are supposed to be governing them. So let's get started. There are going to be a few episodes around this theme in the coming months, but Brian Christian is where I wanted to begin. He's the author of the book, The Alignment Problem. There are a lot of good books out there on AI right now, a couple of them from colleagues of mine at the New York Times. But Christians, in my view, is the best book on the technical questions of machine learning, written for a general audience. It's a very, very, very deep work on how machine learning works. And it also ends up being a pretty deep look into how human learning works and the very fraught relationship between the two, because that's the fear at the core of all this. The problems and the possibilities

• 3:51Ezra Klein?

  of AI are in a very deep way. The problems and possibilities of humanity, they are generated by us. The fear is that it will learn the worst of us, and it will take our mistakes and our dark impulses and reorder society around them. And it will do so for the profits of a few. That's a really important part of this conversation that often gets missed. The business models of AI, the political economy of AI, it really, really matters. So I asked Christian to join me for the first of our AI episodes to talk about it. As always, my email is [email protected]. Here is Brian Christian.

• 4:29Ezra Klein?

  I want to start with the concept that gives your book its name, the just the very idea of an alignment problem. And I was interested to learn from you that it comes from from economics, from commentary about capitalism. I actually think it's a useful place to start before we get to the whole artificial intelligence side of it. So tell me about the history of alignment problems.

• 4:47Speaker 01

  Yeah, so the term alignment gets borrowed in 2014 by the computer science community from the economics literature. So going back to the 90s and 80s, economists were talking about, you know, how do you make a value-aligned organization where everyone's pulling in the same direction? Or how do you align someone's, you know, the subordinate's incentives with what the manager wants them to do? And obviously, there's a huge literature on how this can go horribly wrong. And, you know, it also connects to the parenting literature. Every parent has had these experiences of the example I love is that University of Toronto economist Joshua Gans decided to give his older child, I think, a dollar

• 5:28Speaker 01

  or a piece of candy, I forget, every time they helped the younger child use the toilet. And he later discovered that the older child was force-feeding water to the younger sibling so that they could use the toilet more times per day. And so, you know, I think this just points to how fundamentally human this problem of incentives really is. And so, this idea of how do you capture in a system of rewards, or incentives, or goals, or KPIs, or indices, what you really want in a way that's not going to lead itself to some loophole, or some kind of horrible side effect that you didn't intend? That is a much bigger human story than merely the history of AI.

• 6:12Ezra Klein?

  So, there is a moth to the flame of superintelligent AI that will kill us all dynamic in this conversation. People like to sort of throw the ball way down the field and imagine Skynet. But a lot of what is happening right now is we're building machine learning into things we currently do and predicting whether or not a violent offender will reoffend. And so what kind of bail should they get or what kind of parole should they get? Or, you know, deciding whether or not somebody's going to be a good fit for a job. And so can you talk a bit about the ways that the problems you're looking at, the problems of aligning what we want machines to do and what they actually do are operating in the here and now, not just the far-flung future?

• 6:52Speaker 01

  Yeah, absolutely. So, I mean, this, this question of, you know, are these systems actually doing what we want has this long history. It goes back to 1960. The MIT cyberneticist Norbert Wiener has this famous quote where he's talking about kind of the sorcerer's apprentice. We all know it as the lovable Mickey Mouse cartoon where he tells this broom to fill up a cauldron with water and then ends up almost drowning. Wiener has this quote where he says, you know, this is not the stuff of fairy tales. Like this is coming for us. And the famous quote is, "If we build a machine to achieve our purposes with which we cannot interfere once we've started it, then we had better be quite sure that the purpose we put into the machine is the thing we really desire."

• 7:36Speaker 01

  And, you know, this has continued through the early 20th century as the thought experiment of the paperclip maximizer that turns the universe into paperclips, killing everyone in the process. But to your point, I don't think we need these thought experiments anymore. We're now living with these alignment problems every day. So one example is there's a facial recognition data set called labeled faces in the wild. And it was collected by scraping newspaper articles off the web and using the images that came with the articles. Later, this data set was analyzed and it was found that the most prevalent individuals

• 8:17Speaker 01

  in the data set were the people who appeared in newspaper articles in the late 2000s. And so you get issues like there are twice as many pictures of George W. Bush as of all black women combined. And so if you train a model on that data set, you think you're building facial recognition, but you're actually building George W. Bush recognition. And so this is going to have totally unpredictable behavior. Similar thing in criminal justice. You may think you're building a risk assessment system to tell you whether someone will reoffend or recidivate. Right. But you can't actually measure crime. You can only measure whether people were arrested and convicted. And so you haven't built a crime predictor.

• 9:02Speaker 01

  You've built an arrest predictor, which in 21st century America is a very different thing. And so there are many cases like this, large and small. You see the same thing happening in the research community. You think you're trying to build a program that can win a boat race and you use as this proxy incentive, get as many points in this video game as you can. But maybe getting the most points involves just doing donuts in this little harbor and collecting these little power ups forever. And so each of these is in its own way, an example of this alignment problem. It turns out to be really hard to actually get the system to internalize the goals and the behavior that you actually have in mind.

• 9:45Ezra Klein?

  Can you talk through the story of Amazon trying to build this into their recruiting efforts?

• 9:50Speaker 01

  Yeah, so Amazon was building a machine learning tool to help them prioritize which resumes should get filtered through when they made a job opening. And famously, they were rating their job candidates on a scale of one to five stars, just like Amazon users rate their products. And in order to do this, they were using what's called an unsupervised language model. But the basic idea is it looks at resumes for candidates that were hired in the past. And it says, what are the terms that tended to appear on the CVs and resumes of people who we'd hired before? And so we'll just kind of upvote the resumes that look like that.

• 10:32Speaker 01

  The problem was that, you know, by default, this is going to perpetuate any kind of, you know, bias or prejudice that existed. So if your engineering department was mostly male, then you're going to discover, as the Amazon engineers did, that it is penalizing resumes that have the word women's in them. So if you played women's soccer, or if you attended a women's college, or whatever it might be, the system says, "That doesn't look like the kind of language that has appeared on resumes of people we've hired in the past. Therefore, we don't think you should hire this person in the future." And, you know, they ended up penalizing, you know, terms like "field hockey," or "sewing,"

• 11:13Speaker 01

  or things like that, that all is kind of skewed towards female applicants. And even it went so far as identifying idioms that were more typical of male engineering candidates, like the use of the word "executed." So it was giving you bonus points if you used this kind of martial speech in your resume. Eventually, they essentially decided to scrap the project, that it was this kind of irredeemable.

• 11:40Ezra Klein?

  So, one very tricky, quasi-philosophical issue here is whether or not this is in fact an alignment problem at all. So what you're basically saying is what happens is you turn these algorithms loose, and they replicate the way our society actually looks. They look at who Amazon hires, and then they learn based on that hiring process, and they say, "Here's who you're probably going to like to hire." And Amazon says, "Oh, no, no, no, not us. That isn't how we wanted to hire at all." But on some level, it is how they were hiring. And maybe somebody who's less politically correct, you know, to use that language, might say, "No, the machine had it right, and Amazon just doesn't want to admit what it actually does and what actually works for it." So these aren't alignment

• 12:21Ezra Klein?

  problems. These are, we're turning almost too powerful a spotlight on the way our society, you know, really functions. But that's not a problem of the machines.

• 12:32Speaker 01

  You know, there's this adage that is famous in statistical circles, and it says, "All models are wrong, but some are useful." And I think part of the danger that we have at the current moment is that Our models are wrong in the way that all models are wrong, but we have given them the power to enforce the limits of their understanding on the world. So one example here being the self-driving Uber that killed a pedestrian in 2018. If you read the National Transportation Safety Board report, you discover a number of interesting things. One was that apparently there was no training data of jaywalkers. It was only ever expecting to encounter someone

• 13:16Speaker 01

  at an intersection or a crosswalk. And so it just wasn't prepared for how to deal with someone crossing in the middle of the street. The other thing you find is that it was using this kind of classic machine learning thing of every object belongs into exactly one out of N categories. And this particular woman was walking a bicycle across the street. And so the object recognition system couldn't make its mind up. It said, okay, well, I can see her walking. No, there's definitely a bike frame. I'm seeing the tires. I'm seeing this little triangular support thing. She's definitely a cyclist. No, she's, we can see her walking on the ground. And each time it changed its mind, it had to kind of recompute from scratch, whether it thought it was going to hit her or not. And that's part of what led to the accident.

• 14:01Speaker 01

  So this comes back to this idea that all models are wrong, but we've now, in some cases, given them the ability to use effectively lethal force to ensure that the world conforms to their, you know, simplified preconception of what the world is. You know, if you live in a world where you don't think jaywalkers exist and you can only be a cyclist or pedestrian and you kill anyone who doesn't fit into that conceptual scheme, then the world does in fact come to resemble the model that you have in your head, but you know, not, not in the good way.

• 14:33Ezra Klein?

  So what are we trying to achieve here? Before we get to the question of how do you handle the problem of machine learning algorithms creating disasters that we didn't intend? Like, what is the promise of these algorithms? They've been around for a little while now. We don't have supercharged economic growth. We've not cured cancer. Like, why is there so much promise and investment in this field?

• 15:01Speaker 01

  I think there's Broadly, this idea that we can deploy human-level expertise or intelligence at scale, you know, for kind of zero marginal cost. I think that's, that's the broad idea that if we, you know, not everyone can afford to go to a world-class dermatopathologist to decide whether that discoloration on their shoulder needs to be looked at or not. But if you could train a model that's as good as the best dermatopathologist in the world, then everyone with a smartphone could have access to that level of skill or that level of kind of insight.

• 15:46Speaker 01

  I think that's the broad story in terms of why this technology is so attractive. There's a well-intentioned societal motive behind some of this. A lot of the ways that hiring has been done in the past is just sort of through the social network of the people that already exist at the company. But that privileges people with certain demographic attributes. They live near the other people or they're in the same kind of economic class, et cetera, et cetera. And so there's this meritocratic idea of, no, let's create a job posting and anyone in the world can apply. But now you have to filter the candidates at a scale that you didn't have to deal with before. And so what do you do? Well, often they turn to machine learning. And, you know, as, as we've discussed,

• 16:31Speaker 01

  you know, there's these kind of predictably terrible outcomes often. But I think that's the idea in terms of your question. What are we really trying to do?

• 17:00Ezra Klein?

  So if people are following the AI conversation at all, they maybe know there, there are a couple of big companies doing this, you know? People hear about Google's DeepMind. They hear about OpenAI. There's obviously a lot of work at Tessa. And then, of course, also at Uber on how to do driverless, you know, cars. It's not going to be the case that every small hospital system develops its own remarkable AI player so it can rent out, you know, its AI dermatology department. So is what's happening here that all a bunch of these different institutions are trying to create the AI interface that other people are going to rent out for their projects? Is what's happening here that is just going to be a few people have it early on and then other people develop it?

• 17:44Ezra Klein?

  Like when when we talk about this, who's going to control this resource and how do others get access to it?

• 17:50Speaker 01

  I think this question really depends on the level of sophistication of the particular tool. So on the more spreadsheet end of the spectrum, I think, for example, if you look at criminal justice, you know, pre-trial risk assessment algorithms, many jurisdictions have rolled their own. So Minneapolis rolled their own risk assessment in the 90s and made a note to themselves to audit it a year later, but then forgot. And then it was in operation for about 15 years before they even thought to check whether it was making accurate predictions or not. So those sorts of things happen. On the other hand, yeah, to your point, there is a certain kind of economy of scale.

• 18:33Speaker 01

  At the moment, a lot of the most performant models are big. They take tens of billions of dollars to train and so that's a certain barrier to entry. I think this is a really interesting question because if you look at the academic literature on AI safety, it's kind of premised on this almost hobbyist relationship of there's a human called H who wants to do something, so he goes to his garage and builds a robot called R. Is R going to do what H wants? And maybe that's a useful way for framing some of the, you know, actual math, but I don't think that's a useful way for thinking about the actual relationship we're going to have with something like, you know, advanced AI. I think it's more likely that it's going to be like, you know, the OpenAI API or like

• 19:18Speaker 01

  the, you know, user agreement that we have with iOS or Android where we never really own anything. We're just kind of subject to the terms and conditions that can change at any moment. So I think. That is much more likely, from my perspective, to be the kind of relationship that we have, especially early on.

• 19:38Ezra Klein?

  But you spent a lot of time with these different companies. I mean, you read through this book. It is a festival of like this researcher at DeepMind and this researcher at OpenAI and this researcher at Microsoft and so on. What are they all competing to do? Like, what are they, what is their implied long-term business model after they spend billions of dollars winning this race to create some kind of general, or at least very, very powerful AI?

• 20:04Speaker 01

  Well, that's a very interesting question because if you look at the kind of the narrative that's being told, and this is, yeah, so just to use DeepMind and OpenAI as an example, they both tell a story that's something like, we're going to solve intelligence and then everything will follow. You know, once we sort that out, then we can cure cancer, we can solve world hunger, you name it. And it remains to be seen how the development of AI, and in particular, AI safety, would handle something like a kind of late business cycle, you know, Revenge of the Bean counters, where they say like, okay, and why are we spending, you know, $40 million on this like generic thing that plays chess all day long?

• 20:49Speaker 01

  So, you know, it does explain why you get some of these contractions and I am personally very, very interested to see whether safety research stays as robust in the next five years as it has been in the last five years. But that's the idea. I mean, I think at some level, they're waiting to find the business model down the line.

• 21:15Ezra Klein?

  Well, that worries me, to be honest. And I want to hold here for a minute because it's something your book gestures at, but I always think needs to be a little bit of a deeper conversation here because you've been talking about AI safety. And for people who aren't fully into that term, that's around this alignment problem. Sometimes it's like, you know, long-term, will the AI kill us all? Short-term, will it do the things we want it to do? But you have a good line at one point where you're imagining just AI that sort of follows us around and helps us make better decisions in the things we need to make decisions on. And you write, These computational helpers of the near future, whether they appear in digital or robotic form, likely both, will almost without exception have conflicts of interest, the servants of two masters, their ostensible owner and whatever organization created them. In this sense,

• 22:00Ezra Klein?

  they'll be like butlers who are paid on commission. They will never help us without at least implicitly wanting something in return. And so I imagined Google building, you know, through DeepMind, like the winner of the AI race. And I know how Google works. And everywhere I go on the internet, Google is serving me ads that are built on my own personal data and that are trying to get me to either complete a purchase I've begun or seem interested in, or trying to get me to make a purchase it think is adjacent to something that I would like. If you had much, much, much, much smarter AI that was much more integrated into my life, that was built on an advertising model, you know, you're kind of entering a pretty nerve-wracking space of personal manipulation.

• 22:45Ezra Klein?

  And I don't hear this honestly talked about as much in the AI safety conversation. It's like much more like, we're going to create this amazing thing, but what if it goes wrong? More than, we're going to create this amazing thing, and what if we do wrong?

• 22:56Speaker 01

  Yeah, and I think this is extremely central. And it goes back to your highlighting of kind of the term alignment was originally an economic term, that, you know, Google can build an AI that's aligned with the interests of Google Corporation, that may not be aligned with the interests of the end users or, you know, third parties that are affected without even using the software, et cetera, et cetera. There's this question of what replaces like the right-hand margin? Like, for those of us that remember, you know, the internet of the early 2000s, right? There was this right-hand margin on every website that was full of ads. But you move from that to an oral interface where you're talking to Alexa, or you're talking to your smart home

• 23:36Speaker 01

  speaker. What is the equivalent of this kind of right margin of space that we can fill? Is it that you ask, you know, Alexa, what's the temperature? And it says, oh, it's 72. By the way, I thought you might be interested in, you know, tonight, there's a new show premiering, I don't know that people have the patience for that. And so there's this question of is the advertising model going to survive the move towards these digital assistants? Is it going to have to be replaced with essentially like product placement slash commission driven model where you say to your smart robot of the future, like just get me some toilet paper.

• 24:18Speaker 01

  And behind the scenes, there's been a huge bidding process for which toilet paper it's going to get you. And at some level, maybe you care, maybe you don't. That's the kind of thing where I think a lot of the action is going to start to move to that. And so, yeah, you can really ask this question of is this interface that I'm using actually working for me when I tell it what I want? Probably mostly not, right? It's probably mostly worried about, you know, which kind of toilet paper is giving the best commission that week or whatever it might be.

• 24:51Ezra Klein?

  Or something deeper than that. I mean, I'm very interested in this question of the alignment problem between the the end user and the owner, right? With the the machine in the middle. Because another possibility here is geopolitical. So we've been in a debate over the past year or two in this country over TikTok, which is this remarkable social networking app that is owned by a Chinese company ByteDance. And there's been a lot of worries that, you know, maybe TikTok is is spyware or what is it really doing? But but something very essential about TikTok is it its underlying algorithm is amazing. You know, if you look at the analysis of why TikTok does so well, its ability to intuit what you like through machine learning and feed it to you, it's absolutely best of class. It needs a lot less information about you than a Facebooker or or some of these other players do.

• 25:35Ezra Klein?

  Now you imagine, I mean, China's making tremendous, tremendous investments in AI. Now you imagine that some of those actually pay off. You build some of that into TikTok. And just on the margin, they're trying to make you like China more, which maybe is not even the worst thing. We've had propaganda efforts in this country forever, trying to make people like America more. Why not use your algorithm on your free, you know, video app to serve up things that, you know, improve cross cultural communication? But over time, this stuff becomes really, really, really out of alignment and it is even hard to know if it's going on. And I just don't really know, I guess, what we do about it.

• 26:16Speaker 01

  Yeah, the question of what we do about it. I mean, there's two halves to your question, I guess. One is kind of what's the end game of this? I am very curious to see whether, for example, Twitter and Reddit in particular, these kind of quasi anonymous, text-based discussion forums, whether they can actually survive the ability to produce like site-specific propaganda at scale. You know, so we're now entering this era of large language models, things like BERT and GPT-3, et cetera, that can produce, you know, hand-tailored responses to a given comment thread that, you know, wittily reference the previous comments,

• 27:01Speaker 01

  but maybe have a slight 5% positive skew about, you know, pick your political party. I really don't know how the idea of like anonymous discourse survives. And so that, I think, is a really open question. What do we do about it? I think partly there is a transparency issue, and I don't, I don't know what the regulatory framework is going to be. But one of the things that I always want to know is what is the objective function of the company? What is being optimized for? And ideally, you'd want to have some agency over that. You know, I mean, one of the things I like about Reddit

• 27:44Speaker 01

  is that there is this little dropdown where you can say, "Show me the most recent things, show me the most upvoted things, show me the most controversial things." And, you know, that's only a couple of degrees of freedom, but it's something. You feel like you have your hand on the wheel to some degree. Whereas when I use other social networks, I'm very aware of the fact that my behavior is sending some kind of training signal back to the mothership, but it's really not clear what that relationship actually looks like. And so I don't know. For me, transparency is a starting point. Just what is it you were trying to do in the first place? Like, what's, what is driving the recommendations that are being sent?

• 28:25Ezra Klein?

  Well, let's take transparency at two levels, because the transparency of these algorithms and transparency of what these machines are actually doing when they learn is a huge part of your book. But before we even get to that question of do we know what is happening in them, there is a question of Do we have the right to know what is happening in them? I can't pop the hood on the Facebook news feed.

• 28:45Speaker 01

  Yeah.

• 28:45Ezra Klein?

  They don't give me that option. And they would say, and it's not a crazy thing to say, the Facebook news feed is our comparative advantage. The algorithm that feeds us is something we spent however much money on. You can't make us turn that over to the public. Do we need to think of algorithms in some kind of different class than we have thought of a lot of traditional forms of IP?

• 29:07Speaker 01

  Yeah, I mean, in some ways, it reminds me a little bit more of financial regulation, where you have to deliberately make the regulation very vague, because by the time the bill is actually passed, you know, the technology has moved on. And so the only hope you have of any meaningful regulation is to kind of do it on the fly tactically. I mean, that that may be the case here. I, you know, one of the other things Facebook may say is like, well, we can try to provide you transparency into our algorithm, but we change the algorithm, you know, 20 times a day, and we're constantly A B testing 1,000 different variations on any given user at any given moment. So what do you mean by the Facebook algorithm? You know, you might have transparency one minute and then you refresh the page and now it's a completely different process affecting your results.

• 29:51Speaker 01

  So I think there are huge questions and I I don't claim to have any idea what the regulatory foothold is here. But I do think from the scientific side, there's been a lot of progress on the idea that you can actually constrain the model in a way that makes it intelligible to someone from the outside without sacrificing a lot of performance. So there's a really encouraging scientific story. How that actually rolls out into something that's user facing, I think, is much less clear.

• 30:25Ezra Klein?

  Talk me through a bit of that scientific story, because it is interesting. I mean, five years ago, we were, I mean, all these things were more rudimentary, but we were much worse at figuring out what was happening inside of them. What got figured out and to what degree such that we can now know better? Like what it is a machine has learnt when it is producing an output?

• 30:46Speaker 01

  Sure. So I guess one place to start is.

• 30:52Speaker 01

  If you think about a model, it has kind of three parts. There's what are the inputs, what goes on in the middle, and what are the outputs? And so, one way to make a model simple is to have fewer inputs. One way to make it simple is to have like, less stuff going on in the middle. So, one of the big advances that we've seen in machine learning since about 2010, 11, 12, has been the use of this technology called deep neural networks, which basically just is these simple rudimentary mathematical elements that kind of sort of resemble what goes on in a neuron. It's like, they have a bunch of inputs, and the inputs are just numbers. It's like, 1, 0.5, whatever. You add them up, and then if they're greater than some threshold, you know, you output some other number.

• 31:37Speaker 01

  Otherwise, you output 0. And it turns out that if you have tens of millions of these things stacked into layers, they can do essentially arbitrary tasks. They can tell cats and dogs apart, they can tell cancerous and non-cancerous lesions apart, etc. But there's this real inscrutability, so even if you could pop the hood, you would see like, 60 million of these rudimentary things that all have slightly different thresholds of when they output and when they don't. And so, the question is like, is that level of transparency actually giving you anything? So, that's been, there's been two pushes. One push is, do we need to use this inscrutable technology?

• 32:18Speaker 01

  Or can we use simpler models, you know, your more classic 20th century statistics? Then, the other push has been, okay, for certain applications, let's just say you have to use a giant neural network. Can we actually visualize what's happening with the interior of this network? Can we actually see, for example, that, okay, layer 1 has detected this edge? Or layer 2 is detecting this pattern in this part of the image?

• 32:45Ezra Klein?

  You're talking here for visual recognition?

• 32:47Speaker 01

  Yeah, as an example.

• 32:48Ezra Klein?

  Yeah.

• 32:50Speaker 01

  So that's the kind of the two-front attack that's being made is making simple models competitive with the complex models and making the complex models somehow more intelligible.

• 33:00Ezra Klein?

  I want to put the simple model question to the side because I take it as almost axiomatic that if these different players like DeepMind get where they want to go, we're going to be using some very, very, very complicated models and some very, very complicated systems. And if we get to the kind of general artificial intelligence, it's not going to be a simple model or we'd have it already. So at some point, we're going to need to know what these programs searching through all of the information humanity has ever been able to generate are finding. And we know that when they start searching, they find things we didn't, right? We know that, say, the the model that is now the best AlphaGo player in the world realized different things playing against

• 33:42Ezra Klein?

  itself in AlphaGo than human players had ever realized. And so how do we see what the model is seeing?

• 33:49Speaker 01

  Yeah. One way that you can do it would be this idea that's called perturbation, which is to say, you know, to use your example of Go, you start with a Go board and then you iteratively add and subtract stones to every location on the board. And you see which of those perturbations have the biggest effect in what the model thinks is going on. And you can say, oh, well, when I remove this one stone here at the bridge between these two clumps, it totally changes its evaluation of who's winning. Therefore, I can kind of infer that it's quote unquote focusing on this area or that this area is kind of salient. And that type of process can be really helpful. There are many,

• 34:34Speaker 01

  many horror stories of researchers finding that the model was essentially focused in completely the wrong area. So a group of dermatologists built a system to determine whether these, you know, marks on your arm are cancerous or not. But when they use these kind of perturbation or saliency methods, they discover that actually what the model is looking for is the presence of a ruler next to the lesion because the medical textbook images Had a ruler next to them for scale. So it thinks that the ruler is the cancer, things like that. So that ends up being really helpful. The other thing that you can do with go programs as being an example is to run the model forward and say,

• 35:15Speaker 01

  okay, what do you think is going to happen if you take the action that you're planning to take? And that's another way to get a sanity check of just if we kind of look through the crystal ball, does the model's sense of what's downstream actually track with what makes sense to us?

• 35:50Ezra Klein?

  So we've been talking here about what happens once a machine has learned something. But a lot of your book is about how we are learning to help machines learn and the places we're taking inspiration from on that. And a lot of where we're finding some inspiration is actually us. So could you tell the story just of dopamine and what we have learned dopamine is for in the human mind?

• 36:11Speaker 01

  Yeah. So in the seventies and eighties, we were learning a lot about the dopamine system and we were developing, you know, the actual technology to monitor dopamine, individual dopamine neurons in real time and watch them spike. And it was producing a pretty mysterious story. We could see a monkey reach into a little compartment and find a piece of fruit and boom, there'd be this dopamine spike. But the fifth or sixth time, the spike would go away. And so what was going on here? And to make a long story short, you know, there's this question of, is dopamine encoding our sense of reward? No, not exactly. Is it encoding our sense of surprise? No, not exactly. So what's going on?

• 36:56Speaker 01

  Because we know it's related to those things, but we can't really pin down what this signal actually corresponds to. In parallel, there had been kind of this movement within the computer science AI community called reinforcement learning. And the basic idea was let's build systems that can learn to take actions within an environment to get as many points as possible. Whether you define, you know, you define the points however you want, right? So playing chess, you want to capture pieces or whatever it might be. And one of the methods that was successful in the 80s for solving this problem is what's called temporal difference reinforcement learning. And the basic idea was, you know, you take an action, you don't know if it's good or bad.

• 37:41Speaker 01

  You might not have to wait till the end of the entire game in order to know whether you made a mistake. If you suddenly lose your queen, you know that something went wrong. You don't have to wait till you get checkmated, you know, 30 moves later. And so you can learn from these minute-to-minute differentials in how well you think things are going. It creates kind of an error signal. And you can learn, you know, if you think about trying to predict the weather. On Monday, you predict the weekend weather, and by Tuesday, you have a slightly different prediction. You don't even have to wait and see if you're right. You can just update and say, okay, my Tuesday prediction is going to be slightly more accurate than Monday, so my Monday prediction should have been a little bit more like that.

• 38:26Speaker 01

  And these are called temporal difference errors. It's kind of, you're learning what a guess should have been from a subsequent guess that you make. Okay, so this was all happening, you know, in the computer science departments. So these kind of neurophysiology data land on the desk of some of the computer scientists, in particular Peter Deehan, who had kind of crossed over into the Salk Institute and was doing some neuroscience work. And when the computer scientists looked at it, they said, oh, this is, the brain's doing temporal difference learning. It is learning a guess from a guess. Suddenly, when you find a piece of food that you weren't expecting, you know, life just got a lot better than you thought it was going to be a moment ago.

• 39:09Speaker 01

  But if you come to realize that fruit's always in the box, then life is always about as good as you currently expect it to be. So there's nothing more to learn. I just think this is a remarkable story for a couple of reasons. I mean, primarily, it is this idea that computer science and cognitive neuroscience are engaging in this dialogue, this kind of feedback loop, where each is now informing the other. And it also, I think, tells us a story about AI, that we are, in my view, this gives us some evidence that we're not just merely working out some engineering hacks that happen to solve video games or whatever, but that we're actually discovering some of the fundamental mechanisms of intelligence and learning.

• 39:52Speaker 01

  We've kind of independently found the same mechanism that evolution found. And in fact, there are many different kind of evolutionarily independent temporal difference mechanisms. If you look at bees, if you look at, you know, different species. We really are on to, in my view, the philosophical pay dirt of artificial intelligence, which is to figure out like how our minds work to begin with.

• 40:15Ezra Klein?

  So I love this. So basically, what is being said here is that dopamine is a way of updating our expectations about the world.

• 40:25Speaker 01

  Yes.

• 40:26Ezra Klein?

  That we don't feel good because things got better or worse. We feel good because we are now projecting things to be better or worse. And you say in the book, something that has left me thinking about it for quite a while, which is that this helps explain the idea of the hedonic treadmill, the idea that we, you know, that people get used to winning the lottery, they get used to losing a limb. Why does this help explain that?

• 40:46Speaker 01

  There is this funny connection between the dopamine system and the subjective experience of pleasure and happiness. You know, so this is obviously a major front in philosophy of mind slash neuroscience is how do we link the physical mechanisms in the brain that we can observe to the feeling of being happy or of liking something? And to the degree that dopamine is part of that story, it tells us that it feels good to be pleasantly surprised about how promising your life is. But because you are using that signal to learn, you know, it's like...

• 41:30Speaker 01

  You can't be permanently pleasantly surprised. Like eventually you're gonna learn how to actually make accurate predictions. And so some of that pleasure goes away. And so for me, it tells a story about not just the hedonic treadmill as adults, but the way that evolution has given us this really general purpose learning mechanism that when you're one year old, you know, or six months old or whatever, waving a hand in front of your face, at first it's really delightful because you don't expect what's gonna happen. Or you push something off a table and it falls on the ground and you're delighted because you had no idea what was gonna happen. And eventually, right, you need to get your kicks by playing sports

• 42:15Speaker 01

  or by writing academic papers or writing books or whatever it is. And it's, I think, pretty remarkable that there is this general purpose, take delight in the ways that your predictions are wrong, but also improve your predictions, right? And this kind of sets up this whole trajectory of our life in a way.

• 42:34Ezra Klein?

  But one thing that made me reflect on is the way that we have alignment problems with ourselves.

• 42:39Speaker 01

  Oh, yeah.

• 42:40Ezra Klein?

  So take, you know, the dopamine function you're talking about there. One way of describing the hedonic treadmill is that the things we believe in advance will make us happy don't end up making us happy. So here we are, like wandering through our lives, telling ourselves that if we just like work so hard and get to this point, we're gonna be happy and then we're not. And now dopamine's not doing something wrong. Like it's, you know, optimized for fitness and all the other things evolution wanted for us, which is not always happiness. But but there's a funny way in which we're sitting here talking about how it's hard to create the correct reward functions for machines. But we also don't even really understand day to day how to create the correct reward functions for ourselves. And so we're constantly doing things, like every time I pull out my phone

• 43:23Ezra Klein?

  and open up an app because I'm tired, that don't actually make us happier, but we somehow learned, you know, give us a little hit of dopamine because well, maybe something is gonna happen now that is a little bit better than me being, you know, bored and exhausted right here. You know, even though what usually happens, I get annoyed at everybody fighting on Twitter. And so there's a way in which there's, there's a lot of pathos in us trying to teach. Teach other, you know, beings sentient or not, organic or not, how to think and how to live, when, you know, day to day, I'm not sure we're so good at ourselves.

• 43:54Speaker 01

  Yes, I, I think there's something really deep here, which is that, you know, you could think of humans' relationship to evolution as having this alignment problem, where there are certain things that evolution, quote unquote, wants us to do that will make us, you know, fit to stick around in an environment. That process is really complicated and hard to directly encode into the motivations and desires of people. So instead, we get, we get this weird reward function, or these weird sets of incentives, where, you know, we want to eat chocolate and have sex and open our Twitter on our phone and check our notifications, and all the things that we actually want or are motivated by.

• 44:38Speaker 01

  And this really is the problem of reward design, right? Like, evolution has designed these rewards, and we've, in some ways, over-optimized for them. And I guess, you know, part of, part of the human condition is realizing that you have some degree of agency, you know, just, just because evolution wants you to propagate your genes and do this and that, like, you don't have to. You have some degree of agency over what your own goals are. And I think that's, I mean, that's interesting from a parenting perspective, right? So one of the things Alison Gopnik talks about as a parent is that, you know, being

• 45:17Speaker 01

  a parent is not like building an AI system, because you don't necessarily have the, in mind at the beginning, the notion of what you want your children to want. I mean, maybe you do within certain parameters, but you also want to give them the, the leeway to become their own people. So, yeah, this question of how to behave with respect to a set of rewards that are to some degree kind of hardcoded, but you also have kind of control over your environment, you can sort of shape your own reward function to a degree. I think this is, yeah, intrinsic to the human condition, absolutely.

• 45:57Ezra Klein?

  So, one of the things that the, the research here seems to be doing is giving AI researchers more respect for parts of the human reward system that seemed softer, weirder, more idiosyncratic, more, uh...

• 46:13Ezra Klein?

  Fuzzy. And one of them is very much curiosity.

• 46:15Speaker 01

  Yes.

• 46:16Ezra Klein?

  Can you talk a bit about what we've learned from trying to get computers to play Montezuma's Revenge?

• 46:21Speaker 01

  Yeah, this is one of my favorite stories in AI. So there's a DeepMind team that came together around 2013 to 2015 to try to build an AI system that could beat not just a single Atari game, but every Atari game using the same generic architecture. And they managed to achieve, I think, some pretty incredible results. It was like 25 times better than a human at, you know, video boxing. It was like 13 times better at pinball, et cetera, et cetera.

• 46:55Speaker 01

  But there was one game at which this model scored a total of zero points. And that game is called Montezuma's Revenge. And so there is this question of why was this one Atari game so difficult for this AI system to beat? And the basic answer is that it has what's known as sparse rewards. So most Atari games, you can essentially just mash buttons until you get some kind of points. And for an AI system, that's enough to bootstrap the learning process. And you can say, okay, how did I get those points? Let me do a little bit more of that in the future and so on. But in Montezuma's Revenge, you have to execute this huge sequence of really precise movements.

• 47:39Speaker 01

  Any mistake basically kills you. And only if you do this huge, long chain of things correctly, do you get even the first points of the game. And so the whole premise of this learning algorithm was that you would mash buttons until you got points and then figure out how to get more. But how do you learn if you can never get the first points to begin with? And this is a riddle that is, I think, wonderfully solved by babies. So the computer science community starts looking over the fence at ideas from developmental psychology. Because of course, human beings play these games with no problem. So there's something going on that enables us to understand. How to how to play these games.

• 48:24Speaker 01

  We've known since the 60s that infants have this really strong novelty drive. And psychology is sometimes known as preferential looking. So if you show an infant, you know, a toy and then, you know, an hour later, you give it a choice between that toy and a new toy. They almost always pick the new toy. And this is such a bedrock result that it's used as a way to study memory and perception and things like this in, you know, basically newborns. So there's a very fundamental reward essentially that people get from seeing new things. The idea was, what if we just plug this novelty reward into our video game system?

• 49:06Speaker 01

  Such that we treat the encountering of kind of new images on the screen as literally tantamount to getting in game points, just as good as getting in game points. Suddenly, when you do this, the program has this kind of human like drive. It wants to explore the limits of the game. It wants to go through the locked door just to see what's on the other side. It wants to kind of climb the ladder and jump the fence. And that's what it takes to beat this particular game. So that to me is just another one of these wonderful convergences where some of the insights that we're getting into these very fundamental drives in human intelligence

• 49:48Speaker 01

  end up being imported in a very literal and direct way into AI software. And then suddenly it can do things it could never do before.

• 49:56Ezra Klein?

  Do you believe we're going to get super intelligent general AI? Or do you believe what we're going to get is sort of like kids with savant like capabilities and certain things that we need?

• 50:09Speaker 01

  I think at the limit, I see no fundamental principled reason why we're not going to get some kind of super intelligent general AI. The question is, what does the road there look like? And I think the road there does look like these weird sort of savant like grown up kids. You know, you can think about GPT three.

• 50:31Ezra Klein?

  So GPT-3, for people who don't know, that is OpenAI's predictive text artificial intelligence platform. It's gotten a lot of buzz. It's one that a lot of people are able to use. So like, that's a big place where a lot of people have begun to see how AI could work.

• 50:45Speaker 01

  It's essentially like someone who has lived their entire life in a windowless room with an internet connection and has read everything that's ever been written on the internet, but has no idea what anything actually is beyond how it's spoken about. And it turns out you can fake it pretty well, but eventually some of that ignorance will catch up to you. So for the foreseeable future, a lot of what I'm worried about with AI is how we're going to accommodate systems like that. Because to even, in some ways, to even get to worry about the superintelligent AI, we have to survive the kind of savant grown-up children AI, right?

• 51:29Speaker 01

  And that's the story of The Sorcerer's Apprentice. It was just this animated broom that knew nothing except how to, you know, pour water, and that was dangerous enough.

• 51:38Ezra Klein?

  So I gave a version of this question to Ted Chang, the science fiction author, which was, you know, does he think we're going to get superintelligent AI? And he said, "No, not really. We're not going to get sentient AI either. Maybe we could, but should we?" And then he said, "Absolutely not." And I thought his reasoning was interesting, and I was thinking about it while I was reading your book, which is, he said, "Long before we got sentient, superintelligent AI, we would have AI that could really suffer."

• 52:09Ezra Klein?

  And given how human beings have treated animals, given how they've treated each other, given how they treat machines, we would make this AI suffer on a tremendous level. And as I read your book, there's a pathos to basically every program you describe. We have every time embedded it with an unbelievable want for something. A want to see new screens in Montezuma's Revenge, a want to get points in a video game, a want to, you know, be able to fulfill whatever question has been posed to it. A want, a want, a want. We're creating these little desire machines. And these are Often things that it can't do, or often things that we're going to lose interest in it doing. And I don't know at what point, I know philosophers think about this, you have to think about the moral like weight of this.

• 52:52Ezra Klein?

  At what point for a program to not be able to fulfill its wants, it is feeling pain. But it does strike me as well before we're going to have things that are so intelligent, we have a lot of sympathy for them. And I'm curious how you think about this question.

• 53:06Speaker 01

  It's a great question. You know, there, there is a computer science research group that has, that's, I think, somewhat tongue in cheek. Title of People for the Ethical Treatment of Reinforcement Learning Agents. But there are people who absolutely sincerely think that we should start now. Thinking about the ethical implications of making a program play, you know, Super Mario Brothers for four months straight, you know, 24 hours a day.

• 53:32Ezra Klein?

  You, you talked about one that did Super Mario Brothers and it's just like caught in this game that has no more novelty and it's a novelty seeking robot. And I thought it was so sad.

• 53:40Speaker 01

  Yeah, it just learns to like, sit there because it's like, well, why would I jump across this little, you know, pipe? Because I'm, I'm just, just the same old shit on the other side. Like, well, I might as well just do nothing. I might as well just, you know, kill myself. And there, there have been reinforcement learning agents that because of the nature of the environment. Essentially learn to commit suicide as quickly as possible because there's like a time penalty being assessed. For every second that passes that you don't achieve some goal. And they can't achieve it. So they're like, well, the next best thing is to just like die right now. And again, it's like, we're somewhere on this slippery slope. I mean, there is this funny thing for me where the more I study AI.

• 54:26Speaker 01

  The more concerned I become with animal rights. And I'm not saying that AlphaGo is equivalent to a factory farm chicken or something like that necessarily. But.

• 54:39Speaker 01

  Going back to some of the things we've talked about, you know, the dopamine system, some of these drives that are, you know, the fact that we are building artificial neural networks that at least, you know, to some degree of approximation are modeled explicitly on the brain. We're using TD learning, which is modeled explicitly on the dopamine system. We are building these things in our own image. And so the odds of them having some kind of subjective experience, I think, are higher than if we were just writing, you know, generic software. This is the huge question of philosophy of mind is like, are we going to know if we manage to create something with a subjectivity

• 55:17Speaker 01

  or not? I'm not sure. But these questions, I think, are going to go from seemingly crazy now to, you know, maybe on a par with something like animal welfare by the end of the century. I think that's not a crazy prediction to make.

• 55:35Ezra Klein?

  Yeah. And then you add in the fact that you can create, it's not literally an unlimited number because there's a computing power associated with this. But but at some point, the idea is this will be simple enough and computing power cheap enough that you can create marginal AI agents at very, very low cost. Right? That is how you all of a sudden get all these, you know, super low cost, human level labors. And it does seem a little scary. I've given a lot of attention to this question of what would AI do to us? You know, including well before super intelligent, right? Just putting people out of work, that kind of thing. And very little to the question of what would we do to it? But I don't know, I read your book. And as you say, some of these stories, they already make you feel

• 56:18Ezra Klein?

  terrible to hear, like the ones of the AI just killing itself because there's a penalty to doing anything else at this point. And the idea that we wouldn't know when it was feeling pain, which seems very plausible to me, is pretty profound. So I don't know, I don't know at what point that should actually affect what kind of research we're doing.

• 56:36Speaker 01

  Yeah, I mean, we have, we have a lot more options than we do with human welfare and animal welfare in terms of, for example, if the agent is this novelty seeking agent that then gets really burned out and bored, could we just like?

• 56:55Speaker 01

  Wipe its memory so that, oh, wow, you know, everything is new again and everything is delightful all over again. And it's sort of living in this weird kind of 51st dates environment. Is that itself unethical? Or is that sort of the only ethical thing to do at that point? I mean, it gets pretty head spinning. I think there's also this question of, you know, will there be an ethical imperative to make models simple enough that they don't have ethical standing? And I don't know, I mean, there's a joke that computer scientists make about, you

• 57:37Speaker 01

  know, you have this household robot and, you know, if you pay extra, you get the version that doesn't have a subjectivity. So, you know, it's not suffering. But, you know, by default, the cheap one, they couldn't afford to add that in. So it's, it will do what you want, but it won't like it.

• 57:53Ezra Klein?

  This whole, I have to say, this whole part of the conversation, it leaves me feeling real chilled.

• 57:59Speaker 01

  Yeah.

• 57:59Ezra Klein?

  And like, oh, we're just going to start mind wiping our robot slave helpers because then, I mean, I've, I'm not the most read up in science fiction of anybody you could possibly talk to in a day.

• 58:11Speaker 01

  Yeah.

• 58:11Ezra Klein?

  But I'm read up enough to have read a few stories about ideas like that. And they don't, I wouldn't trust us with that kind of power in general. And having, you know, creating a new class of possibly suffering servant workers, you know, but who we've paid nothing for and who we don't, you know, really think, I mean, given, because I am an animal rights person and think like what we do to chickens is really appalling. I have no illusions that there is some limit of suffering that we would inflict on a species that we have enough justification to not care about, but whose labor we find useful to us. Let me ask you about the more near term cost of this for, for human beings that people

• 58:56Ezra Klein?

  talk about more. You know, Andrew Yang ran for president and to some degrees running for, for mayor of New York on the idea that machine learning is going to, and automation will put people out of, out of jobs. It has a little bit of that quality that the internet had for a while, where you can see it everywhere, but the statistics, machine learning, algorithms, computation, automation has gotten way, way better in recent decades. It has obviously put some people out of work, but we do not have a significantly higher rate of unemployment than we did a few decades ago. There are some changes to labor force participation, but not really gigantic ones that I think you can trace to automation. So why don't we see 10 or 15% unemployment from what we can do now and will we?

• 59:38Speaker 01

  To some extent, you know, I'm, I'm partial to the idea that we sort of create the problems we then have to solve. Like anything that makes email easier to send tells you that it's going to solve the problem of email, because you're only thinking about it from your own perspective. It's going to make it easier for you to deal with your inbox. But if everyone has that program, then there's just more email and everyone's still spending the same amount of time. So it's a treadmill. There are other treadmills like that. I do think that AI has been a big part of the story of inequality. That, you know, if you broadly take the lens of society in the Marxian view, right? The struggle between labor and capital.

• 1:00:22Speaker 01

  AI is the labor that doesn't, it's not human. It doesn't need a wage. It doesn't advocate for itself. And so I'm somewhat sympathetic to the idea that we're permanently tilting this millennia-long tension, or at least centuries-long tension, in favor of capital. And there's a lot of deep psychological questions to think about. You know, the political rhetoric around the economy is framed in terms of jobs and rather than these more primitive things like good health, good education, whatever. Are jobs going to be the salient rhetorical framing of those things going forward?

• 1:01:07Speaker 01

  I'm not convinced that that's the case. I will be very curious to think about what it means for someone to be in the economy when most of the things that they can do are able to be done by machines better, right? So you think about Amazon Mechanical Turk employs a ton of people and the tagline of Amazon Mechanical Turk is artificial, artificial intelligence.

• 1:01:36Speaker 01

  Namely, people.

• 1:01:37Ezra Klein?

  So dystopic.

• 1:01:38Speaker 01

  Right? I mean, it's this kind of soylent green, you know, it's people. But almost in its very premise, like, those are the jobs that are ready to get automated away. So I think it's interesting to think about what it means to contribute economic value in that kind of society. Right now, there are a lot of people whose economic value comes from like what they can see, like their ability to process information visually, or their ability to sort of manipulate objects with their hands, right? If you think about sewing, like, we still don't really have sewing at scale, even though we have all this advanced manufacturing for steel, et cetera, et cetera.

• 1:02:19Speaker 01

  What would it mean to have robots that are dexterous enough to sew? I don't have an answer to that personally. I mean, I think these questions of, does AI kind of lend itself towards this kind of, you know, rich get richer type scenario? In my view, yes. And so I think we have to mitigate that probably politically. But those are huge questions. I think they're open questions, from my view.

• 1:02:46Ezra Klein?

  But so it sounds to me like what you're saying is that the question directly is not jobs so much as it is dignity and status, that you could imagine a world where AI is doing quite a bit. People still have jobs. They're just a little bit ridiculous because, you know, you're operating around the margins of algorithms you don't understand, or you're just sort of mopping up behind them. And then particularly, if you add in that the people who own the AI are getting all the money out of this, so the people with these now kind of crappy jobs also don't have this other generator of status and dignity, which is money in our society, then you have a real social standing problem. And so, you know, one of the questions is like, do we understand?

• 1:03:31Ezra Klein?

  We can give a lot of things dignity. Compared to what people were doing 500 years ago, you know, what I do for a living, which is talk to people and write some stuff about politics on the internet. I am given a lot of status by that in our society, but compared to feeding people at times when people needed to be It's not that useful of a role. And there are a lot of things like that. I mean, we like choose to have some of these roles that are imbued with dignity, but they really do depend on how we see them socially, how much we like give cultural respect to the people in them and how much we actually pay for them. You know, teachers get a lot less dignity than I think they deserve compared to, say, investment bankers because they make so much less money. I think if teachers were paid $500,000 a year,

• 1:04:12Ezra Klein?

  the money combined with the obvious utility of the role would make them like society's most honored people.

• 1:04:17Speaker 01

  Yeah. I mean, I'm also very struck by

• 1:04:22Speaker 01

  a lot of the things we do for pleasure are the very things that we have sort of automated away. Like, for example, the pastimes of the upper class in Victorian England were, you know, fox hunting and gardening, essentially hunting and gathering, right? That's like, we've built this entire civilization so that you don't have to hunt and gather. But then, you know, the people with the most privilege and the most leisure want to hunt and gather for fun because that's at some deep level like what we are built to do. So, yes, I think there's also questions of status from the perspective of we're now, everyone is now comparing themselves to the entire global population.

• 1:05:08Speaker 01

  I remember I went to a panel at UC Berkeley a few years ago where there were, it was Jaron Lanier, Neil Stephenson and Peter Norvig. And they were sort of talking about some of these like long-term economics of AI things. And Peter Norvig said, you know, it used to be the case that if you made pizza and your pizza was 20% better, you might get some super linear amount. You know, you might make 30% more money. But nowadays, if you make an email client that's 20% better, you make all the money and everyone else goes bankrupt or, you know, whatever it is. And we just need to sort of fundamentally rethink.

• 1:05:51Speaker 01

  Yeah, these questions of. I think standing, right? Like, it used to be the case that you could be the best guitar player in a 10-mile radius, and that would make you really cool. And nowadays, it's like, oh, well, you're only the third-best guitar player, so why would I watch your YouTube videos when I could watch the best? I wonder if we are going to see a kind of willful parochialism come back as people realize that there are advantages to being, you know, essentially the big fish in the small pond. I don't know exactly what that looks like.

• 1:06:22Ezra Klein?

  One thing you might say about our society as it currently exists is we've given status to ridiculous things, maybe because we need to keep people busy, maybe because capitalism rewards somewhat absurd things sometimes. But however it is, like, the way we actually attach dignity to roles and then train people from the time they are born to be like achievement monsters trying to get through those roles is also not a great way of doing society. And so maybe it'll be the case that at some point in the kind of post-scarcity future, you have automated agents doing a lot of the fundamental work of the economy, and people can concentrate on things that, say, the classical philosophers would have thought are closer to the good life, things that John Maynard Keynes thought we'd be doing

• 1:07:06Ezra Klein?

  when he imagined how rich we'd be, which is like, we'll be painting and thinking about philosophy, or even just more, I think, prosaically, spending time with our families and going to the park and playing sports with our friends and like having a drink with our buddies. And there will just be more time to enjoy being human, and that won't be looked down upon, because the reason it is looked down upon is we have needed to make that a low-status, low-class activity in order to keep everybody very engaged in this, you know, huge economic machine we want to feed.

• 1:07:35Speaker 01

  Yeah, I mean, I'm very sympathetic to Keynes's vision of like, you know, in hindsight, you're like, well, what went wrong, right? Because we really were on track. And I think about the promise that technology offers society in the broadest terms is to make people happier. But when I think about, am I happier as a function of having been born in the 1980s than if I had been born in the 1880s?

• 1:08:06Speaker 01

  I don't think so. I mean, I have better dental care. I'm not worried about like an abscessed tooth or something like that. But broadly speaking, like I care about my family, my marriage, my friendships. I want to do interesting work and, you know, write books, which I could have done 100 years ago. Viewed from that perspective, technology has like surprisingly little to offer. I don't think it's bringing a lot to the table in terms of addressing the fundamental things that make people happy. I mean, relieving, you know, the sort of creature comforts and the physical drudgery associated with them, I think is huge. But like, we've been past that threshold for several generations at

• 1:08:51Speaker 01

  this point. And I think people are getting less happy, rather than more.

• 1:08:57Ezra Klein?

  Well, doesn't this speak to a possible alignment problem that we were getting at in human beings earlier? I think there's an endless view that if we lift the condition of scarcity, well, then finally, people will have what they need to be happy. But if you believe that we evolved with our reward function, right, our dopamine system and all the rest of it is optimized for getting us through scarcity, for surviving and reproducing conditions of scarcity, then its absence drives us a little bit crazy. We're like the machine trapped in a system with no novelty. Like, we need to find something to keep ourselves busy because, you know, otherwise, we just get listless and a bit lame. Now, I don't fully buy that. I mean, there's a very interesting discontinuity in research where people are

• 1:09:39Ezra Klein?

  very unhappy when they're unemployed, but if they then just shift into retirement during that period, they get happier. Because like, the status of being retired is a much better cultural status than being unemployed. And there have been many, many human societies that have not worked on like neoliberal, capitalistic, or for that matter, like, modern communistic or socialistic thought, right? Hunter-gatherers had a lot more free time than we do. So there are different ways of constructing a society. But I do think there's something possibly to the idea that when you say technology is built to make us happy, fundamentally, what you're saying is it's going to lift scarcity. And it may not be that humans beings are happier, at least beyond a certain point, in a condition of less scarcity, or at the very least, that that condition

• 1:10:22Ezra Klein?

  takes a lot of adapting to and sort of culturally reworking around in order to get the most out of it.

• 1:10:29Speaker 01

  I want to offer a complimentary story to that, which is, I think, as some of this work on the dopamine system and, you know, intrinsic novelty-seeking behavior in inference shows us, not everything is about scarcity. There's something pleasurable about just visual unpredictability. It's why we like looking at screensavers, it's why we like looking at campfires, or, you know, moving water. And I think there's something valuable there, too. And for me, the natural world offers something that's really an antidote to the world of tech. There's something that I have come to through the classic stuff of,

• 1:11:13Speaker 01

  you know, mindfulness meditation, and just hiking, and being in nature, looking at trees, and finding that beautiful. You realize that there's a lot more psychological sufficiency in the act of just existing in the world, you know, controlling your own attention, letting the world just as it is, interest you and surprise you. And there's a problem, which is that, you know, no one's making money when you do that. And so, this is kind of a macro version of what we were talking about earlier in our conversation about, you know, all models are wrong. And in some ways, the danger is that the models can reshape

• 1:11:55Speaker 01

  reality to become right. In this case, we are creating these built environments in which there is just frankly, not a lot of visual novelty, because people live in an apartment with, you know, they can only view the building right next to them, or there's not any trees near them, or whatever it is. And so, in order to get that, like, very primitive level of just visual surprise, you know, you have to check your Twitter feed, or whatever it might be. And that puts you into this world of status competition, because that's how they get you to engage and create the content that other people consume. But if you just walked to your neighborhood park, you know, the park doesn't require anything from you. And so, I think it is worth

• 1:12:41Speaker 01

  Bearing in mind, like, I'm at some level skeptical to the idea that, you know, everything we do is about this kind of positional good, this kind of status competition. When I'm walking through a park, I don't think, this is really cool because other people aren't in this park. I mean, you appreciate the absence of other people for not kind of getting in the way of your experience of nature, but you don't think about it as like, this is cool because it is scarce, or I am the victor of this competition to be in this park, or whatever it is. But I get as much from that as I get from Instagram. You know, like, that's the funny thing.

• 1:13:19Ezra Klein?

  I think it's actually a good place to bring it to an end here. So, let's go back to Analog for a minute. What are three books you'd recommend to the audience?

• 1:13:27Speaker 01

  Yeah, so, thinking about both what's coming down the highway and also kind of how do we think about human motivation and human desire. So, the first book that comes to my mind is by Julie Shah and Laura Major. Julie and I were high school classmates, and she's now an MIT roboticist who works on aerospace manufacturing and all sorts of things. Their book is called What to Expect When You're Expecting Robots. I think it's a really interesting and persuasive look at kind of the next decade-ish in terms of human robot interaction. I'm also thinking about a book by James Karss, who was a professor

• 1:14:09Speaker 01

  of religion at NYU, called Finite and Infinite Games. And there's this wonderful backstory to this book where he is a professor of religion who attended a game theory conference in the 80s, and then writes this book, which is religion meets game theory meets like Wittgenstein's tractatus. It's this very weird, very unique book that is all about, what are people really trying to do? And there are certain things that we do to achieve like a specific end that we can envision in advance, other things that we do in this more sort of horizontal, open-ended way to kind of surprise ourselves or prolong an experience. And I think it's a useful way of actually thinking about living one's

• 1:14:52Speaker 01

  life. It also maps to, you know, problems in AI in a very interesting way. The third one, kind of coming back to what we were saying about the pleasure of being in the neighborhood park, I'm thinking about a book by my friend Jenny Odell called How to Do Nothing: Resisting the Attention Economy. And it's, on one level, you know, a love letter to your neighborhood park. And at another level, you know, an invitation to think about a world in which most of our activity is directed at some kind of objective. And again, I think there are surprisingly relevant resonances here with AI. You know, you can't make an AI system without an explicit objective function that it's trying to maximize.

• 1:15:36Speaker 01

  What does it mean to, quote unquote, "do nothing"? And there's something, I think, powerful about that. Again, both for thinking about what intelligent machines might be like, but also in terms of thinking about these deep questions of human motivation. What makes life enjoyable?

• 1:15:53Ezra Klein?

  Brian Christian, your book is The Alignment Problem. It is fantastic. I highly recommend it. Thank you very much.

• 1:15:58Speaker 01

  It's been a pleasure. Thank you.

• 1:16:10Ezra Klein?

  The Zoklan Show is a production of New York Times Opinion. It is produced by Jeff Geld, Virja Karma, and Annie Galvin. Fact checking by Michelle Harris. Original music by Isaac Jones and mixing by Jeff Geld.