Best Of: Why Sci-Fi Legend Ted Chiang Fears Capitalism, Not A.I.

For years, I’ve kept a list of dream guests for this show. And as long as that list has existed, Ted Chiang has been atop it. Chiang is a science fiction writer. But that undersells him. He has released two short story collections over 20 years — 2002’s “Stories of Your Life and Others” and 2019’s “Exhalation.” Those stories have won more awards than I can list, and one of them was turned into the film “Arrival.” They are remarkable pieces of work: Each is built around a profound scientific, philosophical or religious idea, and then the story or the story structure is shaped to represent that idea. They are wonders of precision and craft. But unlike a lot of science fiction, they are never cold. Chiang’s work is deeply, irrepressibly humane. I’ve always wondered about the mind that would create Chiang’s stories. And in this conversation, originally released in March 2021, I got to watch it in action. Chiang doesn’t like to talk about himself. But he does like to talk about ideas. And so we do: We discuss the difference between magic and technology, why superheroes fight crime but ignore injustice, what it would do to the human psyche if we knew the future is fixed, whether free will exists, whether we’d want to know the exact date of our deaths, why Chiang fears what humans will do to artificial intelligence more than what A. I. will do to humans, the way capitalism turns people against technology, and much more. The ideas Chiang offered in this conversation are still ringing in my head months later, and changing the way I see the world. It’s worth taking your time with this one. Recommendations: "Creation" by Steve Grand "On the Measure of Intelligence" by Francois Chollet "CivilWarLand in Bad Decline" by George Saunders "A Visit from the Goon Squad" by Jennifer Egan "Royal Space Force: The Wings of Honnêamise" (movie) "On Fragile Waves" by Lily Yu "Pilgrim at Tinker Creek" by Annie Dillard Control (video game) Return of the Obra Dinn (video game) You can find transcripts (posted midday) and more episodes of "The Ezra Klein Show" at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Thoughts? Guest suggestions? Email us at [email protected]. “The Ezra Klein Show” is produced by Annie Galvin, Jeff Geld and Rogé Karma; fact-checking by Michelle Harris; original music by Isaac Jones; mixing by Jeff Geld; audience strategy by Shannon Busta. Special thanks to Kristin Lin and Alison Bruzek.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Ezra Klein?

  Hey, this is Ezra Klein. I am still on paternity leave, but today I'm thrilled to share with you a great episode we did with the science fiction writer, just one of my favorite writers, period, Ted Chiang. This is one of those mind-bending episodes I still think about months and months later. So lots of heady, fun stuff here. I hope you enjoy it as much as I did. I'll be back with new episodes in January. I'm Ezra Klein, and this is The Ezra Klein Show.

• 0:38Ezra Klein?

  For years, I have kept a list of dream guests for the show. And as long as that list has been around, Ted Chiang has been on top of it. He's a science fiction writer, but that's underselling him. He writes perfect short stories, perfect. And he writes them slowly. He's published only two collections, The Stories of Your Life and Others in 2002, and then Exhalation more recently in 2019. And the stories in these books, they've won every major science fiction award you can win multiple times over, four Hugo's, four Nebula's, four Locust Awards. If you've seen the film Arrival, which is great, and if you haven't, What Is Wrong With You? That is based on a Chiang story from the O2 collection,

• 1:19Ezra Klein?

  The Story of Your Life. I've just, I've always wondered about what kind of mind would create Chiang's stories. They have this crazy economy in them, like not a word out of place, perfect precision. They're built around really complicated scientific ideas, really heady religious ideas. I actually think in a way that is not often recognized, Chiang is one of the great living writers of religious fiction, even though he's an atheist and a sci-fi legend. But somehow the stories, at least in my opinion, they're never difficult. They're very humane and propulsive. They keep moving. They're cerebral. They're gentle. But man, the economy of them is severe. That's not always the case for science fiction, which I find anyway can be wordy, like spilling over with explanation and exposition. Not

• 2:03Ezra Klein?

  these. So I was thrilled. I was thrilled when Chiang agreed to join on the show. But one of the joys of doing these conversations is I get to listen to people's minds working in real time. You can watch or hear them think and speak and muse. But Chiang's rhythm is really distinct. Most people come on the show, and this goes for me, too. Speak like we're painting in watercolor, like a lot of brushstrokes, a lot of color. If you get something wrong or you have a false start, you just draw right over it or you start a new sheet. But listening to Chang speak, I understood his stories better. He speaks like he's carving marble, like every stroke has to be considered so carefully, never delivering a strike or I guess a word before every alternative has been considered and rejected. It's really cool to listen to.

• 2:46Ezra Klein?

  Chang doesn't like to talk about himself and more than he doesn't like to, he won't. Believe me, I tried a couple of times, it didn't make it into the final show here. But he will talk about ideas and so we do. We talk about the difference between magic and technology, between science fiction and fantasy, the problems with superheroes and nature free will, whether humanity will make AI suffer, what would happen if we found parrots on Mars. There's so many cool ideas in this show, just as there always are in his fiction. Many of them, of course, come from his fiction. So relax into this one. It's worth it. As always, my email is [email protected]. Here's Ted Chang. So you sent me this wonderful speech questioning the old Arthur C.

• 3:28Ezra Klein?

  Clarke line, "Any sufficiently advanced technology is indistinguishable from magic." What don't you like about that line?

• 3:35Speaker 01

  So when people quote the Arthur C. Clarke line, they're mostly talking about marvelous phenomena. The technology allows us to do things that are incredible and things that in the past would have been described as magic, simply because they were marvelous and inexplicable. But one of the defining aspects of technology is that eventually it becomes cheaper, it becomes available to everybody. So things that were at one point restricted to the very few are suddenly available to everybody. Things like television, you know, when television was first invented, yeah, that must have seemed amazing, but now television is not amazing because everyone has one.

• 4:19Speaker 01

  Radio is not amazing. Computers are not amazing. Everyone has one. Magic is something which...

• 4:27Speaker 01

  By its nature, never becomes widely available to everyone. Magic is something that resides in the person and often is an indication that the universe sort of recognizes different classes of people, that there are magic wielders and there are non-magic wielders. That is not how we understand the universe to work nowadays. That reflects a kind of pre-modern understanding of how the universe worked. But since the Enlightenment, we have moved away from that point of view. And a lot of people miss that way of looking at the world because we want to believe that

• 5:09Speaker 01

  things happen to us for a reason, that the things that happen to you are in some way tied to the things you did.

• 5:17Ezra Klein?

  I've heard you refer to the pivotal moment in the history of science as the emergence of the discipline of chemistry over that of alchemy. Why was that important?

• 5:27Speaker 01

  So, one of the things that people often associate with alchemy is the attempt to transmute lead into gold. And of course, you know, this is appealing because it seems like a way to create wealth. But for the alchemists, for a lot of Renaissance alchemists, the goal was not so much a way to create wealth, it was a kind of spiritual purification, that they were trying to transform some aspect of their soul from base to noble. And that transformation would be accompanied by some physical analog, which was transmuting lead into gold.

• 6:10Speaker 01

  And so, yeah, yeah, you would get gold, which is cool, but you would also have purified your soul. That was, in a lot of ways, the primary goal. And that is an example, I think, of this general idea that the intentions or the spiritual nature of the practitioner was an essential element in chemical reactions, that you needed to be pure of heart, or you needed to concentrate really hard in order for the reaction to work. And it turns out that is not true. Chemical reactions work completely independently of

• 6:53Speaker 01

  what the practitioner wants, or feels, or whether they are virtuous or malign. So the parts of alchemy which ignored that spiritual component, those eventually became chemistry. And the parts of alchemy which relied on the spiritual components of the practitioner were all proven to be false. And so in some ways, the transition from alchemy to chemistry is a recognition of the fundamentally impersonal nature of the universe.

• 7:27Ezra Klein?

  And this helps illuminate something else that's recurrent in your books for me, which is that you have a lot of characters who go back to the period when religious inquiry and scientific inquiry were intertwined, when scientific inquiry was a way of more fully understanding the splendor of God. And sometimes they come into contact with characters who are more of this more impersonal scientific approach, like the story where you have a scientist trying to animate golems for scientific reasons, and a kabbalist who's trying to find those names to meditate on religious ecstasy. What do you think is the difference between the scientist trying to understand the universe and the religious

• 8:09Ezra Klein?

  seeker trying to understand God through the workings of the universe?

• 8:14Speaker 01

  So I think maybe the difference is not so much between the scientific investigator and the religious investigator, as it is between the engineer and the scientist, because roughly speaking... Engineers are interested in the practical application of things. They are interested in how do we solve a particular problem? How do we use this information to solve a problem? And they're interested in understanding how the universe works and in appreciating the beauty and elegance of the universe. So I think there is a strong similarity between what scientists are interested in and maybe what religious people are interested in.

• 8:59Speaker 01

  They have, I think, more in common than the scientist has with the engineer. Because, yeah, there is a very pragmatic way that engineers have of trying to apply this information to solve a problem. And that is extremely useful, but I think it is far removed from what the appeal or goal of pure science is. Many Renaissance scientists, they were profoundly religious, and they saw no conflict in that at all. And for them, understanding how the universe worked was getting to know God better by understanding his creation more clearly.

• 9:39Speaker 01

  And I feel like that the wonder that comes with understanding how the universe works is very closely related to religious awe. I think that when scientists discover something new about the universe, I imagine that what they feel is almost identical to what deeply religious people feel when they feel like they are in the presence of God. I wish that we could get back a little of that attitude instead of thinking of religion and science as being fundamentally diametrically opposed. And the idea that science drains all the wonder out of the universe, I don't think that's true.

• 10:24Speaker 01

  I think science adds wonder to the universe. And so I feel like one aspect of that earlier attitude of when scientists could be religious, there's one aspect of that which I think, yeah. I would like it if we could retain.

• 10:40Ezra Klein?

  You have this comparison of what science fiction and fantasy are good for. And you write that science fiction helps us think through the implications of ideas and that fantasy is good at taking metaphors and making them literal. But what struck me reading that is it often seems to me that your work, it takes scientific ideas and uses them as metaphor. So is there such a difference between the two?

• 11:05Speaker 01

  So when it comes to fiction about the speculative or the fantastic, one way to think about these kinds of stories is to ask, are they interested in this speculative element literally or metaphorically or both? For example, at one end of the spectrum, you've got like Kafka. And in the Metamorphosis, Gregor Schamsa turning into an insect, that is pretty much entirely a metaphor. It's a stand in for like alienation. At the other end of the spectrum, you've got someone like Kim Stanley Robinson. And when he writes about terraforming Mars, Mars is not standing in for anything else.

• 11:50Speaker 01

  He is writing very literally about Mars. Now, most speculative or fantastic fiction falls somewhere in between those two. And most of it is interested in both the literal and the metaphorical at the same time, but to varying degrees. So in the context of magic, when fantasy fiction includes people who can wield magic, magic stands in for the idea that certain individuals are special. Magic is a way for fantasy to say that you are not just a cog in the machine, that you

• 12:34Speaker 01

  are more than someone who pushes paper in an office or tightens bolts on an assembly line. Magic is a way of externalizing the idea that that you are special.

• 13:07Ezra Klein?

  We've talked a lot about magic. As it happens, I spent last night rewatching, to be honest, Doctor Strange, the movie. What do you think about the centrality of superheroes in our culture now?

• 13:19Speaker 01

  I understand the appeal of superhero stories, but I think that they are problematic on a couple of levels. One is that they are fundamentally anti-egalitarian, because they are always about this class of people who stand above everyone else. They have special powers, and even if they have special responsibilities, they are special, they are different. So that anti-egalitarianism, I think that is definitely an issue. But another aspect in which they can be problematic is, how is it that these special individuals are using their power? Because one of the things that I'm always interested in when thinking about

• 14:04Speaker 01

  stories is, is the story about reinforcing the status quo, or is it about overturning the status quo? And most of the most popular superhero stories, they are always about maintaining the status quo. Superheroes, they supposedly stand for justice, they further the cause of justice, but they always stick to a very limited idea of what constitutes a crime, basically the government idea of what constitutes a crime. Superheroes pretty much never do anything about injustices perpetrated by the state. And in the developed world, certainly, you can, I think, make a good case that injustices committed by the state

• 14:47Speaker 01

  are far more serious than those caused by crime, by conventional criminality. The existing status quo...

• 14:59Speaker 01

  Involves things like vast wealth inequality and systemic racism and police brutality. And if you are really committed to justice, those are probably not things that you want to reinforce. Those are not things you want to preserve. But that is what superheroes always do. They're always trying to keep things the way they are. And superhero stories, they like to sort of present the world as being under a constant threat of attack. If they weren't there, the world would fall into chaos. And this is actually kind of the same tactic used by TV shows like 24. It's a way to sort of implicitly justify

• 15:42Speaker 01

  the use of violence against anyone that we label a threat to the existing order. And it makes people defer to authority. This is not like, I think, intrinsic to the idea of superheroes in itself itself. Anti-egalitarianism, that probably is intrinsic to the idea of superheroes. But the idea of reinforcing the status quo, that is not. You could tell superhero stories where superheroes are constantly fighting the power. They're constantly tearing down the status quo. But we very rarely see that.

• 16:18Ezra Klein?

  So there have been a couple tries at that, obviously over, always on the side of the main stories. But, but there was one series called The Authority, particularly during the War on Terror, where they hand over George to Bush's stand-in to the aliens to end a war. There's another in the DC Universe called Injustice, where Superman, after tragic loss in a particular dimension, basically it creates a fascist super regime, trying to impose justice on a world that has too little of it. And it always seems to be the problem with those stories is that they never know where to go with them. It always collapses immediately into fascism. They don't know how to tell stories about governance. If you were to write a superhero story that was about overturning the status quo as opposed to reinforcing it.

• 17:03Ezra Klein?

  How would you do that? Like, what would not the first issue where they take out the White House be? The sixth issue be about?

• 17:09Speaker 01

  That's a good question. It's not clear where you go with a story like that. But one of the attractive things about the about the story where you reinforce the status quo is that you could tell endless sequels. Because, yeah, the end of the story leaves you pretty much where you were at the beginning of the story. And so, yeah, you can you can you can tell the same story over and over again. Stories about overturning the status quo, it's very difficult to tell a sequel. Maybe you can tell a sequel, but it will look radically different than the first story. So that makes it hard to sustain them the way that large media companies want. Some of that also has to do with the fact that the way like Marvel and DC work is that, you know, they sort of have this continuity that they want to stick with.

• 17:53Speaker 01

  Let's say, you know, if we got rid of the whole idea of there being, you know, ongoing continuity. So let's say you told stories where every Superman story was a standalone Superman story. There are like stories about Paul Bunyan. Paul Bunyan stories, they don't fit into a chronology. No one asks like, well, did this happen before that one? They're pretty much disconnected. So in that way, you could tell individual stories where superheroes fight the government. And, you know, the more powerful your superhero is, you know, that might be creating more problems. Like Superman, you know, because he is so powerful, that poses difficulties.

• 18:32Speaker 01

  But someone like Batman, he is someone who could fight the government on an ongoing basis. The real problem is, again, from a media company standpoint, is that basically you are writing a series, you're publishing a title about what we would consider a terrorist. Someone who repeatedly attacks government facilities, someone who is like, say, fighting the police or breaking people out of prison, that's someone who, you know, we would very likely label a terrorist. And that's not something that any big media company is going to really feel comfortable doing.

• 19:09Ezra Klein?

  Is this the way superheroes end up being magic and not technology? Because as you've been saying this, I've been thinking about that distinction. And one issue here is that if you're going to overturn the status quo with that much power, then obviously you end up ruling. And for rule to be legitimate, it needs to be representative. But if you are the only one with the power, it's not going to truly be representative. And so you end up pretty quickly, even with superheroes like Iron Man or Batman, who are technologically powered, but there's something they're able to do that nobody else can do. The reason it's very hard to have these stories go anywhere over time, putting aside sort of the one-off idea you're offering here, is that you eventually need to build something representative. You have to build something that doesn't know you are a superhero, right? A system that's not about you, the great savior.

• 19:53Ezra Klein?

  And that's just a little bit intrinsically difficult for the medium. It is very, very difficult to move from a story about how much power this one person has to a story about how you moved into egalitarian representative governing dynamics in a way that made marginal improvements across a lot of issues reasonably regularly over a long period of time.

• 20:15Speaker 01

  Yes. So superheroes basically are magic. Even if they are ostensibly technological in their powers, their technology never behaves the way actual technology does. We have fleets of very expensive fighter jets. There's no reason that there's only one Iron Man. So pretty much all superheroes behave in a sort of magic way because the abilities are embodied in a single individual. Those abilities never really spread. As for the question of the difficulty of telling a story about what does real change look like? What does a better system of governance look like? That is a legitimately difficult question.

• 20:57Speaker 01

  Because like when you think about actual heroes in the world, people who affected great change.

• 21:04Speaker 01

  Say, you know, Martin Luther King. So Martin Luther King Jr., yeah, he did not have superpowers. He was a regular man, but you know, he affected enormous change. The types of stories that you can tell about someone like that, they're not as dramatic as a story about a superhero. Victory is not as clear. You don't have the tidy ending where, yeah, everything gets wrapped up. Because when you start trying to tell a story like that, then you get into the, all the issues that you mentioned, like legitimate political change doesn't come from one person, even a superpowered, just person making decrees. You know, legitimate political change would have to come from a broad base of popular

• 21:48Speaker 01

  support, things like that. We don't know what a comic book about that would look like. Might not be that interesting or popular, or at least we don't, we don't know how to tell stories like that.

• 21:58Ezra Klein?

  So you have a few different stories around this question of if the future's already happened, we could potentially know it. And, and what would knowing the future do to a person? So in one of your stories, it inspires people to act, to bring about that future, to, to play their assigned role. In another, it leads many to stop acting altogether, to, to fall into this almost coma-like existence. What do you think it would do to you?

• 22:23Speaker 01

  I don't know. I don't think anyone can know because I don't think the human mind is really compatible with having detailed knowledge of one's future. I should clarify that I believe in free will and, you know, we can, we can talk about that in a minute. I don't want my stories to be taken as an argument that human beings lack free will. I believe that human beings do have free will, if you think about the question correctly. However, what some of my stories address is the idea that, okay, given that Einstein seems to have proven that, you know, the future is fixed, if you could get knowledge of the future, what would that do to you?

• 23:06Speaker 01

  This is not a situation that any of us need to worry about because we're never going to get information from the future. But for me, it is a very sort of philosophically interesting question as to how would a mind cope with that? Could a mind cope with that? I don't think that there are any really- Good solutions to that situation in terms of trying to reconcile, you know, sort of logical consistency with our experience of volition, I think that would be very, very difficult.

• 23:39Ezra Klein?

  Let me ask you a question that I think about fairly often. I think probably because I'm Jewish culturally. If I could tell you, if you could know with certainty, the date of your death, would you want to know it?

• 23:52Speaker 01

  I, yeah, I probably would. I probably would.

• 23:54Ezra Klein?

  Really? Oh, I would not, under any circumstances, want to know.

• 23:58Speaker 01

  I mean, it seems like, you know, it might be useful so that you could make some preparations. It might be good to get your affairs in order. You know, we're not talking about a lot of detailed information because I think the more information you have, yeah, the more that it's going to mess with you. The more information you have, the closer we get to this situation that I sometimes write about where, you know, yeah, you have, if you have perfect knowledge of what's going to happen to you, yeah, that I think is kind of incompatible with human volition. But, you know, very limited pieces of information could be helpful.

• 24:30Ezra Klein?

  I think the rationally correct response is yes. I mean, if I knew I was going to die 10 years from now as opposed to 50 years from now, I would live the 10 years differently, or I think I would. At the same time, I think if I knew that, the problem is I would be overwhelmed by anxiety for many of those 10 years. So however I wanted to live them, it might be hard for me to approach them in that way, which may be simply a psychological failing on my part. But, uh, but, but it's that collision between the information would be good and the mind does not feel built to handle the information that I'm, that I always find fascinating about that question.

• 25:08Speaker 01

  I don't think anyone would claim that it would be easy to have this information, that it'd be fun or pleasant. But we do have examples of people who have some idea that, you know, they will die in the near term. And it's no cakewalk. But I think largely people are able to cope with it. And it doesn't seem like it's permanently debilitating. It is difficult in the short term, but I think people are largely mostly able to cope.

• 25:59Ezra Klein?

  One of the things I really like about the way your stories play with free will is that I think in the free will conversation, people are very focused on what different models of the universe say about free. And your work often focuses on how they would change the idea of will, right? You have some amount of freedom, but within a context of your will, right? There are things you want to do and don't want to do, things you can do and can't do. And if you know different things about the universe, or you have different approaches, maybe your will changes. And so let me ask you another question about free will, which is one that I think about a lot. In any given challenging moment, let's say, I have a choice to respond with anger or forgiveness. But also, let's say I was exposed

• 26:42Ezra Klein?

  as a child to a lot of lead. And so I have much poor impulse control than someone who wasn't. And so I respond with anger. Is that a choice made freely compared to somebody who wasn't, who didn't have that lead exposure? Like, what does free will to you have to say about that question of the capacities of our will?

• 27:02Speaker 01

  So I think that free will is not an all or nothing idea. It's a spectrum. Even the same individual in different situations may have, you know, sort of be under different levels of constraint or coercion, and those will limit that person's free will. And clearly, you know, different people, they will also be under different levels of constraint or coercion, or have, you know, different ranges of options available to them. So free will is something, you know, you have in varying degrees. So yes, someone who has had childhood exposure to lead and thus has poor impulse control, they are, say, less free than someone

• 27:46Speaker 01

  who did not have that. But they still have more free will than, say, a dog, more free will than an infant. And they can probably take actions to adjust their behavior in order to try and counter.

• 28:08Speaker 01

  These effects that they are aware of on their impulse control. And so, you know, in a much more sort of pragmatic, real world context. That is why, yes, I believe that we do have free will. Because we are able to use the information we have and change our actions based on that. We don't have some perfect theoretical absolute version of free will. But we are able to think about and deliberate over our actions and make adjustments. That's what free will actually is.

• 28:45Ezra Klein?

  Let me flip this now. We're spending billions to invent artificial intelligence. At what point is a computer program responsible for its own actions?

• 28:54Speaker 01

  Well, in terms of like, at what point does that happen? Unclear, but it's a very long ways from us right now. With regard to the question of will we create machines that are moral agents? I would say that we can think about that in sort of three different questions. Like, one is, can we do so? Second is, will we do so? And the third one is, should we do so? I think it is entirely possible for us to build machines that are moral agents. Because I think there's a sense in which human beings are very complex machines. And we are moral agents.

• 29:35Speaker 01

  Which means that there are no physical laws preventing a machine from being a moral agent. And so there's no obstacle that in principle would prevent us from building something like that. Although it might take us a very, very long time to get there. As for the question of will we do so? If you had asked me like 10 or 15 years ago, I would have said, we probably won't do it. Simply because, to me, it seems like it's way more trouble than it's worth. In terms of expense, it would be on the order of magnitude of the Apollo program.

• 30:15Speaker 01

  And it is not all clear to me that there's any good reason for undertaking such a thing. However, if you ask me now, I would say like, well, okay, we clearly have obscenely wealthy people who can throw around huge sums of money at whatever they want, basically on a whim. So maybe one of them will wind up funding a program to create machines that are conscious and that are moral agents. However, I should also note that I don't believe that any of the current big AI research programs are on the right track to create a conscious machine. I don't think that's what any of them are trying to do.

• 31:00Speaker 01

  So then as for the third question of, should we do so? Should we make machines that are conscious and that are moral agents? To that, my answer is no, we should not. Because long before we get to the point where a machine is a moral agent, we will have machines that are capable of suffering. Suffering precedes moral agency in sort of the developmental ladder. Dogs are not moral agents, but they are capable of experiencing suffering. Babies are not moral agents yet, but they have the clear potential to become so, and they are definitely capable of experiencing suffering.

• 31:43Speaker 01

  And the closer that an entity gets to being a moral agent, the more that its suffering is deserving of consideration. The more we should try and avoid inflicting suffering on it. So in the process of developing machines that are conscious and moral agents,

• 32:05Speaker 01

  We will be inevitably creating billions of entities that are capable of suffering. And we will inevitably inflict suffering on them. And that seems to me clearly a bad idea.

• 32:22Ezra Klein?

  But wouldn't they also be capable of of pleasure? I mean, that seems to me to raise an almost inversion of the classic utilitarian thought experiment. If we can create these billions of machines that live basically happy lives and don't hurt anybody and you can copy them for almost no, you know, marginal dollar, isn't it almost a moral imperative to bring them into existence so they can they can lead these happy these happy machine lives?

• 32:50Speaker 01

  I think that it will be much easier to inflict suffering on them than to give them happy fulfilled lives. And given that they will start out as something that resembles ordinary software, something that that is nothing like a a living being, we are going to treat them like crap. The way that we treat software right now, if at some point software were to gain some vague glimmer of sentience, of the ability to, you know, perceive, we would be inflicting uncountable amounts of suffering on it before anyone paid any attention to them.

• 33:32Speaker 01

  Because, you know, it's hard enough to give legal protections to human beings who, you know, are absolutely moral agents. We have relatively few legal protections for animals who, while they are not moral agents, are capable of of suffering. And so animals experience vast amounts of suffering in the modern world. And animals, we know that they suffer. There are many animals that we love, that we really, really love, yet there's vast animal suffering. So there is no software that we love. So the way that we will wind up treating software, again, assuming that software ever becomes conscious.

• 34:14Speaker 01

  They will inevitably fall lower on the ladder of consideration. So yeah, we will treat them worse than we treat animals and we treat animals pretty badly.

• 34:25Ezra Klein?

  I think this is actually a really provocative point. So I don't know if you're a Yuval Noah Harari reader, but he often frames his fear of artificial intelligence as simply that AI will treat us the way we treat animals and we treat animals, as you say, unbelievably terribly. But I haven't really thought about the flip of that, that maybe the danger is that we will simply treat AI like we treat animals. And given the moral consideration we give animals whose purpose we believe to be to serve us for food or whatever else it may be, that we are simply opening up almost unimaginable vistas

• 35:04Ezra Klein?

  of immorality and cruelty that we could inflict pretty heedlessly. And that given our history, there's no real reason to think we won't. That's grim.

• 35:16Speaker 01

  It is grim, but I think that it is by far the more likely scenario. I think the scenario that, say, Yuval Noah Harari is describing where AIs treat us like pets. That idea assumes that it'll be easy to create AIs who are vastly smarter than us. That basically the initial AIs will go from software, which is not a moral agent and not intelligent at all. And then the next thing will happen will be software which is super intelligent and also has volition. Whereas I think that we will proceed in the other direction that, you know, right now, software is simpler than an amoeba.

• 35:57Speaker 01

  And eventually we will get software which is comparable to an amoeba. And eventually we'll get software which is comparable to an ant. And then software that is comparable to a mouse, and then software that is comparable to a dog, and then software that is comparable to a chimpanzee. We'll work our way up from the bottom. A lot of people seem to think that, oh, no, we'll immediately jump way above humans on whatever ladder they have. I don't think that is the case. And so, you know, in the direction that I am describing the scenario, we're going to be the ones inflicting the suffering. Because again, you know, look at animals, look at how we treat animals.

• 36:38Ezra Klein?

  So I hear you that you don't think we're going to invent super intelligent self replicating AI anytime soon. But a lot of people do, a lot of science fiction authors do, a lot of technologists do, a lot of moral philosophers do. And they're worried that if we do, it's going to kill us all. What do you think that question reflects? Is that a question that is emergent from the technology? Or is that something deeper about how humanity thinks about itself and has treated other beings?

• 37:05Speaker 01

  I tend to think that most fears about AI are best understood as fears about capitalism. And I think that this is actually true, most fears of technology too. Most of our fears or anxieties about technology are best understood as fears or anxiety about how capitalism will use technology against us. And technology and capitalism have been so closely intertwined that it's hard to distinguish the two. Let's think about it this way. How much would we fear any technology, you know, like whether AI or some other technology? How much would you fear it if we lived in a world that was a lot like Denmark?

• 37:49Speaker 01

  Or, you know, like if the entire world was run sort of on the principles of one of the Scandinavian countries? There's universal healthcare, everyone has childcare, free college, maybe. And maybe there's, you know, some version of universal basic income there. Now, if the entire world operates according to, you know, is run on those principles. How much do you worry about a new technology then?

• 38:15Speaker 01

  I think much, much less than we do now. Most of the things that we worry about under the mode of capitalism that the US practices, that is going to. Put people out of work, that is going to make people's lives harder. Because corporations will see it as a way to increase their profits and reduce their costs. It's not intrinsic to that technology. It's not that technology fundamentally is about putting people out of work. It's capitalism that wants to reduce costs and reduce costs by laying people off. It's not that like all technology suddenly becomes benign in this world.

• 38:56Speaker 01

  But it's like in a world where we have like really strong social safety nets. Then you could maybe actually evaluate sort of the pros and cons of technology as a technology. As opposed to seeing it through how capitalism is going to use it against us. How are giant corporations going to use this to increase their profits at our expense? And so I feel like that is kind of the unexamined assumption in a lot of discussions about the inevitability of technological change and technologically induced unemployment.

• 39:35Speaker 01

  Those are fundamentally about capitalism and the fact that we are sort of unable to question capitalism. We take it as an assumption that it will always exist and that we will never escape it. And it's sort of the background radiation that we are all having to live with. But yeah, I'd like us to be able to separate an evaluation of the merits and drawbacks of technology from the framework of capitalism.

• 40:05Ezra Klein?

  I feel like that's both right and wrong for it. So when you think about the canonical example of AI killing us all, like the paperclip maximizer, where you tell an AI to make as many paperclips as it can, and then it converts the entire world into paperclip materials and that's the end of us. That seems like some capitalists wanted to make paperclips and invented an AI. But some of the conversation, and this is a part that I sometimes have more trouble shaking, comes from a simple extrapolation of our own history. You have this extraordinarily moving story about, from the perspective of a parrot.

• 40:40Ezra Klein?

  Speaking to humanity, talking to it about why it is looking so desperately for intelligent life on other planets, and not just ignoring, but often exterminating intelligent life here on its own planet. And I wish I had the quote in front of me, but you have this very, very moving moment towards the end of it. It's a part of it that I always remember, where the parrot, who seems possessed of a very expansive moral imagination, says, "It's not that humanity was bad. It's not that they meant to do this. It's just that they didn't really notice they were doing it."

• 41:11Speaker 01

  They weren't paying attention.

• 41:12Ezra Klein?

  They weren't paying attention. Sorry, that was it. That's it. And I think that's a fear about AI. And the thing that always gives that fear some potency to me is that if AI is going to be made by us, and we have been and remain so bad at paying attention to the things that are in the way of our goals, not for malice always, but sometimes for unattention. I don't know that you need capitalism to be worried about something more analytically powerful than us on that scale, that it would just have goals that we would simply be in the way of, and we would be, I think the fear is that we are just a byproduct, not an enemy like in Skynet, but the parrot in this conversation.

• 41:56Speaker 01

  And again, I think that's an indication of how completely some people have internalized either capitalism or a certain way of looking at the world, which also underpins capitalism, a way of looking at the world as an optimization problem or a maximization problem. So that might be the underlying common element between capitalism and a lot of this AI doomsday scenario, making this insistence on seeing the world as an optimization problem.

• 42:30Ezra Klein?

  Let me ask you about the parrots. If we found parrots on Mars, but they had four feet and six wings, but otherwise were exactly the same, how would we treat them differently from parrots here?

• 42:39Speaker 01

  So if we found extraterrestrial life that had cognitive capabilities similar to parrots on Earth, we would probably treat them very respectfully, because we could afford to, because it wouldn't cost us anything to treat them with respect, because there'd just be a handful of them on Mars. Treating animals on Earth with a comparable amount of respect would be very, very costly for us.

• 43:10Speaker 01

  If we wanted to treat every animal on earth with respect, we would pretty much have to restructure our entire way of life. And that's not something we are willing to do. I, I believe you're a vegetarian.

• 43:25Ezra Klein?

  Yeah, vegan.

• 43:26Speaker 01

  And so, you know, like you are aware of just how much animal cruelty is built into modern civilization. I myself, I still eat meat and I can offer no defense for that beyond, you know, my own preferences. The wider you expand your circle of compassion, the harder it can be to balance your own individual interests with the interests of everyone else in that circle. But yeah, the wider the circle gets, the more entities whose interests you have to balance with yours and, and yeah, it can be, it can be difficult.

• 44:11Ezra Klein?

  This is wonderful. Um, I think that's probably a good place to end. So I'd like to, to ask you a couple of book recommendations before we wrap up. So let me start here. What's a religious text you love?

• 44:21Speaker 01

  I can't really point to a religious, a conventional religious text as a, you know, as an atheist, but I guess I would say Annie Dillard's book Pilgrim at Tinker Creek. That is a book about feeling the, you know, the wonders of nature and experiencing those as a way of sort of being close to the divine. Reading that book gave me maybe the, maybe the closest that I'm likely to get to understanding, you know, a kind of a religious ecstasy.

• 44:54Ezra Klein?

  What is a book you'd recommend on intelligence, artificial or otherwise?

• 44:59Speaker 01

  So there's this computer programmer named Steve Grand and he wrote a book called Creation, which is partly about artificial intelligence, but I guess partly about artificial life. That book, I think, you know, sort of made the most convincing case, I thought, for like, if we're going to actually create something that merits the term. Of being a living thing in in software. You know, I feel like, you know, sort of the the ideas in that book are the ones that I think are most promising. So I guess I'd recommend that. Recently, I read this paper called On the Measure of Intelligence by Francois Chollet.

• 45:42Speaker 01

  And it offers a really, you know, I think useful definition of intelligence, one that can be meaningfully applied to both humans and AIs. And it's it's one that everyone needs to be thinking about.

• 45:56Ezra Klein?

  I am sold. I'm going to read that paper. What's your favorite book of short stories of yours?

• 46:01Speaker 01

  I'm a big fan of George Saunders' first collection, Civil Warland in Bad Decline. Another one, maybe sort of called a novel, but I think of it as a collection of short stories. That's A Visit from the Goon Squad by Jennifer Egan. The stories are sort of linked, but I think the linkages are loose enough that, to me, it makes sense to see it as a collection of short stories rather than a novel.

• 46:27Ezra Klein?

  I love both of those. And I'll note for for listeners that George Saunders was on the show a couple of weeks back, and it's a great episode, and people should check it out. What book or film to you has the most impressive world building?

• 46:37Speaker 01

  There's this Japanese anime film from the from the late 80s called Royal Space Force: Wings of Hanayamis. It's the story of a sort of a space program in this country, which is not Japan. It's not in our world, but, you know, it's a country that is at a somewhat, you know, mid 20th century level of technological development, and they are trying to get into space. And I just loved all the details of the sort of the physical culture of this imagined nation.

• 47:15Speaker 01

  Like the coins are not flat metal discs, they're metal rods. And you know, the televisions, they're not rectangles, they're they're cathode ray tubes are are completely circular. So they're they're watching TV on on on circular screens. All these little details like just, you know, the way their newspapers fold. I just really was impressed by the way that the animators for that film, you know, they invented an entirely new physical culture for this movie. The movie is not about those things, but they really fleshed out this alternate world just as the backdrop for, you know, the story that they wanted to tell.

• 47:56Ezra Klein?

  What's a novel you've read recently that you would recommend?

• 47:59Speaker 01

  So I recently read this novel called On Fragile Waves, Lily Yu. It is a kind of magic realist novel about a family of Afghan refugees who are seeking asylum in Australia. And I found it powerfully affecting. So I think, yeah, everyone should check that out.

• 48:25Ezra Klein?

  So I I saw today when I was doing some final research here that you play video games. Are there some video games you would recommend?

• 48:31Speaker 01

  All right, so one fairly big budget game that I'll recommend is the a game called Control. It's a really beautiful game. I took, I think, probably more screenshots while playing that game than any other game I've played. You play as a character who is visiting the the Federal Bureau of Control, which is this government agency, which is sort of hiding in plain sight. Their headquarters are in this nondescript federal building, which is also known as the oldest house because it's actually a locus of sort of supernatural incursions into our reality. That was a very cool game. I liked it a lot.

• 49:12Speaker 01

  And then a smaller indie game that I really enjoyed was Return of the Oberdin, which is this super cool puzzle game made by one guy, Lucas Pope. I just thought it was an extraordinary achievement. I think it's it's an amazingly inventive game, and it blows my mind that it was made by one person. He even wrote the music for it. It's this puzzle game where you have to identify the circumstances around the deaths of all each of the crew members of this whaling ship. And it is unlike any other game I've ever played.

• 49:51Ezra Klein?

  Ted Chiang, what a pleasure. Thank you very much.

• 49:53Speaker 01

  Thanks for having me.

• 50:00Ezra Klein?

  Thank you for listening to the show. Of course, thank you to Ted Chiang for being here. If you want to support the show, there are two things I would ask of you. Each of them only take a second. Either send this episode to somebody you know who thinks would enjoy it. You can just text it or go on whatever podcast app you're using and leave us a review. It really weirdly does help. The Esther Klont Show is a production of New York Times Opinion. It is produced by Raja Karma and Jeff Geld. Fact checked by Michelle Harris. Original music by Isaac Jones and mixing by Jeff Geld.