Sam Altman on the A.I. Revolution, Trillionaires and the Future of Political Power

“The technological progress we make in the next 100 years will be far larger than all we’ve made since we first controlled fire and invented the wheel,” writes Sam Altman in his essay “Moore’s Law for Everything.” “This revolution will generate enough wealth for everyone to have what they need, if we as a society manage it responsibly.” Altman is the C. E. O. of OpenAI, one of the biggest, most important players in the artificial intelligence space. His argument is this: Since the 1970s, computers have gotten exponentially better even as they’re gotten cheaper, a phenomenon known as Moore’s Law. Altman believes that A. I. could get us closer to Moore’s Law for everything: it could make everything better even as it makes it cheaper. Housing, health care, education, you name it. But what struck me about his essay is that last clause: “if we as a society manage it responsibly.” Because, as Altman also admits, if he is right then A. I. will generate phenomenal wealth largely by destroying countless jobs — that’s a big part of how everything gets cheaper — and shifting huge amounts of wealth from labor to capital. And whether that world becomes a post-scarcity utopia or a feudal dystopia hinges on how wealth, power and dignity are then distributed — it hinges, in other words, on politics. This is a conversation, then, about the political economy of the next technological age. Some of it is speculative, of course, but some of it isn’t. That shift of power and wealth is already underway. Altman is proposing an answer: a move toward taxing land and wealth, and distributing it to all. We talk about that idea, but also the political economy behind it: Are the people gaining all this power and wealth really going to offer themselves up for more taxation? Or will they fight it tooth-and-nail? We also discuss who is funding the A. I. revolution, the business models these systems will use (and the dangers of those business models), how A. I. would change the geopolitical balance of power, whether we should allow trillionaires, why the political debate over A. I. is stuck, why a pro-technology progressivism would also need to be committed to a radical politics of equality, what global governance of A. I. could look like, whether I’m just “energy flowing through a neural network,” and much more. References: “Moore’s Law for Everything” by Sam Altman Recommendations: Crystal Nights by Greg Egan The Last Question by Isaac Asimov The Gentle Seduction by Marc Stiegler “Meditations on Moloch” by Scott Alexander If you enjoyed this episode, check out our previous conversation “Is A. I. the Problem? Or Are We?” You can find transcripts (posted midday) and more episodes of "The Ezra Klein Show" at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Thoughts? Guest suggestions? Email us at [email protected]. “The Ezra Klein Show” is produced by Annie Galvin, Jeff Geld and Rogé Karma; fact-checking by Michelle Harris; original music by Isaac Jones; mixing by Jeff Geld, audience strategy by Shannon Busta. Special thanks to Kristin Lin.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Ezra Klein?

  I'm Ezra Klein and this is the Ezra Klein Show.

• 0:20Ezra Klein?

  So a few months ago, I came across this really fascinating essay by Sam Altman called Moore's Law for Everything. Altman is the CEO of OpenAI, which is one of the biggest and most interesting of the companies trying to create general purpose artificial intelligence. So an AI that can improve itself and learn and actually do all sorts of things in the economy and in the world. I've met him a few times and he is a believer. He sees a world coming. He believes he is bringing a world into existence that is really, really different than the one you and I know, and that it's going to happen fast. And who knows, maybe he's right. But, but what caught my eye about this essay, Moore's Law for Everything, is Altman's effort to try and imagine the political consequences of true artificial intelligence

• 1:03Ezra Klein?

  and the policies that could decide whether it ushers in utopia or dystopia. So Moore's Law is the observation, the number of transistors on microchips doubles every two years. Or to put it more simply, the computer power has been growing exponentially for decades now, but prices have actually been falling. That's not been the case for housing or healthcare or higher education. But what if it was? Altman's argument is that true AI could get us closer to Moore's Law for Everything. It can make everything better, even as it makes everything cheaper. And I have some points of skepticism here. You're going to hear them in the conversation, but, but maybe he's right. I hope he's right. That would be good. But if you grant his argument, if he is right, then he says that the world looks like this.

• 1:47Ezra Klein?

  AI will create phenomenal wealth, but it'll do so by driving the price of a lot of labor to basically zero. That is how everything gets cheaper. It's also how a lot of people lose their jobs. To make that world a good world for people, to make that a utopia rather than a dystopia, it requires really radical policy change to make sure the wealth AI creates is distributed broadly. But if we can do that, he says, well, then we can improve the standard of living for people more than we ever have before in less time. So Altman's got some proposals here for how we can do that. They're largely proposals to tax wealth and land. And I push on them here. But more than that, I want to push on and interrogate the political economy behind them.

• 2:29Ezra Klein?

  Because if he's right, and I assign a real possibility that he is right, at least to some degree, then AI, it's not just going to redistribute wealth and jobs. It's going to redistribute power. And these people who will then have so much more power, they're going to offer themselves up for more taxation. They're going to give up parts of their company every year and their wealth and do it willingly? I'm not sure. So I want to offer this warning before we get into the show. If you don't spend a lot of time in the AI debates, you're going to hear some stuff in this conversation that just sounds weird to you. And I think it's important not to get scared off by that. Go back over the past few hundred years. There are times when technology just upends society.

• 3:12Ezra Klein?

  We may or may not be on the cusp of one of those times now. But on the chance that we are, even if it's only 50% as much as Altman thinks, then it is important to think about the politics of that world, to try to be ahead of it, not just catching up to it. And that's what this conversation is about. As always, my email is [email protected]. Sam Altman, welcome to the show.

• 3:37Speaker 01

  Thank you for having me.

• 3:38Ezra Klein?

  So let's start where we are now. What can AI do now?

• 3:43Speaker 01

  So I think we have just begun the realm of AI being able to be what we call general purpose AI, instead of these narrow models that can do one thing. So now we have models that can understand to some sense, to some definition of that word, the world, what's going on. And then a single model can accomplish a wide variety of tasks for someone and can learn new things pretty quickly. So last year, OpenAI released something called GPT-3. It's a large text model. And that one model is now being used by thousands of people, developers, more than that in terms of end users, for all sorts of tasks. And I think we will see In the coming years, AI medical advisors that give anyone like really high quality medical advice, higher than they can maybe get anywhere in the world, AI tutors that can

• 4:28Speaker 01

  teach you math or other topics, we're starting to see people develop AI programming assistants that can help you write code. And these same models are being adapted for all of these different uses. And I think we're starting now, heading into a world where a lot of the things that people want, they can, through a text interface or a visual interface, have an AI help them do.

• 4:51Ezra Klein?

  As I understand what these systems are doing now, they're predictive. Like, how, how is that intelligence?

• 4:56Speaker 01

  So let, let's think about what you need to make a smart agent. You need to understand what's going on in the world, and you need to be able to think of new things, and you need to be able to take action to accomplish goals. In a very deep sense, I think the biggest miracle that we need to create this super powerful AI is already behind us. It's already in the rearview mirror. We just, we needed an algorithm that could learn and a network architecture that could somehow encode the knowledge and the capability to reason. And then, once we had that, however imperfect it is, if you make, apply it to bigger and bigger systems with more and more computing resources, there's no obvious

• 5:37Speaker 01

  upper limit. Once you have a system that can take in observations about the world, learn to make sense of them, and one way to do that is to predict what's going to happen next, the, I think that is very near intelligence. So we have the ability to build systems that can learn, update, remember, create. We have the ability to build agents that try to accomplish a goal with that knowledge. And, you know, we've gotten surprisingly far just putting those few simple ideas together.

• 6:05Ezra Klein?

  And so if I basically understand how GPT-3 works, it's a system that has read a lot of stuff on the internet, and it's predicting the next word in the sequence.

• 6:15Speaker 01

  Slightly oversimplified, but very close. Yes, it is trying to predict what comes next in a sequence.

• 6:20Ezra Klein?

  And so to what you're saying, then? We've figured out algorithms that can learn.

• 6:24Speaker 01

  Yep.

• 6:25Ezra Klein?

  And then as they learn and as they know more, then as we are able to ask them to orient their predictions in different directions, like that's basically

• 6:34Speaker 01

  Yeah.

• 6:35Ezra Klein?

  a generalized intelligence that we could

• 6:36Speaker 01

  And there's

• 6:37Ezra Klein?

  that we could use.

• 6:37Speaker 01

  There's no obvious upper bound to that process. Like, think about how a kid works. They observe the world, they start to learn, they clearly make some predictions and you can see when like something doesn't go that the way they think it's going to go. And then they update and the next time they're not surprised, or maybe they even can come up with new ideas based off of that. And you can also teach them stuff, you can correct them. And this idea of giving human feedback to a neural network in the way that we've given to our children for a long time, where we say, no, that's right, no, that's wrong. And the model, the AI or the kid or whatever is updated each time that happens. But then the, the, let's say the agent can make better predictions and do better and need to get corrected less and less on increasingly subtle issues.

• 7:20Speaker 01

  There is no upper bound how far that can go as you think about increasing the skies and scale.

• 7:25Ezra Klein?

  I obviously don't know how to use GPT three the way, the way you do or what's coming after it. But it sometimes seems to me that the, the harder thing to see from the outside of where all these are going is how flexible they are or they will become in being able to understand the way I want them to make a prediction. Right? So there's a lot that is being predicted, but right now it's sort of a mimicry, like do stuff that is like the other thing.

• 7:49Speaker 01

  Have you tried our newest instruction following model by chance?

• 7:53Ezra Klein?

  I'm not sure.

• 7:54Speaker 01

  You should. I think we've released something like that just quite recently, where you can say, here's what I want you to do. And it will do it.

• 8:01Ezra Klein?

  What have you done with it? What can it do?

• 8:04Speaker 01

  A lot of, it's still very simple, but a lot of simple tasks. You know, if you say like,

• 8:10Speaker 01

  summarize this for me or translate this from into French, no matter what language it's coming in as, or like even like a small multi-step problem where you need to like manipulate natural language, you can just say what you want it to do. And you could even say like, I mean, you could probably say, write this in the style of Ezra Klein and it would probably just do it.

• 8:28Ezra Klein?

  What is the way AI changes the economy or changes people's lives without using the word AI? What is the force it exerts on economic activity?

• 8:38Speaker 01

  It's a great question. There's a lot of different ways to answer that. But this idea that things that humans used to have to do that maybe they didn't want to do, that took a lot of time, that took a lot of effort, that weren't done that well, automated systems that can do those better. So we saw this happen with translation, for example, where if you couldn't afford a translator, which most people couldn't, and you were going to like wander around some foreign country. I remember what that was like before we had AI translation. And I know what it's like now where I just speak into my phone and it speaks out the other language. And I love that. Like that, that really changes my experience of traveling. We're beginning to see that now with certain subsets of the medical field, where AI assisted doctors can do better than doctors on their own. Or let's say, computer assisted doctors.

• 9:23Speaker 01

  But this idea that you can have intelligence, smartness, creativity, whatever you want to call it, that a computer is doing to help you do the things you want and have a life you want at a marginal cost of zero or very close to it. That's, I think, a transformative thing happening in the world.

• 9:39Ezra Klein?

  So that last piece is a piece I want to focus on, that marginal cost of zero. So the idea is that as machine learning, as these computer systems become better, basically, everyone will have at their disposal sort of a staff, like a corporation unto themselves, where they will be able to hire systems to help them out

• 9:58Speaker 01

  Yes.

• 9:59Ezra Klein?

  for almost nothing.

• 10:00Speaker 01

  Yes.

• 10:00Ezra Klein?

  Of course, that means you were framing this a couple of minutes ago in terms of jobs people maybe don't want to do or don't do that well. But that also means jobs people do want to do or did do reasonably well, but not as well, go away.

• 10:14Speaker 01

  That's been happening for a long time. And I don't think we do a service to anyone to pretend the technology does not eliminate some jobs. Some jobs, it just like makes much better. But the arc of this has been that every technological revolution eliminates one class, one sets of jobs. And we find new ones on the other side that are hard to imagine from where we sit today. So before we had the computer, it would have been hard to imagine the computer programmer would be such an important profession. But I think if you look back at the great technological revolutions, which have sort of been the punctuations where there's been a lot of shift at once, there's always been this worry. We've always found other jobs on their side. Now, it may be that this time it's different, right? Like, if we really do think about what it means to have intelligence in a computer,

• 10:58Speaker 01

  maybe it's different, but I so deeply believe that human ingenuity and sort of desire for ever sillier kinds of status is so unlimited that we will find a lot of new things to do. I also think what we are likely to find is that a lot of classes of jobs that people talk about AI taking away stay, but the role of the human is very different. And the human does what humans are really good at. The AI does the part that the human might like to do less and the AI is really, really good at. And you see this sort of like synergistic effect, I hate that word, but I couldn't think of something else, where humans continue to be programmers or doctors or whatever.

• 11:34Ezra Klein?

  Journalists, maybe.

• 11:35Speaker 01

  Journalists, but they focus on a different part of the job.

• 11:39Ezra Klein?

  Two places I want to go with that. One is, I'm usually on the side of this argument that says, I don't think machine learning is going to bring a near-term job apocalypse. But I've become friendlier in the past couple of years to the idea that it will destroy a lot of jobs pretty quickly. And to me, it gets to the thing you were saying about generalized systems. In a lot of past revolutions, the revolutions, the technological changes, they happen fairly slowly and practically their dispersal through society is pretty slow. And then second, they are general over time, but they're not general all at once. Now, if you think of machine learning as part of, and partly what we're talking about here, as part of a long swing of automation, it isn't all at once. But as I understand the way you see the world,

• 12:24Ezra Klein?

  you think this is going to happen in a much more punctuated period than, say, electricity?

• 12:29Speaker 01

  A couple of years ago, if you talked about general purpose AI at all, people said, that's ridiculous. It's not happening. If you talked about systems that could really do meta-learning and learn new concepts quickly that they weren't trained for, people said, that's not going to happen. And we've gone from a world where many of the experts in the field said that was sci-fi and irresponsible to talk about, to like clear existential proof that we have it. And it certainly doesn't seem to be slowing down. Moore's Law. In varying definitions, but let's say that was like a doubling of transistors every two years. Maybe AI is sort of growing at a rate of like 10x per year in terms of these these model sizes and the associated capabilities. So I do think we're on a very steep curve.

• 13:14Speaker 01

  We will hit limits, but we don't know where those will be. We'll also discover new things that are really powerful. We don't know what those will be either. We're like deep in the scientific discovery phase, which is awesome. It's like so exciting and fun. But I think what we can say is that we are on an exponential curve. And when you're on an exponential curve, you should generally, in my opinion, take the assumption that it's going to keep going. And humans are very bad at intuition for this, like trying to think about exponential curves for stock prices, for technology, for population growth, whatever.

• 13:46Ezra Klein?

  Viruses.

• 13:46Speaker 01

  Viruses. Very difficult. I had this moment that I'm sure you had one too, when it looked like COVID was really going to take off and most of the world wasn't paying attention. And I was walking the streets of San Francisco one night at like 10pm, walking home. And everybody was just like frolicking and doing their thing. And it was just like, this is the last moment of normalcy and no one's paying attention because most people don't understand exponential curves. And it was such a strange feeling. And I've often thought about the parallels of that moment that hit so viscerally with AI.

• 14:17Ezra Klein?

  My version of that, I remember sending an email to my entire family and saying, you need to go to the grocery store. Like in three weeks, you're going to be living in a different world.

• 14:26Speaker 01

  Yeah.

• 14:26Ezra Klein?

  And like, you need to listen to me on this. Let's talk about where exponential growth gets us in. You're saying there's a 10x-ing every year. It's pretty big.

• 14:35Speaker 01

  Yeah.

• 14:35Ezra Klein?

  20 years from now, what are you confident AI will be able to do? And what do you think it is likely that it will be able to do?

• 14:42Speaker 01

  Man, 20 years is a hard timeframe.

• 14:44Ezra Klein?

  Let's do 10.

• 14:44Speaker 01

  Let's do 10. 10 I can do. In 10 years, I think we will have basically chatbots that work for an expert in any domain you'd like. So you will be able to ask an expert doctor, an expert teacher, an expert lawyer, whatever you need, and have those systems go accomplish things for you. So you're like, I need a contract that says this. You know, I need a diagnosis for this problem. I need you to go like, book me this flight. I want a movie created. I want you to like make me, you know, an animated short or a photorealistic short that looks like this. I need you to help me write this computer program. So let's say most repetitive human work.

• 15:26Speaker 01

  And some creative human work you will be able to ask an AI to do for you. And that is like a massively transformative thing.

• 15:35Ezra Klein?

  I want to get at whether we are talking about a singular when we talk about an AI system or a plural. So the natural metaphor in people's mind will be a software program. I buy some stuff from Microsoft, I buy some stuff from Apple, I buy some stuff from all kinds of developers whose names I forgot. My understanding of the way the AI systems are developing is that it's turned out using the the sort of big machine learning neural networks that have turned out to be successful. You need ungodly amounts of data and amounts of computer power. And that means only a couple groups are big enough to build these general intelligence systems, assuming they do get built.

• 16:14Ezra Klein?

  And so is the world you're talking about here a world in which OpenAI and maybe a couple other players have a system that is licensed out and everybody is building on top of that? Like, is it all just coming from you? Like, what is happening here economically?

• 16:30Speaker 01

  I think it's going to be somewhere in between these sort of narrow models that anybody creates and these like mega models that only a few people can create. There's this concept of fine tuning. So where someone like OpenAI creates this powerful base model that only a few organizations in the world can do. But then maybe you want to use that for a chatbot or a customer service agent or like a, you know, creative multiplayer video game. And you take that base model and then with just a little bit of extra training and data, you push it in one direction or the other. So I could easily imagine a world where a few people generate these base models. And then there's the medical version, the legal version, whatever else that get sort of fine tuned or polished in one direction or another. And a lot of people, a lot more people are capable of doing that.

• 17:16Speaker 01

  We're starting to experiment with offering that to our customers now. But I could see a world, and I think it like does no one any good to pretend otherwise, where as these models get really smart, the general purpose one can just do everything really well. And this idea that we think right now, that OpenAI thinks, which is we're gonna like, you know, push one to be a coding expert and one to be a medical expert, turns out not to be necessary because. 10x compound is just so powerful that, you know, 10 to the 10th, 10 years from now, like the base model is plenty good at everything.

• 17:44Ezra Klein?

  Staying on the idea of the big base models, as I understand it, there are a little handful of companies who are in the race to build one of these in a serious way. You're one of them. But DeepMind is another. There's a lot of work happening in China.

• 17:58Speaker 01

  Yeah.

• 17:59Ezra Klein?

  Players in America like Amazon and Apple are trying to create, maybe not always general, but but pretty big things. How should I feel about the idea of these will be privately owned if they're going to be this powerful?

• 18:11Speaker 01

  Strange. If we think about the major big iron engineering projects of the past, like the ones where sort of there were large geopolitical and certainly social consequences, you know, just to pick two examples to talk about here, we could talk about like the Apollo project, the Manhattan project. You can like find different cost estimates of what those were. And like, who knows what you want to believe, but let's say something with the word 100 billion in it. Those really took the wealth of nations to do, like no, no company like us was going to do that, right? But like, the, you know, cost of technology does its thing. Companies and people kind of get wealthier over time. You now do have

• 18:52Speaker 01

  a world where certainly the mega caps like Google can join this effort. But even like much smaller organizations like OpenAI can get enough capital together, barely, to be able to be competitive here with what anybody else can do. So that's weird. I have misgivings about that. Probably like the non-technical thing I think most about is, let's say we do make the true AGI, like the one from the sci-fi movies.

• 19:21Ezra Klein?

  The artificial general intelligence.

• 19:23Speaker 01

  How do we want to think about how decisions are made there? How it's governed? Who gets to use it? What for? How the wealth that it creates is shared? If this is going to be like one of these species defining moments, it for sure should not be in the hands of a company, certainly not ours. But saying what we want the structure to be there, how we want to make decisions about it, what the equivalent of our constitution should be, that's like new ground for us and we're trying to figure it out now.

• 19:49Ezra Klein?

  OpenAI begins as a nonprofit.

• 19:50Speaker 01

  Yeah.

• 19:51Ezra Klein?

  It becomes a for-profit in part because it needs to raise money and resources. So it got a billion dollar investment. It's partially money, partially compute power from Microsoft.

• 20:00Speaker 01

  Actually, it was all in cash, but we spent most of it on compute.

• 20:03Ezra Klein?

  Oh, there you go.

• 20:04Speaker 01

  So close enough.

• 20:04Ezra Klein?

  Partially on Microsoft computing power, correct?

• 20:06Speaker 01

  Yeah, yeah, yeah.

• 20:07Ezra Klein?

  One of the worries I have about this is that even if people want to be very cautious about what the incentives of it are, that just in order to do it, you have to submit to those incentives, that just in order to raise the money, there has to be a business model, a backer. And I was reading that and I wondered this from a different direction, too. Was that a missed opportunity for the public sector? Should it be that the public sector is spending the money to build this, either by funding groups like yours or consortium of academic groups or something?

• 20:39Speaker 01

  Little known fact, we tried to get the public sector to fund us before we went to the cap profit model. There was no interest. But yeah, like I think if the country were working a different way, I would say a better way, this would be a public sector project, but it's not, and here we are. And I think it's like important that there is an effort like ours doing this, that even if not a kind of an official American flag effort, will represent some of the values that we all hold dear. That's better than a lot of other ways I could imagine someone else doing this project that was going. And one of the incentives that we were very nervous about was the incentive for unlimited profit, where more is always better.

• 21:21Speaker 01

  And I think you can see ways that's gone wrong with profit or like attention or usage or whatever, where if you have this like well-meaning people in a room, but they're trying to make a metric go up and to the right, some weird stuff can happen. And I think with these very powerful general purpose AI systems in particular, you do not want an incentive to maximize profit indefinitely. So by putting this voluntary cap on ourselves, above which like none of the employees or investors get any more money, which I think if you do have a powerful AI will be somewhat trivial to hit, I think we avoid the worst of the incentives, or at least the one that we were most worried about.

• 21:58Ezra Klein?

  How about speed? So there's an incentive to get there first. There's going to be huge financial returns and otherwise returns to being the first one. You're to some degree in a race with other companies to do it. I'm not saying that leads to cutting corners, but it leads to things where maybe you'd ideally want to wait for a governance structure to emerge, want to wait for a public conversation to happen. Well, if you don't do it, the other folks will. And one of the constant ways things get justified in both government and business is it better us than them?

• 22:28Speaker 01

  For sure. So I think we were able to design a system that addressed a lot of the incentives that I was particularly concerned about. The one that remains that I am, for the entire field, not just us, most concerned about is actually closer to the super powerful systems, like the ones that people talk about creating existential risk to humanity, where there's a race condition. And that, I think, will be on us and the other players in the field to put together a sufficient coalition to stop ourselves from racing when sort of like, safety is in the balance. And we're trying to figure out how to do that. Like, that's part of the governance question. Before you kind of push go on this like extremely powerful system, you would like as much time as you can get, and it won't be totally in your control, right? Because some other government can be doing whatever they're going to do.

• 23:13Speaker 01

  But you'd like as much time as you can have to be really thoughtful about, do we understand what the system is going to do?

• 23:19Ezra Klein?

  And how do you get that time?

• 23:21Speaker 01

  You have people partner and say like, okay, lots of other industries have done this. I think the recombinant DNA conversations in the seventies are a good example. But you say like, we're the experts on this. We're the set of companies with the resources to do this. How can we work together to make sure we all have the time we need?

• 23:38Ezra Klein?

  Is there a different kind of pressure that comes from the geopolitical push? It's interesting me use a metaphor of like an American flag operation. So there's a competition between you and other American companies. And then there is the sense that there's also a competition from China, potentially certainly down the road, other countries. That can create a different kind of pressure. It could even be a pressure coming from the public sector, but a pressure to finish first.

• 24:00Speaker 01

  This is something that like some parts of the field have spoken about for a long time, which is, you know, sure, the private sector companies can do whatever they want. But what if there's huge public sector pressure? And that, I don't think we'd be in a position to have too much of an impact on.

• 24:16Ezra Klein?

  You mentioned before the cap on profits, which I believe for you all is 100x.

• 24:21Speaker 01

  It started as that. It's come down every time we've raised more money. So it's like, single digits now.

• 24:25Ezra Klein?

  So.

• 24:27Ezra Klein?

  There's a question of how much money people can make and there's a question of how money gets made.

• 24:30Speaker 01

  Yeah.

• 24:31Ezra Klein?

  And I've been thinking a bit about that as an important incentive question. So I'll use your competitors over at Google as an example here. I worry a bit that if the big AI system is built by Google, it's going to be oriented towards being very, very, very good at manipulating consumer preferences. Right? It's an advertising-based business. Other great, interesting internet companies ended up building themselves around manipulating how people spend money online. I wouldn't like to see something as powerful done that way. But Microsoft has other ones. You guys would have other ones. It got me thinking about whether or not there are ways to shape the direction these systems are tuned in, in useful ways. Like, for instance, you could imagine a prize system where the public sector puts out multi-billion dollar prizes

• 25:15Ezra Klein?

  for solving this technological problem, this scientific problem.

• 25:18Speaker 01

  Yeah.

• 25:20Ezra Klein?

  Tell me a bit about this sort of end game business models here, what they are turning out to be and what the options for them are.

• 25:26Speaker 01

  So I can't speak to what Google's going to do other than I probably won't like it.

• 25:32Speaker 01

  But I can tell you how we're thinking about it and I hope other people will too. What we're excited about doing is the best research in the world and trying to build this eventually quite powerful general purpose system. What I think we're not the best in the world at, nor do we want to really kind of divert our attention to, are all of the wonderful products that will be built on top of this. And so we think about our role as to figure out how to build the most capable AI systems in the world and then make them available to anybody who follows our rules to build all these systems on top of them. We have thought about prizes, like you can imagine saying whoever, you know, like here's this API, do something great for the climate and here's like an incentive to do it or any number of other categories.

• 26:17Speaker 01

  But I do trust that on the whole, if we can deliver super powerful new tools at an affordable price, the market will do its thing and we will get wonderful solutions to some of the most pressing problems facing humanity.

• 26:31Ezra Klein?

  So you don't think pro-social goals need to be built in by something explicit? You think that, you think certainly once you can just ask questions, at least those questions will be asked and they'll be answered.

• 26:40Speaker 01

  Yeah, I mean, I think we're going to do it too, because it seems like a good thing to do. But I do think that even without that incentive, a lot of great pro-social things will happen.

• 26:50Ezra Klein?

  So you have a version of this. You say that the rallying cry for this generation should be a Moore's Law for everything. Tell me about that idea.

• 26:57Speaker 01

  Look, it's not a popular moment to say something positive about capitalism by will anyway, which is that I think the power of capitalism has done more to help more people than probably any other single idea in human history. And the innovations, the new technology that we have both developed and gotten into the hands of people and the sort of almost unimaginable improvements to quality of life, even though it's like fun to talk about how awful the world is and is in many ways. That's great. And part of the reason that has happened is things have gotten much cheaper. You know, pick your example, the cost of an hour of light at night, the cost of computing power, the cost of

• 27:43Speaker 01

  so many things are now free that like when we were kids were not entertainment, high quality education. You can go learn anything you want on the internet so well. It's amazing to me.

• 27:52Ezra Klein?

  My family, when I grew up, a lot of it was in Brazil and long distance calls were a big deal.

• 27:56Speaker 01

  Yeah.

• 27:57Ezra Klein?

  And now it's like every morning my kid is on FaceTime with his grandparents.

• 28:01Speaker 01

  Yeah. And this relentless march of technology making the things that we need and want cheaper and cheaper and cheaper. That's great. Like that's how people's lives get better every year and all of us want our lives or almost all of us want our lives to get better every year. I think one of the big promises of AI is because of this idea of the marginal cost of labor going towards zero, things can get dramatically cheaper quickly and way more of us can afford way more of the things that we want. And for a generation that has had a lot of, I think, economic headwinds and sort of the deck stacked against them. I think this is like a specific thing we can imagine that gets a lot of people's lives way better pretty quickly.

• 28:43Ezra Klein?

  So I want to make this specific. When you talk about a Moore's Law for everything, things that are expensive that millennials, zoomers are having trouble affording when it was sometimes easier, sometimes also not for their parents. But housing is very, very big in this. Healthcare is big in this. Higher education is huge. Those three are big. And I would say housing is primarily a zoning problem.

• 29:06Ezra Klein?

  In health, technology and the cost of technology is a real player there. And then there's a fair amount of regulatory questions in both of them. So how does, how does building AI allow for a Moore's Law in health, education and housing?

• 29:20Speaker 01

  So I think technology is a necessary, but not sufficient part of the solutions here. I think that if we can't get the politics and the policy right, it will face a lot of headwinds. I can still imagine it working on its own in some cases. I think there's like a lot of people who are paying $75,000 a year for college right now, sort of sitting at home on Zoom, realizing that they can like watch better lectures than they are. And I think a lot of things in higher ed are going to change no matter what the policy is. But I think if AI and policy can work in harmony, or let's even say like technology and policy can work in harmony,

• 30:00Speaker 01

  we can get to a much better outcome much more quickly than we otherwise might. And I think like, again, to stick on the example of higher ed, I have never seen more energy from the consumers, like the people who are going to college or thinking about going to college soon, saying like, what am I really doing here? And maybe there's like some much better way. So that's the kind of pressure where I think things can change super quickly. The prestige of a degree from a top college probably doesn't go away that quickly, but maybe a lot of the rest of it does.

• 30:30Ezra Klein?

  How about housing?

• 30:32Speaker 01

  I think it is time to consider radically new ideas there. I mean, it's just so sad to me to watch what San Francisco is doing to itself. I think it's so incredibly unfair. Like, it really bothers me that it's people who would describe themselves as liberal and progressive that are most in the way of, I think, like the justice that comes from affordable housing and access to opportunity with that in the places where it matters most. And I think we should continue to fight that as hard as we can. But we should also realize that like, maybe we're dealing with people here who are not reasonable actors. And there may be other cities where it's just much easier to solve this problem. I think like, there's been a mini boom in new real estate startups

• 31:16Speaker 01

  as people in the pandemic have thought about leaving the big cities, where people can like realize like, wow, I can get a, I can build a new house in some other city for not that much money and I can get exactly what I want. And I can now do it quickly with technology, surprisingly affordably. And maybe a competitive market will do its thing here.

• 31:36Ezra Klein?

  You're giving me, though, the answer here that I give to people when I'm in a techno skeptic kind of mood, which is, well, the problem in our way on a lot of things is not that the technology isn't there, it's that the political will isn't there, it's that the structure isn't there, it's that there are other impediments. So you have this piece about how AI is going to move all this power from labor to capital and it can transform humanity way for the better or not. That's sort of where the Moore's Law for everything idea comes in. What is the role of the system you're building here? How's it going to help housing or any of this?

• 32:07Speaker 01

  I mean, technology has already done a lot to help housing, I would say. And one specific thing that it did was enable remote work. And so, like, fine, you know, like, a city like San Francisco is not going to be reasonable on policy. People can live elsewhere and still work for the most important companies in the industry that have been here, and they are freed from some of the geographic constraint. Look, remote work certainly does not work for everything, and it's not something that I'm personally excited to do. But I think it does work for a lot of things and a lot of people. And I think that's an example of where faced with a seemingly intractable political reality, technology produces a

• 32:48Speaker 01

  solution that is a total curveball and not that imaginable, you know, a few decades ago.

• 33:16Ezra Klein?

  Tell me about the underlying idea that AI will move not just money, but also power from labor to capital.

• 33:24Speaker 01

  I think the best way to frame this is this idea that the marginal cost of an AI doing work is close to zero. Once you've created this model, which requires huge amounts of capital and expertise and difficulty and data to do, and I think it's like a very interesting question about who should benefit from that if the, you know... Who generates the data, whatever. But once you train this model, maybe you used to have to pay an expert lawyer $1,000 an hour to answer a question, or a computer programmer $200 an hour. And there weren't that many, and they had a lot of, you know, like, you needed it. And that was the market. That was what it was worth. And that was what people were able to command. But maybe now it costs like, a couple of cents of electricity for the computer to think, or less. And you can do it as many times as you want. You can

• 34:08Speaker 01

  sort of get to answers that no human could come up with. Labor, then, in this case, like extremely high skilled and highly paid labor, all of a sudden has a lot less power because the services are available at a wildly different cost.

• 34:22Ezra Klein?

  So you write in the piece on that three things follow from that. One is that phenomenal wealth can get generated because goods and services can become so cheap and so widely deployed, that very rapid social change follows that because you're just undergoing an extraordinary upheaval in the economy. And so then you have this kind of branching possibility, which is either very high standards of living for everyone, or not, right? Sort of, some people get like better entertainment and whatever, but they never get back the dignity, they never get back the money they were making. How do you assess the political economy of AI?

• 35:01Speaker 01

  I'm curious if you agree with sort of the first two, if you think the first two things are going to happen.

• 35:05Ezra Klein?

  Phenomenal wealth and social upheaval?

• 35:07Speaker 01

  Yeah.

• 35:09Ezra Klein?

  If you're right about what the systems can do, definitely that will create wealth, right? I mean, it just has to. I think it's almost mechanical. Upheaval, to me, is about how quickly it happens. And I suspect it could be technologically possible far before it is actually happening, or that the dispersion of the technology might be a lot slower than the, than the realization of it. So you mentioned a minute ago, lawyers.

• 35:37Ezra Klein?

  You ask lawyers questions because you need answers. You also ask lawyers questions because if you don't get the answers from them, you might get sued. And so the question of when do you not get sued for asking the question of your system as opposed to of the lawyers is to some degree a regulatory and judicial issue, more than it is even a technological one.

• 35:56Speaker 01

  Yeah, that's sort of why I assume there will be these like hybrid models where you're, you know, an AI is helping the lawyer, but the human lawyer and the existing system of the protection that you get from that is still the one giving you the answer. So maybe what happens is like, you still go to a lawyer, but that lawyer can be a factor of 10 or 100 times more efficient and they don't need sort of the staff of researchers they have today.

• 36:19Ezra Klein?

  So I could see that and I think that makes sense, but that'll reduce the amount of economic upheaval it brings. Health is a really big one. I've done a lot of work on healthcare over the years.

• 36:28Ezra Klein?

  I find it very hard to think through what the effect will be here, but let me put it this way, I've got a long running argument that we think too much about price and health and not enough about value. So budget work in Washington is like always dominated by these graphs that show like by 2070, you know, 50% of the economy is on healthcare. And people will say like, that's wild. We can't have 50% of our GDP go into healthcare. But how wild it is depends on what that is buying you, what the value of it is. If we're spending 40% of GDP on healthcare, but everybody's living a healthy life till 200, maybe it's fine. If we're getting what we're getting now, it's terrible. And so one of the questions I don't really know how to answer is, does AI, whether or

• 37:11Ezra Klein?

  not it is being used with doctors and nurses or, you know, otherwise, does it actually change radically the development of drugs, the sort of device orientation and devices we're able to use? Do we get way better at spotting things on radiology, which I think is pretty likely? So how you change the value? You might not see that show up in wealth statistics exactly, but it would be a huge improvement in utility and in how humans live. And so like, that stuff is very interesting.

• 37:40Speaker 01

  I think wealth is like definitely an imperfect metric and I think price is even more imperfect. I totally agree with you, by the way, like if we had like a magic pill that everybody could take and have an extra 20 years of perfect health span and that cost 40% of GDP or whatever, maybe it's fine. So I think these metrics are all imperfect. And what we really want is like someone's overall quality of life and happiness and fulfillment and contentment and whatever. But I think for like something numeric that we can measure today, wealth is the best thing. And I do think in terms of the branching that you talked about somehow or other, this is maybe sort of a techno optimist in me, almost everyone's lives are going to get better. People will demand it. And the question is, what form is that going to go in?

• 38:24Speaker 01

  I don't actually subscribe to the sort of Silicon Valley, UBI will solve all problems. We can just do that and stop talking about it. I think it's actually a small part of the solution, but I think we're going to do it. I think somehow or other that's going to happen. But for me, then the question is like, how are we going to construct the rest of society in a way that people have the dignity, the sort of ability to decide where, where we all collectively go that, that is so important.

• 38:48Ezra Klein?

  Well, let's talk about UBI for a minute because the place you go with this in your, in your piece is toward something you call the American Equity Fund. I think actually a useful way to think about it is universal distribution of wealth.

• 39:00Speaker 01

  Yeah.

• 39:01Ezra Klein?

  Talk through that idea.

• 39:03Speaker 01

  A thing that I did not understand when I was a kid is the difference between salary and equity ownership. And I sort of like assumed that the way people got rich was salary. And I kind of knew that people like traded stocks, but I didn't think it was like a big thing and not that important. And it was basically like about, you know, you get paid monthly. And, and, and really I think the way that value gets created is the compounding effects of equity, basically. So like, that's the pro case on capitalism is that that works really well. The negative case is most people don't own very much equity or, or land. Like I, I sort of said in the piece that I think land and shares of companies are going to be the two dominant sources of wealth in the future.

• 39:45Speaker 01

  And so better than like a government sort of paying you some fixed fee every month is where citizens of a country own a slice of the important asset classes of the future. And that that's how wealth gets redistributed, which I think is different than redistributing. The equivalent of salary, much more important.

• 40:05Ezra Klein?

  I probably would agree with that. The problem I see, and this is why I asked very specifically about the political economy of AI. So i- in this piece, you make an argument about how it'll change sort of the distribution of money.

• 40:17Speaker 01

  Yeah.

• 40:17Ezra Klein?

  And, you know, how could you tax around that? But this is one of those places where power really matters. And so another way of saying what you're saying is that AI will make owners of capital and land much more powerful. And we are then somehow going to tax the owners of capital and land much more aggressively. And one of my certainly concerns about the world being sketched here is that my sense as a political person is you would somehow need to get a more equitable distribution of power in order to have the more equitable distribution of resources.

• 40:50Speaker 01

  Totally agree with the statement about the equitable distribution of power being the important thing. Unlike paying cash transfers, I think if you're actually like transferring shares in these companies and like the accumulating wealth there, or the ownership of land or whatever else, that actually transfers some of the power over time, too.

• 41:12Ezra Klein?

  But in order to get to that policy, you need the power to transfer first.

• 41:16Speaker 01

  Yeah, here's where we're deeply out of my area of expertise. So I will ask you, what do you think it takes to get something like this?

• 41:22Ezra Klein?

  I don't know. I'm not sure you can.

• 41:25Ezra Klein?

  We were talking about this before we started the show, but but something I've been starting to try to think through is how how do you have a technologically aware progressivism? Have you ever read the book Fully Automated Luxury Communism?

• 41:37Speaker 01

  I've heard the phrase.

• 41:39Ezra Klein?

  The argument it makes is that the set of technological changes coming down the pike, of which AI is central to that, but it's not the only one. They actually make a case for a much more radical form of public ownership. He's using communism a little bit facetiously, but forms of communism, of socialism, make a lot of planning, of ownership, make a lot more sense under the technological structure that's coming. But the question then becomes, in order to have that happen, I think you you actually need to set up the politics before the technology hits, because if the technology hits first and then it just pools power among the people who own it. Well, power increases its own wealth and increases its own power.

• 42:22Ezra Klein?

  And then particularly, we've got these crazy AIs that are, you know, out there trying to manipulate public opinion, whatever else. It can get a little bit dystopic. And it's not all going to happen overnight, but I do think this is a real, it's a real question for me. And it's one reason I've become, in some ways, what I want to see is a politics on the left or on the progressive side that is simultaneously more pro-technology, but is trying to take a stronger hand in predicting and then guiding that technology. But that's really hard. I mean, we can't pass even simple things right now. So the question of how we do it, I mean, I don't know how we do basic climate change legislation. So how we completely restructure the way taxation works in this country, when one of the two major political parties has literally signed a pledge to never raise taxes for any reason on anyone at any time.

• 43:08Ezra Klein?

  I don't really think it's a question that is answerable, but I am worried about what happens if some of these things hit and we haven't come anywhere near an answer.

• 43:16Speaker 01

  Our version of this is that technology is good, capitalism is fine, but you can put structures in place now that kind of get you to the goal that you want. And I think like we have this thing where hopefully we designed a structure that let us benefit from capitalism to bring the thing into existence. And I think that we couldn't do without it. But once we've hit certain return thresholds, we have like already figured out how our public ownership is going to work. You know, the ink is dry on that. And I think that's cool. And I think more people should be doing things like that. And I think there may be versions of that that we could get done as a policy conversation where you sort of say, this is how it's going to work now.

• 43:58Speaker 01

  Don't freak out too much, but here's where it's going to transition to over time.

• 44:02Ezra Klein?

  One of the problems with the policy conversation is that this is the set of issues it's way behind on. So there are topics we know how to talk about in Washington from a policy perspective. Taxes is one of them. But I mean, you were saying earlier in the show that at least the power of these systems is on an exponential curve.

• 44:19Speaker 01

  Yeah, taxing income, like fighting over the rate on income taxes, so the wrong way to be thinking about this.

• 44:24Ezra Klein?

  And, you know, my concern is that To the extent there is a Washington conversation over artificial intelligence is one of the set of technologies I'm interested in and worried about, but to the extent there's a Washington conversation about it, and you probably know this better than I do, my sense is it has like three prongs. It's about algorithmic bias, which is important.

• 44:44Speaker 01

  Yep.

• 44:45Ezra Klein?

  It's about competition with China.

• 44:46Speaker 01

  Yep.

• 44:47Ezra Klein?

  And it is sort of on the margins about how do you subsidize maybe academic access to enough computing power.

• 44:56Speaker 01

  The first two seem to dominate.

• 44:57Ezra Klein?

  But yeah, so I mean, it's basically it's bias, it's national defense, and then it's some other stuff like a grab bag. It's not really about what do you do if society begins changing really fast? And also, and I mean this as exactly as important, what do we want from it as a public and how do we get that from it?

• 45:15Speaker 01

  Let me tell you something. If you walk into one of these rooms and you sort of, you know, they're like, oh, you know, we're want to learn from you. Like, what should we be thinking about AI? And you say anything other than, you know, it's really important that we remain competitive with China and it's really important that we have fair, transparent, unbiased systems. If you say those two, you're like, oh, that's great. That's what we want. And those are really important for sure. And we think a lot about those. But if you say like, it's also important that we talk about like 20, 30, 40, 50 years from now, when we have these systems that are like way more capable than any human, like leaving from Earth to go off and explore and colonize the universe and figuring out what the role for humans are and how we want to set up that society, you get like a real

• 45:58Speaker 01

  eyes glazed overlook and sort of, you know.

• 46:01Ezra Klein?

  Do you believe in 30 years, we're going to have self-intelligence systems going off and colonizing the universe?

• 46:06Speaker 01

  Look, timelines are really hard. I believe that will happen someday. I think like, it doesn't really matter if it's 10 or 30 or 100 years. The fact that this is going to happen, that we're going to like help engineer or merge with or something, our own descendants that are going to like be capable of things that we literally cannot imagine, that is like somehow seems like way more important than the tax rate or like most other things.

• 46:27Ezra Klein?

  But in a weird way, I'm not sure it is. I mean, it is more important than the marginal tax rate on high income. I very much agree with that. But I do think sometimes there is a tendency to say that the intermediate questions between here and AGI don't matter that much. And I think they do because I think that they they matter very much in terms of what the future systems of the world will will do. I'll give an example of this. I think there's a lot of question on how do we distribute income, which if you could get the politics right to do it, is a quite manageable question. If you gave me the wand,

• 47:01Speaker 01

  Yes.

• 47:01Ezra Klein?

  Like, I could work with five tax experts and figure it out. The question of how do you distribute dignity and status in that society is unbelievably harder. So for instance, I'm pretty high on the idea that, you know, maybe wage labor is an intermediate step in human society. And wouldn't it be great if we got past the point where it was widely needed? But that is much more a cultural question in the worlds we're talking about here than it even is an economic one.

• 47:30Speaker 01

  100%. I think this is such a harder problem, and in many ways, more important problem than how we're going to redistribute income and wealth. And this is why I think the intermediate things do matter a lot. It's because how we, the steps that we take now will determine who has power and voice and input in the long term decisions that are kind of some of these rare irreversible human decisions that we'll ever make. You are probably sitting there, or at least part of you is sitting there, looking at me thinking like, you know, probably don't really believe this guy all the way. But if he's right, like, you know, like, fuck that dude. Why does he get to decide what happens with AI? That's not, that doesn't seem fair. And like, you're totally right. And if I were you, that's totally what I would think. And more importantly than that, everybody has a right

• 48:13Speaker 01

  to a say in the future and the dignity of being part of it and getting to live their lives the way that they want. And I don't think we've ever quite faced a technology like this. And I don't think we quite know what it looks like to have equitable input in those decisions.

• 48:32Ezra Klein?

  And I'll say more than that. I don't even know if what you're telling me is true, right? What I think is that there's a probability it is true that I cannot quite assess. And if that probability is significant, I think it at least is for the economic side of this. I don't know about the space colonization side or on what timeframe. Then it's important. Two things. One is that I think an endless truth of human society, at least in the age of capitalism, is that status follows money.

• 48:56Speaker 01

  Yep.

• 48:56Ezra Klein?

  And so getting the money right is going to be crucial to getting the status right, which is why the redistribution questions are really, really important. But the second then becomes, what are the governance structures that might work here? Or the participatory or representative structures that might work here? Because I've been pushing so far in this conversation on the problems of capitalism. It isn't like I'm confident that a state-run AI is a great idea, too. I don't really want China running one of these systems. And I wouldn't have wanted Donald Trump running one of these systems. And so the governance question here strikes me as really hard and not one where I'm comfortable with either of the dominant options.

• 49:33Speaker 01

  I think we need something fundamentally new. We're trying to run some experiments and do some investigatory work here. I don't think anyone's ever figured out what a global democratic governance system looks like. But I think we better try to figure that out soon. And there's a lot of reason to be pessimistic. Look, I also, I think that AI, I think everything that I've said is going to happen. But there could, we could hit some wall, like it's always possible. Maybe none of this happens. But if we, if we kind of like shift the frame a little bit and talk about, you know, we're talking about the environment earlier, so let's go back to that. Like, we could have had nuclear energy deployed everywhere, the full generating capacity needed for Earth and then some quite a long time ago. And we don't.

• 50:15Speaker 01

  And I think that's like a real shame of policy and at least in this country, some quite bad political decisions on the left and an understandable but misplaced fear of technology. And we're paying an awful price for it right now. And if we take all the speculation out of something like AI and just say like, can we look at an example of a collision of technology and policy and politics and public perception and economics and a whole bunch of other things, too? And see where we got the governance wrong? We don't need to make up examples for this. And so that, that is a scary thing.

• 50:54Ezra Klein?

  Yeah, and that I think is also an important thing. I mean, we were talking earlier about, well, what if AI can solve in the reasonably near term, 10, 20 years, pretty big societal problems. You know, a lot of the problems, and this came up in our conversation about health and housing, no computer system can solve a political problem, but it can create alternatives. And then we do have a lot of genuine technical and scientific problems that do need to be solved. And if we don't get governance right, if we don't get public input right, if people don't feel they're going to benefit from this, then we're also going to delay deployment of technologies.

• 51:29Speaker 01

  Yep.

• 51:29Ezra Klein?

  There's a lot of this where you're going to need regulatory decisions for the things to be deployed. And so that becomes a question of to what degree are people comfortable with it? And not just, you know, Phoenix is going to allow self-driving taxis for a little bit to show they can. But like, do you actually begin to say, we are going to create the systems, the liability systems, et cetera, that can allow this to go, you know, all through society or not? And I don't know the answer to any of that. Like, I'm partially doing these shows for this reason, that it's just not something that I think is enough part of the political conversation. But then I look at the past year, and I see a lot of failures of governance, followed by a gigantic success of mRNA vaccines. And if we had not

• 52:13Ezra Klein?

  gotten, you know, that technology right, we would just be in a continued mess here. And so like, getting the technologies right, fast enough that they can actually solve the problems, like, you can really see there, like, what it would have meant if, like, we had not, like, I don't think regulators did a crazy good job on this, but we got the vaccines out.

• 52:30Speaker 01

  I don't think they did a crazy good or a crazy bad job on this one, but I do think they've done a crazy bad job on a lot of other medical innovations that could have saved a lot of lives, where something has gone wrong in terms of our risk tolerance for those. Now, look, it does warm my heart that when a heroic effort is needed, we can sometimes step up. But I do think it is worth thinking about how many things like mRNA vaccines we have missed or still delayed or took, you know, came 10 years too late.

• 52:56Ezra Klein?

  But isn't there, in some of the things you're talking about here, a cultural question that afflicts your own industry pretty centrally? I wouldn't say that the people who run major technology companies are heavily anti-tax, not in general, but there is a pretty heavy strain of the venture capitalists and others who love their carried interest tax, who get, and it's not just the taxes, who get real angry at the idea of taxes as a moral phenomenon. That, I think, is an important dimension of this, get real angry at the idea that being taxed might imply they don't deserve what they have. There was a vogue a couple of years ago where folks were comparing this to Nazism, that if this stuff is going to deploy at the rate people want it to out here, I think there's going to need to be a kind of different politics in Silicon Valley.

• 53:42Speaker 01

  I actually think that most people in the industry are pretty happy to pay taxes. Like, sure, they may advocate for less. There's two comments that you hear a lot, one I agree with and one that I don't. One is that I'm happy to pay taxes, I just wish they were better spent. And, you know, because they're so badly spent, I don't want to pay any, like, while California is in shambles, why am I paying 13.3% tax? Like, this is embarrassing, we have terrible schools, terrible roads, traffic, like, the money is being so wasted, I don't want to pay anymore, and so I'm moving, or I'm gonna, like, you know, try to figure out some other, like, tax setup or whatever. I don't agree with that, you hear it a lot. I do understand where people are coming from, they're like, well, I can allocate the capital better than the government can to still help people.

• 54:24Speaker 01

  The one that I do agree with is, I think there's a lot of people in the tech industry, founders, venture capitalists, whatever, that really object to messaging like billionaires should not exist, which is very different from saying billionaires shouldn't be taxed. I would love to have trillionaires in the world if they were paying a lot of tax, and I think everybody else should want that, too. Now, there is a challenge that comes with the point you were making about, a lot of power comes with that, too.

• 54:50Ezra Klein?

  Yeah, that's what I was gonna ask.

• 54:51Speaker 01

  But on the whole, like, as long as people are paying taxes and creating economic value, I want more of that. And I think most members of society should want more of that, too. If, if, if, we can figure out how to make sure that we have fair access to power and voice and governance.

• 55:13Speaker 01

  And maybe what you would say is that's not possible.

• 55:14Ezra Klein?

  But I would say I think there might be a more deeper tension there than you think. So like, take the couple of richest people in the world right now. Is it such an accident that Jeff Bezos and Bill Gates live in Seattle, a state with no income tax? It's all fine to be for taxes in some conceptual way, but in practice, very rich, very powerful people live in that state. It's not like they're helping to get an income tax passed or that Elon Musk just re-domiciled himself in Texas to pay, at least in part, lower taxes. I mean, power speaks. And I mean, my worry, I don't think there should be a trillionaire, not because I don't want somebody to invent the thing that would have made them a trillion dollars. I do, but one, the marginal utility of having a trillion dollars is like crazy low for one person.

• 55:57Ezra Klein?

  That should just be distributed to people who then won't die from malaria because they have some money. But two, that's a lot of power for one person to have. And I think if I have one critique of your approach to politics on this stuff, it's that I think you take policy seriously and power less seriously. But one of my observations from covering policy for a long time is policy reflects power. It doesn't precede it.

• 56:20Speaker 01

  I can totally believe that. Again, like, I think if there is like a big bug in my thinking, this, this is likely it. Very sympathetic to the idea of being wrong here. I doubt that it's an accident that those people all live in no income tax states. But I also think like, I haven't like talked to Elon about this, but I know him well enough that I would bet that his bigger reason was just like an increasing unfriendliness in California towards his businesses. And like a real disagreement with how they were handling his company's ability to be successful. And that is, I think, a real shame for all of us. Like, no one is California residents. None of us benefit from having fantastic companies move out of the state.

• 57:04Speaker 01

  Like, that is a real shame. And, you know, like, that was power that the California legislature had that I think they badly misused that preceded the politics. And I wish that didn't happen. On the trillionaire question, like, you know, we were talking earlier about healthcare. Like, let's say someone invents, some scientist invents something that gives everybody on earth 20 years of extra great health span. And they want to make it phenomenally expensive, but like, they create $100 trillion of value and become a one trillionaire in the process. I would cheer them on.

• 57:35Ezra Klein?

  I just don't think that's the binary question. It's hard for me to imagine.

• 57:39Speaker 01

  But then how do you decide like how much they're allowed to make?

• 57:41Ezra Klein?

  I mean, through politics, right? It's hard for me to imagine the person who would invent that, who'd be dissuaded from inventing that because they would only end up with $50 billion as opposed to a trillion dollars.

• 57:51Speaker 01

  So 50, but we're gonna say 50 billion is okay.

• 57:53Ezra Klein?

  I don't think we're gonna find like an optimal. But again, I'd want to know a little bit more about both the product and the power distribution before I became comfortable with there being trillionaires because they can cause a lot of damage. And let me argue that there's actually more of a friction here even than than than we've let on. So your point about Elon Musk is well taken. I'm not saying anything is monocausal. But I think something going on in California in the relationship between the state and the technology companies that are centered here and something is not going to be good for either side is that as the increase in wealth inequality here rises, the sentiment changes, right? The politics change. And so part of the reason there's more friction is because people are more pissed off.

• 58:37Ezra Klein?

  A different example, and one that I think is somebody's easier to see the problems with is Mark Zuckerberg gives a huge amount of money to a hospital, names it after himself. And then there's this fight in the, you know, I think it was the Board of Supervisors voted to condemn the name of the hospital. And part of what's just going on there is people are angry about the power he has. I don't think it was a bad thing. He gave that money to the hospital and given the way we've named things after people forever. I mean, go to a sports stadium around this country. Like, I don't even really care that it got named after him. But there is a tension that begins to emerge and poison relations.

• 59:09Speaker 01

  Yeah.

• 59:10Ezra Klein?

  As people begin to feel that the that the wealth of the economy is unequally shared. And I think that's a pretty big issue. I mean, one place my politics has probably been moving in recent years is I would like to see a much more like radically pro-technology progressivism. But I think in order to do that, you might need a much more economically aggressive form of progressivism, right? That's why I'm sort of interested by the fully automated luxury communism concept. Because I wonder if in order to create enough societal comfort to have the technological society, I think is possible. You actually need a much more radical form of equality than we have. I did this show with Ted Chang, the science fiction writer.

• 59:52Ezra Klein?

  He made this point where he said, you know, I think most of our fears about technology are fears about capitalism. And I depart from that in the sense of if you live in China or a lot of other countries, I don't think that's true. And even sometimes not in this country, there's things to worry about with the state, too. But it is true for a lot of things here. And I think it actually holds back technology and the degree to which people are willing to adopt it in a way that is kind of bad for everyone.

• 1:00:17Speaker 01

  So on this point, I think we're in perfect agreement. I think we have to do this. Like, I think technology can make the world unimaginably great, but it needs a policy and power tweak to have that be distributed in a way where it can happen at all and in a way where it can happen justly. So I agree with that. I also agree that the heads of the biggest technology companies probably do have too much power or do have too much. I'll take out the probably, you know, in the industrial revolution, when the joint stock corporation was created as this sort of second order sovereign entity, everyone was okay with that because it was second order and the real sovereign had more power. But I think you can certainly make a case now that the giant tech companies are more powerful than many countries,

• 1:00:58Speaker 01

  not certainly not the US yet, but they don't have any kind of democratically governed system. So, yeah, I mean, that causes me deep discomfort.

• 1:01:29Speaker 01

  When I asked Ted Chang about AGI, he said something I've been thinking about since, which is that

• 1:01:34Ezra Klein?

  Could we invent it? Maybe. Will we invent it? Maybe. Should we invent it? No. And the reason he said no was that long before we have a kind of sentient, generally intelligent AI, we'll have AI that can suffer. And if you think about how we treat animals, or even just think about how we treat computers, or frankly, workers in many cases, you know, the idea that we can make infinite copies of something that can suffer that we will see in a purely instrumental way is horrifying. And that, fully aside from how human beings will be treated in this world, like the actual AI will be treated really badly. Do you, I mean, you're somebody who thinks out on the frontier of this.

• 1:02:16Ezra Klein?

  I know this part of the conversation is going to turn some listeners off, but I think it's interesting. Do you worry about the suffering of what we might create?

• 1:02:26Speaker 01

  Yes, a lot. And we'd call that like model welfare, but it's the same idea. Fundamentally, I think people worry like, are we going to mistreat the AI like we mistreat animals? Or is the AI going to mistreat us like we mean animals? That's mistreat animals. That's the sort of, you can put people in one of those two buckets usually.

• 1:02:40Ezra Klein?

  Really says a lot about how we treat animals.

• 1:02:42Speaker 01

  I think we're gonna have a lot more vegetarians soon, I hope. There's this great AI short story called Crystal Knights. And it's one of the ways that AI could be developed is what people call multi-agent, where you basically like rerun evolution and simulation. In an environment like that, you probably do get enormous amounts of suffering. A lot of other approaches, I think you do too. Like maybe if a reinforcement learning agent is getting negative reward, it's feeling pain to some very limited degree. And if you're running millions or billions of copies of that, you're creating quite a lot. That's a real moral hazard. The technical path that I think, currently think, is most likely to get us to one of these AGI-like systems is one where you don't have tons of copies of agents. And so maybe there's a moral issue with causing any pain.

• 1:03:25Speaker 01

  But I sleep a little bit better at night, knowing that we're like not torturing trillions of agents in simulation. I also think, and this is like deep into the realm of speculation, that certainly I think as humans have become more knowledgeable and wise and smarter, we seem to be less willing to inflict random pain on each other and maybe animals than we were.

• 1:03:49Ezra Klein?

  See, I think animals is a counterargument to that. Because of our capabilities, the number of animals we mistreat is so wildly out of proportion to what we've ever been able to do before because we can keep them alive in sheds with antibiotics and stuff that our thinking about it is probably better. But what we're willing to outsource is worse.

• 1:04:08Speaker 01

  I don't know, like I have noticed over the last few decades, more and more of my friends consistently become vegetarian. And I think that's like a positive. That's, you know, you used to not have like a lot of vegetarians in the Western world. You know, we have the privilege and luxury to be able to not do that. We now increasingly have the technology to have fake meat. But I think we also have people who are just sort of like more conscious about these issues. And I hope that we are in the late stage of the twilight of factory farming. I mean, look, we're like heading like into the deepest philosophical questions that have been on humanity's mind for a very long time, but maybe you've never been as relevant or as

• 1:04:51Speaker 01

  decision relevant as they are now. But I think a lot of these things really come down to like, A, do you believe that a sense of self exists at all? Or is it like, is everything just like, there's this body and there's this brain and there's like energy flowing through a neural network in your head like there could be in a computer. And as that is running, it creates this like illusion of a sense of self that is getting tortured, but it really is not there at all and that it's all the same thing. That's like a philosophy that I happen to mostly feel is the most true. There's others who would say like quite the opposite, which is that there will never be a sense of self in a, there is a self, there's a soul, whatever. And that will never exist in a computer. And so all of these questions are irrelevant.

• 1:05:35Speaker 01

  Like, there's no way that any neural network running in silicon could ever suffer. And it's a ridiculous question. People have like quite different opinions on this.

• 1:05:43Ezra Klein?

  Why try to make it generally intelligent at all? Jeff Hinton, one of the fathers of neural networks, he had this quote in the book Genius Makers that I recently read where he just says, why do you want the robot digging your ditches to know about baseball? Like, why, why create the, the potential for this kind of intelligence and kind of suffering at all? Why not just create narrow worker machine learning programs?

• 1:06:07Speaker 01

  I have a lot of answers to this question, so I will start with a few. Number one, one thing that I really want. Is new knowledge creation. Like, it is one thing to say you can have an AI that is as good as any human AI doctor. It is another to say you could have an AI that can solve all human diseases in a way that humans are just not capable of doing. And I think you need a generally intelligent system to do that, to generate new knowledge, to learn new things, to do things humans can't do. And it will need to pull together expertise across many different areas that no human is capable of holding in their brain at once, or even team with humans to do. So, like, what a depressing thought to say that, like, we're gonna limit ourselves to what humans are capable of,

• 1:06:51Speaker 01

  rather than benefit from everything that we can build better tools for. Like, we build tools so that we can do better than we can do with our hands, you know, digging up the dirt or whatever. Like, all of this, everything we have here, right, is because we started digging up the ground, finding stuff, and like, we made this room, we made this microphone, we made the internet. Like, we have done incredible work with tools that have let us shoot past what we'd be capable of without them, and let's not stop. That would really be depressing to me. Two is that someone's gonna do it. The upside of these systems are such that, like, Jeff Hinton can certainly decide not to try to build generally intelligent systems, but someone's going to do it. So I think there is no future that doesn't have these systems in it.

• 1:07:32Speaker 01

  And so we have to talk about how we want to use them, what their rights are, what we want the world to look like, the universe to look like with them.

• 1:07:41Ezra Klein?

  If we're sitting, if we get together for coffee in 20 years and none of it happened, it all just stalled out, what is the likeliest reason you're gonna give me for it?

• 1:07:50Speaker 01

  If it doesn't happen in 20 years, I could, I'll talk about like, a bunch of little technical things that could go wrong. If it doesn't happen in like, 100 years, then I was wrong and there is like, some magic, you know, to being a human, like, we are living in a simulation or created by some god or something like that. Like, there's something going on that is not just physics.

• 1:08:09Ezra Klein?

  It's that big? You think that, that like, it will, like, physics is gonna be up for debate at that point? Because some people would say, you know, the, how far I can get with neural networks, pulling in data to do predictive learning, there's just gonna be a limit on that.

• 1:08:22Speaker 01

  Well, here you are, though. I mean, it works for you.

• 1:08:24Ezra Klein?

  Yeah, but only so well. You should see me, you should see me on my off days.

• 1:08:26Speaker 01

  Sure, but like, we can, if we can have an Ezra, like, that's pretty great. And my understanding, my belief is that you are energy flowing through a neural network. That's it. Perception comes in, it like cycles around a neural network in your head and you, some muscle of yours moves. But that's it. That's the whole Ezra. And that is replicatable. Energy flowing through a neural network and is replicable on a very big computer. It takes us a while to figure out. There's like a lot of complexity there, maybe. But yeah, if, if it doesn't happen in a few hundred years or a hundred years even, then some axiom I believe is wrong.

• 1:09:04Ezra Klein?

  I'm going to go meditate on the idea that I'm energy flowing through a neural network.

• 1:09:09Speaker 01

  It is a little depressing, but if you can just like let this one idea go, that there is like, there's a special self, there's like an Ezra that controls all of this and just say like, there's nothing here, like it's just the system and it has this like side effect of me thinking I'm an Ezra, then it all kind of works.

• 1:09:23Ezra Klein?

  I think it's a good place to end. So what are three books you'd recommend to the audience to help the energy flow slightly differently through their neural networks?

• 1:09:30Speaker 01

  Can I recommend, both because I think they're like more likely to get read and I think they're like more relevant to this conversation. I don't think there's any like great books about AI, but there are good short stories. So could I recommend three short stories? Crystal Knights by Greg Egan, The Last Question by Isaac Asimov, and The Gentle Seduction by Mark Stigler. They're all about the development of like a super powerful AI in very different ways. And actually, if I can recommend a bonus, fourth one, is a blog post, not a short story, but it really touches on a lot of this sort of societal governance, power issues we're talking about relative to AI. Meditations on Moloch. It's a blog post on Slate Star Codex. I strongly recommend that one.

• 1:10:13Ezra Klein?

  Sam Altman, thank you very much.

• 1:10:15Speaker 01

  Thank you.

• 1:10:26Ezra Klein?

  The Zerklan Show is a production of New York Times Opinion. It is produced by Jeff Geld, Raja Karma, and Annie Galvin. Fact checking by Michelle Harris. Original music by Isaac Jones and mixing by Jeff Geld.