Holden Karnofsky — History's most important century

Holden Karnofsky is the co-CEO of Open Philanthropy http://www.openphilanthropy.org/ and co-founder of GiveWell http://www.givewell.org/. He is also the author of one of the most interesting blogs on the internet, Cold Takes https://www.cold-takes.com/. We discuss: * Are we living in the most important century? * Does he regret OpenPhil’s 30 million dollar grant to OpenAI in 2016? * How does he think about AI, progress, digital people, & ethics? Highly recommend! Watch on YouTube https://youtu.be/UckqpcOu5SY. Listen on Apple Podcasts https://podcasts.apple.com/us/podcast/holden-karnofsky-transformative-ai-most-important-century/id1516093381?i=1000592080817, Spotify https://spoti.fi/3Qi11SY, or any other podcast platform. Read the full transcript here https://www.dwarkeshpatel.com/p/holden-karnofsky. Timestamps (0:00:00) - Intro (0:00:58) - The Most Important Century (0:06:44) - The Weirdness of Our Time (0:21:20) - The Industrial Revolution (0:35:40) - AI Success Scenario (0:52:36) - Competition, Innovation , & AGI Bottlenecks (1:00:14) - Lock-in & Weak Points (1:06:04) - Predicting the Future (1:20:40) - Choosing Which Problem To Solve (1:26:56) - $30M OpenAI Investment (1:30:22) - Future Proof Ethics (1:37:28) - Integrity vs Utilitarianism (1:40:46) - Bayesian Mindset & Governance (1:46:56) - Career Advice This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If we had AI systems that could do everything humans do to advance science and technology, that would be insane. We live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world. If you just imagine a universe where there actually are some people who live in an especially important time, and then there's a bunch of other people who like tell stories to themselves about how, what, you know, whether they do, how would you want all those people to behave? And it's like, to me, the worst possible rule is all those people should just be like, nah, this is crazy and forget about it.

• 0:37Dwarkesh Patel?

  All right. Today, I have the pleasure of speaking with Holden Karnofsky, who is the co-CEO of Open Philanthropy. In my opinion, Holden is one of the most interesting intellectuals alive, well, given your role. So Holden, welcome to the Lunar Society.

• 0:51Speaker 00

  Thanks for having me.

• 0:52Dwarkesh Patel?

  Okay, so let's start off by talking about the most important century thesis. Do you want to explain what this is for the audience?

• 0:59Speaker 00

  You know, my story is I originally co-founded an organization called GiveWell that helps people decide where to give as effectively as possible. I'm no longer there, but I'm on the board and it's a website called GiveWell.org that I think makes good recommendations where to give to charity to help a lot of people. And, you know, as we were working at GiveWell, we met Kerry Tuna and Dustin Moskowitz. Dustin is the co-founder of Facebook and Asana, and started a project that became Open Philanthropy to try to help them give away their large fortune, again, to help as many people as possible. And so I've kind of spent my career looking for ways to do as much good as possible with a dollar or with an hour, with whatever resources you have, and especially with money. And

• 1:40Speaker 00

  so I've kind of developed this professional specialization in looking for ideas that are underappreciated, underrated, tremendously important, because a lot of the time, that's where I think you can find just kind of outsized, what you might call outsized return on investment. Opportunities to spend some money and just get an enormous impact because you're doing something very important that is being ignored by others. And so it's through that kind of professional specialization that I've actively looked for interesting ideas that are not getting enough attention, and that I encountered the Effect of Ultras community, which is a community of people basically built around the idea of doing as much good as you can. And so it's through that community that I encountered the idea of the most important

• 2:20Speaker 00

  century. It's not my idea at all. I got to it from a lot of people. And the basic idea is that if we developed the right kind of AI systems this century, and that looks reasonably likely, that could make this century the most important of all time for humanity. So the basic mechanics of why that might be, or how you might think about that. So one thing is that if you look back at all of economic history, just the rate at which the world economy has grown, you see acceleration. You see that it's growing a lot faster today than it ever was. And one theory of why that might be, or one way of thinking about it through the lens of basic economic growth theory, is that in normal circumstances, you

• 3:03Speaker 00

  can imagine a kind of feedback loop where you have, people have ideas, and the ideas lead to greater productivity and more resources. And then when you have more resources, you can also have more people, and then those people have more ideas. So you get this feedback loop that goes people, ideas, resources, people, ideas, resources. And starting a couple hundred years ago, if you run a feedback loop like that, standard economic theory says you'll get accelerating growth. You'll get a rate of economic growth that goes faster and faster. And basically, if you take the story of our economy to date, and you just kind of plot it on a chart, and do the kind of simplest thing you can to project it forward, you project that it will go, that our economy will reach like an infinite growth rate this

• 3:43Speaker 00

  century. And the reason that I currently don't think that's a great thing to expect by default is that one of the steps of that feedback loop broke a couple hundred years ago. So it goes more people, more ideas, more resources, more people, more ideas, more resources. A couple hundred years ago, people stopped having more children when they had more resources. They got just more, they got richer instead of more populace. And this is all discussed in the most important century page on my blog Cold Takes. And so what happens right now is that when we have more ideas and we have more resources, we don't end up with more people as a result. We don't have that same accelerating feedback loop. And if you had AI systems that could do all the things humans do to advance science

• 4:24Speaker 00

  and technology, meaning the AI systems could fill in that more ideas part of the loop, then you could get that feedback loop back, and then you could get sort of this unbounded, heavily accelerating, explosive growth in science and technology. So that's the basic dynamic at the heart of it. So that's kind of a way of putting it that's trying to use familiar concepts from economic growth theory. Another way of putting it might just be, gosh, if we had AI systems that could do everything humans do to advance science and technology, that would be insane. You know, what if we were to take the things that humans do to create new technologies that have transformed the planet so radically, and we were able to completely automate them so that every computer we have is potentially another mind working on advancing technology?

• 5:07Speaker 00

  So either way you think about it, you could imagine the world changing incredibly quickly and incredibly dramatically. And so I argue in the most important century series that it looks reasonably likely, in my opinion, more than 50/50, that this century will see AI systems that can do all of the key tasks that humans do to advance science and technology. And that if that happens, we'll see explosive progress in science and technology. The world will quickly become extremely different from how it is today. You might think of it as if there was thousands of years of changes packed into a much shorter time period. And then if that happens, I argue that you could end up in a deeply unfamiliar future. I give one example of what that might look like using this hypothetical technology idea called digital people.

• 5:52Speaker 00

  That would be sort of people that live in virtual environments that are kind of simulated, but also realistic and exactly like us. And when you picture that kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science and technology. We could basically find most of what there is to find and end up with a civilization that expands well beyond this planet, has a lot of control over the environment, and is very stable for very long periods of time, and basically looks sort of post-human in a lot of relevant ways. And if you think that, then this is basically our last chance to shape how this happens. So that's the most important century hypothesis. In a nutshell, is that if we develop AI that can do all the things humans do to advance

• 6:36Speaker 00

  science and technology, we could very quickly reach a very futuristic world, very different from today's, could be a very stable, very large world, this is our last chance to shape it.

• 6:45Dwarkesh Patel?

  Gosha. Okay, so I and many other people are gonna find that very wild. So maybe you can walk us through the process by which you went from doing global development stuff to thinking this way. So in 2014, for example, you had an interview or a conversation, and this is a quote from there. Maybe you can walk me through how you got from there to where you are today. "I have looked at the situation in Africa, have understanding of the situation in Africa, and see a path of doing a lot of good in Africa. I don't know how to look into the far future situation, don't understand the far future situation, and don't see a path to doing good on that front I feel good about."

• 7:27Speaker 00

  Yeah, first, I think I just, I went on for a while, but I want to come back and connect this back up to the, how this relates to the work I was even doing at GiveWell, why this is all kind of one theme. If we are kind of on the cusp for this century of creating these advanced AI systems, then we could be looking at a future that's like very good or very bad.

• 7:45Dwarkesh Patel?

  Okay.

• 7:45Speaker 00

  And I think there are decent arguments that if we move forward without caution and we develop kind of sloppily designed AI systems, they could end up with goals of their own, and basically we'd end up with a universe that contains very little that humans value, or a galaxy that does. We could also imaginably end up with a world in which very powerful technologies are used by just not very well-meaning governments to create a world that isn't very good, or a world where we kind of eliminate a lot of forms of material scarcity and have a world that's much better than today's. And so, a lot of what I ask, a lot of what I ask at GiveWell is how can we help the most people possible per dollar spent? And if you ask how can we help the most people possible per dollar spent, then if you think

• 8:27Speaker 00

  that funding some work to help shape that transition, to help make sure that we don't move forward too incautiously, to help make sure that we do, you know, increase the odds that we do get like a good future world instead of a bad future one, that's helping a huge number of people per dollar spent. So that's the motivation. And now you're quoting me, you know, a discussion, an argument I was having, where we posted a transcript back in 2014, and at that time, was, that was part of my journey of getting here. So I was talking to people who were saying, Holden, you want to help a lot of people with your resources, you should be focused on this massive event that could be coming this century, that very few people are paying attention to, that there might be a chance to make this go well or poorly for humanity. And I was saying, gosh, like, that sure is interesting. And I did think it was interesting. That's why I was spending the time and doing

• 9:12Speaker 00

  the conversation. But I said, you know, When I look at global poverty and global health, I see what I can do. I see the evidence. I see the actions I can take, and I'm not seeing that with this stuff. So what changed? I would say a good chunk of what changed is maybe like the most boring answer possible is I just kept at it. So I think I was sitting there in 2014 saying, gosh, this is really interesting, but it's all a bit overwhelming. It's all a bit crazy. I don't know how I would even think about this. I don't know how I would come up with a risk from AI that I actually believed was a risk and could do something about today. And now I've just been thinking about this for a much longer time period.

• 9:51Speaker 00

  And I do believe, you know, most things you could say about the far future are very unreliable and not worth taking action on. But I think there are a few things one might say about what a transition to very powerful AI systems could look like. There are some, some things I'm willing to say, I'm willing to say it would be bad if AI systems were poorly designed, had goals of their own and ended up kind of running the world instead of humans. That seems bad and I do, and I am more familiar today than I was then with the, the research and the work people can do to make that less likely and the actions people can take to make that less likely. So that's probably more than half the answer, but another thing that would be close to half the answer is I think the world has changed. And I think that basically there's been big changes in the world of AI since then.

• 10:36Speaker 00

  So I think basically in 2014, that was the beginning of what's sometimes called the deep learning revolution. And since then, we've basically seen these, these very computation intensive, but fundamentally simple AI systems and achieve a lot of progress on a lot of different unrelated tasks. And it's looking to me like not totally crazy to imagine that the current way people are developing AI systems, cutting edge AI systems could take us all the way to the kind of extremely powerful AI systems that automate roughly everything humans do to advance science, technology. It's not so wild to imagine that we could just keep on going with these systems, make them bigger, put more work into them, but basically stay on the same path and you could get there.

• 11:19Speaker 00

  And if you imagine doing that, it becomes a little bit less daunting to imagine the risks that might come up and the things we could do about them. So I don't think it's necessarily the leading possibility, but it's enough to sort of start thinking concretely about the problem.

• 11:32Dwarkesh Patel?

  Another quote from the interview that today I find appealing, because I haven't done the work you have yet. Does even the upper crust of humanity have a track record of being able to figure out the kinds of things Miri claims to have figured out? And by the way, for context for the viewers, Miri is the organization Eliezer was leading, which is who we were talking to at the time.

• 11:51Speaker 00

  Yeah, well, I don't remember exactly what kinds of things Miri was trying to figure out, and I'm not sure that I even understood what they were that well. So I definitely think it is true that it is hard to predict the future, no matter who you are, no matter how hard you think, no matter how much you've studied. I think that is true. I think parts of our kind of world, or meme-plex, or whatever you want to call it, overblow this at least a little bit. And I think I was kind of buying into that a little bit more than I could. So I think, you know, probably in 2014, I would have said something like, gosh, you know, really, no one's ever done something like making smart statements about what the, you know, several decades out future could look like, or making smart statements about what we could be doing today to prepare for it. Since then, I think a bunch of people

• 12:36Speaker 00

  have looked into this and looked for like historical examples of people kind of making long-term predictions and long-term interventions. And I don't think it's amazing, but I think I wrote a recent blog post entitled "The Track Record of Future Seems...Fine." Fine is how I'd put it, where I don't think there's anyone who has demonstrated a real ability to predict the future with precision and know exactly what we should do. But I also don't think humans' track record of this is so bad and so devastating that we shouldn't think we are capable of at least giving it a shot. And I think if you enter into this endeavor... With self-awareness about the fact that everything is less reliable than it appears and feels at first glance, and you look for the few things that you would really bet

• 13:21Speaker 00

  on, I think it's worth doing, I think it's worth the bet. My job is to find things that we might do 10 things and have nine of them fail embarrassingly and one of them be such a big hit that it makes up for everything else. And a lot of my job is to find stuff like that. So I don't think it's totally crazy to think we could make meaningful statements about how things we do today could make these future events go better, especially if the future events aren't crazily far away, especially if they're within the next few decades. So that's, that is something I've changed my mind on at least some, to some degree.

• 13:49Dwarkesh Patel?

  Gotcha. Okay, so we'll get to the forecasting stuff in a second, but let's, let's continue on the object level conversation about most important centuries. So I want to make sure I have the thesis right. So is the argument that because we're living in a weird time, we shouldn't be surprised if something like transformative AI happens this century? Or is the argument that since transformative AI could happen this century, it's a weird time?

• 14:11Speaker 00

  Yeah. So, so something we haven't covered yet, but I think is worth throwing in now is that a significant part of the most important century series is kind of just making the case that even if you ignore AI, there's a lot of things that are very strange about the time that our generation lives in. The reason I spent so much effort on this is because my number one, back in 2014 and before that, my number one objection to these stories about transformative AI is, you know, it's, it's not anything about whether the specific claims about AI or economic models or align, you know, alignment research makes sense. It's just, this whole thing sounds crazy. And it's just suspicious. It's just suspicious if someone says to you, you know, this could be the most important century of all time for humanity. I titled the series that way because I wanted people to know that I was

• 14:56Speaker 00

  saying something kind of crazy and that I should have to defend it. I didn't want to be backpedaling or soft pedaling or hiding what a big claim I was making. And so I think my biggest source of skepticism has been just like, I don't have any specific objection. It just sounds kind of crazy and suspicious to say that we might live in one of the most significant times or the most significant time for humanity ever. And so a lot of my series is just kind of saying, It is weird. It is weird to think that, but we already have a lot of evidence that we live in an extraordinarily weird time that would be on the shortlist of contenders for most important time ever, before you get into anything about AI, just using like completely commonly accepted facts about the world. For example, if you chart the

• 15:39Speaker 00

  history of economic growth, you see that the last couple hundred years have seen faster growth by a lot than like anything else in the history of humanity or the world. And if you chart anything about like scientific and technological developments, you can see that everything significant is packed together in the recent past. And there's almost no way to cut it. You know, I've looked at many different cuts of this. There's almost no way to cut it that won't give you that conclusion. One way to put it is that the, you know, the universe is something like 11 or 12 billion years old, humanity, or sorry, life on Earth is like 3 billion years old. And then humanity is just a blink of an eye compared to that. You could call it 300,000, 3, 3 million years old. Human civilization is a blink of eye compared to that. And we're in this really, really tiny

• 16:21Speaker 00

  sliver of time, couple hundred years, when we've seen basically, you know, all or like just a huge amount of the technological advancement, economic growth. So that's weird. You know, I also talk about the fact that the current rate of economic growth seems high enough that we can't keep it going for that much longer. If it went for another 10,000 years, that's another blink of an eye on galactic time scales. It looks to me, and we can get to this, like we would run out of atoms in the galaxy, wouldn't have anywhere to go. And so I think there are a lot of signs that we just live in a really strange time. I think one, one more I'll just throw in there that we can move on is that, you know, I think a lot of people who disagree with my take would say, look, I do believe eventually we will develop space colonization abilities, go to the stars,

• 17:06Speaker 00

  fill up the galaxy with life, you know, maybe have artificial general intelligence. I just, to say it's a century is crazy. I think it might be 500 years. I think it might be 1,000 years. I think it might be 5,000 years. And a big point I make in the series is I say, well, even if it's 100,000 years, that's still an extremely crazy time to be in, in the scheme of things. If you just make, you know, if you make a graphic timeline and, and you kind of show my view versus yours, they look exactly the same down to the pixel. And so there's already a lot of reasons to think we live in a very weird time. We're on this planet where there's, where there's no other sign of life anywhere in the galaxy. We believe that we could fill up the galaxy with life. That alone would make us among the earliest life that has ever existed, or would make us the earliest life that has ever existed in the galaxy,

• 17:51Speaker 00

  a tiny fraction of it. So that is a lot of what the series is about. And I have sort of answered a question, but I'll do it explicitly. You ask, you know, is the series about transformative AI could come, and therefore this century could be weird? Or is it about this century could be weird, therefore transformative AI could come? The central claim is that transformative AI could be developed in this century. And the stuff about how weird a time we live in is just a response to an objection. It's a response to a point of skepticism. It's a way of saying, there's already a lot of reasons to think we live in a very weird time. And so actually, this thing about AI is only a moderate quantitative update, not a complete revolution in the way you're thinking about things.

• 18:30Dwarkesh Patel?

  There's a famous comedian who has a bit where he's imagining what it must have been like to live in 10 B.C.

• 18:36Speaker 00

  Yeah.

• 18:37Dwarkesh Patel?

  People are just numbering. It's 10 years before zero. It's nine years before zero. It's gonna happen. Let's say somebody comes with a proof that current deep learning techniques are not scalable for some reason, and that transformative AI is very unlikely this century.

• 18:50Speaker 00

  Yeah.

• 18:51Dwarkesh Patel?

  I don't know if this is a hypothetical where that would happen, but let's just say that's what it is. Even if this is a weird time in terms of economic growth, does that have any implications other than transformative AI of what you should be doing?

• 19:02Speaker 00

  Yeah, I think, I think in general, having kind of learned about just how strange the time we live in is when you look at it in context. And here I just encourage people to go to my series because I have a bunch of charts illustrating this, and it can be a little bit, little bit hard to do concisely. But I think the biggest thing I take away from that is we should just, you know, we should really look for the next big thing. I think if you'd been living 300 years ago, and you'd been talking about the best way to help people, a lot of people might have been talking about, you know, various forms of, of, you know, Helping helping low income people, maybe they probably would have been talking about spreading various religious beliefs. And I think, you know, it would have seemed crazy to think that what you should be thinking about, for example, was the steam engine

• 19:44Speaker 00

  and how that might change the world. But I think the Industrial Revolution was actually just an enormous deal and probably was the right thing to be thinking about if there's any way to be thinking about it. If there was any way to be thinking about how that would change the world and what might could what one might do to make that a world that could be better. So that's basically where I'm at is on this stuff is I just think, you know, as as a world, as a global civilization, we should have a really high priority on just saying, we live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world. We should make a list of all the things that might radically transform the world. We should

• 20:26Speaker 00

  make sure we've done everything we can to think about them, identify the ways we might be able to do something today that would actually help. And maybe after we're done doing all that, and we have a lot of the world's brightest mind doing their best to think of stuff, and they can't think of anymore, then we can go back to to all the other things that we worry about. But I think right now, the world invests so little in that kind of speculative, hey, what's the next big thing? Even if it's not super productive to do so, even if there's not that much to learn, I feel the world should be investing more in that than there is because the stakes than it is, because the stakes are extremely high. And I think it is just like a reasonable guess that we're living in a world that's recently been incredibly transformed by the Industrial Revolution and the future could be incredibly transformed by the next thing. I just don't think this gets a lot of discussion

• 21:08Speaker 00

  in basically any circles. And if I got some, I would feel a lot more comfortable with the whole thing. I don't think the whole world should just obsess over what the next transformative event is. But I think right now there's so little attention to it.

• 21:19Dwarkesh Patel?

  I'm glad you brought up the Industrial Revolution, because I feel like there's two implicit claims within the most important century thesis that don't seem perfectly compatible. One is that, you know, we live in an extremely wild time, that the transition here is potentially wilder than any other transition there has been before. And the second is, we have some sense of what we can be doing to make sure this transition goes well. I'm curious if you think that somebody at the beginning of the industrial revolution, if, knowing what they knew then, could have done something significant to, you know, make sure that it went as favorably as possible? Or do you think that that's a bad analogy for some reason?

• 21:56Speaker 00

  It's a pretty good analogy for being thought provoking and for thinking, gosh, if you had seen the industrial revolution coming in advance, and this is, you know, when economic growth really reached a new level back in the 1700s and 1800s, you know, what could you have done? And I think, you know, part of the answer is, it's not that clear. And I think that is a bit, you know, a bit of an argument that we should maybe not get too carried away today by thinking that we know exactly what we can do. But I don't think the answer is quite nothing. So I have a goofy cold takes post that I never published and may never publish because I don't know, I kind of lost track of it. But it's kind of saying, well, what if you'd been sitting in that time and you had known the industrial revolution was coming, or you had thought it might be? You'd asked yourself what you could be doing.

• 22:40Speaker 00

  One answer you might have given is, you might have said, well, gosh, if this happens, whatever country it happens in might be just disproportionately influential. And you know what would be great is if I could help transform the thinking and the culture in that country to have a better handle on human rights and more value on human rights and individual liberties and a lot of other stuff. And gosh, it kind of looks like people were doing that. And it kind of looks like it worked out. So this is the Enlightenment. And I think, you know, I even give this kind of goofy example. I could look it up, and it's all kind of a trollish post. But the example is, it's like someone's kind of thinking through, hey, you know, I'm thinking about this sort of esoteric question about what a government owes to its citizens, or,

• 23:22Speaker 00

  you know, when a citizen has a right to overthrow a government, or when it's acceptable to kind of enforce certain beliefs and not. And it's like the other person in the dialogue is just like, this is the weirdest, most esoteric question. Why does this matter? Why aren't you helping poor people? But, you know, these are the questions that the Enlightenment thinkers were thinking about, and I think there is a Good case that they came up with a lot of stuff that that really shaped the whole world since then. Because of the fact that the UK became so influential, really laid the groundwork for a lot of stuff about the rights of the governed and free speech and individual rights and human rights. And and then and then I go to the next analogy and I'm like, OK, now we're sitting here today and someone is saying, well, instead of working on global poverty, I'm studying this kind of esoteric question about how you get an AI system to do what you wanted it to do instead of doing its own thing.

• 24:07Speaker 00

  And it's, you know, you could, you could, it's not, I think it's not completely crazy to see them as analogous. Now, I don't, I don't think this is what the Enlightenment thinkers were actually doing. I don't think they were saying this could be the most important millennium, so let's do this. But it is interesting that you, it doesn't look like there was nothing to be had there. It doesn't look like there's nothing you could have come up with. And in many ways, it looks like what the Enlightenment thinkers were up to had the same kind of esoteric, strange, overly cerebral feel at the time and ended up mattering a huge amount. So it doesn't feel like there's zero precedent either.

• 24:36Dwarkesh Patel?

  Yeah, maybe I'm a bit more pessimistic about that because people like the people who are working on individual rights frameworks like log, I don't think they were like anticipating an industrial revolution.

• 24:46Speaker 00

  Yeah.

• 24:46Dwarkesh Patel?

  I mean, I feel like the person that actually did anticipate the industrial revolution probably, his political philosophy was actually probably a negative, given, you know, I'm talking about Karl Marx.

• 24:55Speaker 00

  OK.

• 24:56Dwarkesh Patel?

  Yeah, yeah. So it's not obvious to me that even if you saw something like this happening that you're.

• 25:01Speaker 00

  Oh, it's totally not obvious. I mean, I think my basic position here is we, I'm not sitting here like highly confident. I'm not saying, you know, there's tons of precedent and we know exactly what to do. That's not what I believe. I believe we should be giving it a shot. I think we should be trying and I don't think we should be totally defeatist in saying, well, it's it's so obvious that there's never anything you could have come up with throughout history and humans have been helpless to predict the future. I don't think that is true. And so, yeah, I think I think that's like enough of an example to kind of illustrate that. And I mean, gosh, like you could, you could make the same statement today is you could say, look, doing research on how to get AI systems to behave as intended. Is a perfectly fine thing to do at any period in time. Um, it's not like a bad thing to do. And I think John Locke was doing his stuff because he felt it was a good thing to do at any period in time.

• 25:44Speaker 00

  But the thing is that if we are at this crucial period of time, it becomes an even better thing to do. And it becomes magnified to the point where it could be more important than other things.

• 25:51Dwarkesh Patel?

  Now, the one reason I might be skeptical of this theory is that I could say, oh, gosh, if you look throughout history, people were often convinced they were living in the most important time.

• 26:00Speaker 00

  Sure.

• 26:00Dwarkesh Patel?

  Or at least in especially important time.

• 26:02Speaker 00

  Yeah.

• 26:02Dwarkesh Patel?

  And if you go back, I mean, everybody can be right about living in the most important time. So maybe I should just, uh, have a very low prior that anybody is right about this kind of thing. How do you respond to that kind of logic?

• 26:12Speaker 00

  Well, I mean, so first off, I don't know that it's really true that it's that common for people who've said they're living in the most important time in history. I mean, this would be like an interesting thing to look at, but just from, you know, from stuff I've read about whatever past works on political philosophy and stuff. And I don't exactly see this claim all over the place. It definitely happens. It's definitely happened. And I think a way of thinking about it is there, you know, there's kind of two reasons that you might think you are especially important. One is that you actually are, and you made reasonable observations about it. And another is that you're, you know, you want to be, or you want to think you are, you're self deceiving. And so, you know, over the long sweep of history, a lot of people will come to this conclusion for the second reason. And most of the people who think they're the most important will be wrong. So that's all true. Um, and that certainly could apply to me and it certainly could apply to others.

• 26:57Speaker 00

  But I mean, so, so I think that's like, that's just completely fine and completely true. And I think we should have some skepticism, uh, when we find ourselves making these kinds of observations. And at the same time, I think it would be like a really bad rule or something, or a really bad norm, that every time you find yourself thinking the stakes are really high, or that you're in a really important position, that you just decide to ignore the thought. I think that would be very bad. Like if you just, if you just imagine a universe where there actually are some people who live in an especially important time. And then there's a bunch of other people who like tell stories to themselves about how, what, you know, whether they do. Um, how would you want all those people to behave? And it's like, to me, the worst possible rule is all those people should just be like, nah, this is crazy and forget about it. That's like the worst possible rule because the people who are living at the important time will then do the wrong thing.

• 27:41Speaker 00

  Um, I think another bad rule would be that everyone should take themselves completely seriously. And literally and just and just promote their own interests ahead of everyone else's. And a rule I would propose over either of them is that all these people should take their beliefs reasonably seriously and try to do the best thing according to their beliefs, but should also adhere to common sense standards of ethical conduct and not do too much ends justify the means reasoning. You know, I think whatever whatever people are trying to do to help the most important century go well, it's totally good and fine to do research on AI alignment. But people shouldn't be like going and, you know, telling lies or breaking the law in order to further their ends. That would be my proposed rules that is that we should, you know, when we have these high stakes, crazy thoughts, we should do what we can about them.

• 28:23Speaker 00

  And not go so crazy about them that we break all the rules of society. And that that seems like a better rule. That's a rule I'm trying to follow.

• 28:28Dwarkesh Patel?

  Okay, can you can you talk more about that? So if for some reason we can convince that the expected value calculation was immense that you had to break some law in order to increase the odds that AI goes well. I don't know what hypothetical this would be. True. And is it just that you're not sure that you would be right? So you just want to err on the side of caution or?

• 28:43Speaker 00

  Pretty much. Yeah. I'm really not a fan of ends justify the means reasoning. I just think it's like historically looks, that's the thing that looks really, really bad. Is is people kind of saying, you know, it's it's worth doing horrible things and coercing each other and, you know, using force to to accomplish these things that, you know, the the the ends we're trying to get to are more important than everything else. So I'm I'm against that stuff. I think that stuff looks a lot worse historically than people like trying to break the future and do helpful things. So I see my main role in the world as as trying to break the future and do helpful things. I can do that without a, you know, without doing a bunch of harmful or or common sense unethical stuff. Maybe someday, you know, there will be one of these intense trade offs. I haven't really felt like I've run into them yet.

• 29:26Speaker 00

  And if I ever ran into one of those intense trade offs, I have to ask myself how confident I really am. The current level of information and confidence I have is, in my opinion, not enough to do to do really unjustify the meansy stuff.

• 29:36Dwarkesh Patel?

  Yeah. Okay, so let's talk about the potential implausibility of continued high growth. One thing somebody might think is, okay, maybe 2% growth can't keep going on forever. But you know, maybe growth slows down to like 0.5% a year or something. And as you know, small differences in growth rates have like big effects on the end result. So by the point that we've exhausted all the possible growth in the galaxy, we'll probably be able to expand to other galaxies. What is wrong with that kind of train of logic where, you know, if there's like 0.5% growth, that still doesn't imply a lock-in? Or it'd be weird if that implied lock-in.

• 30:08Speaker 00

  I think we might want to give a little bit more context here or something. And so, so one of the key arguments of the most important century is it's just hard. It's just one of the arguments that we live in a strange time. But I'm arguing the current level of economic growth just looks too high to go on for another 10,000 years or so. And one of the points I make, which is a point I got from Robin Hanson, it's not my own originally, is that if you just take the current level of economic growth and extrapolate it out 10,000 years, you end up having to conclude that we would need multiple stuff that is worth as much as the whole world economy is today, multiple times that per atom in the galaxy. And if you believe we can't break the speed of light, then we can't get further than that. We can't get outside the galaxy. So in some sense, we like run out of material. And so you're saying, well, okay, but what if the growth rate falls to 0.5%?

• 30:52Speaker 00

  And then I'm kind of like, okay, well, so the growth rate now, I ballparked it in the post, is around 2%. That's the growth rate generally in the, in the most developed countries. Let's say it falls to 0.5%. And then I'm like, okay, so the growth rate falls to 0.5% just like for how long? For, did you calculate how long it would take to get to the same place?

• 31:10Dwarkesh Patel?

  Yeah, I think it was like 25,000 years, 0.5% gets to like one world size galaxy.

• 31:16Speaker 00

  Economy per. It's 10,000 versus 25,000.

• 31:19Dwarkesh Patel?

  But 25,000 is the amount of light years between us and like the next galaxy.

• 31:24Speaker 00

  That doesn't sound right. I don't think this galaxy calculation was very close. There's also going to be a bunch of dead space, like as you get to the outer reach of the galaxy, there's not going to be as much there. That doesn't sound super right, but let's, let's just roll with it. I mean, sure, sure. Let's just, let's just say that, let's just say that you had 2% today and then growth went down to 0.5% and just like stayed there forever. Yeah, I'm pretty sure that's still too big. I'm pretty sure you're still going to hit limits in some reasonable period of time. But also that would still be just like weird on its own. It would just be like, well, we lived in the 200 year period when we had 2% growth and then we had 0.5% growth forever. That would still make this. It's like kind of an interesting time. It would be like the most dynamic, fastest changing time in all of human history, not by a ton, but it's also like you picked the number that's like the closest and the most perfectly optimized here. So, you know, if it went down to 0.1% or

• 32:09Speaker 00

  even down to 0.01%, then it would take longer to run out of stuff, but it would be even stranger, the 2% versus the 0.01%. So I don't, I don't really think there's any way out of like, gosh, this looks like, this looks like it's probably gonna end up looking like a very special time or a very weird time.

• 32:24Dwarkesh Patel?

  This is right not getting worth getting hung up on, but like, from that perspective, then the century in which we had 8% growth because of the industrial revolution, that would be, that you would say that maybe that's the most important century.

• 32:35Speaker 00

  Oh, sure. Yeah, yeah, yeah. No, totally. No, the thing, the thing about the rapid growth is not supposed to be on its own, an argument that this century is the special one. I mean, by, by growth standards, this century looks less special than like the last one or two. It's, it's just, it's, it's saying this century is one of a handful, or I think what I say like, one of 80 of the most significant centuries or something by economic growth standards. And that's only one argument. And then I look at a lot of other ways in which this century looks unusual. But it's, but just to be clear, I mean, to say that something is the most important century of all time, sounds totally nuts, because there's so many centuries. There's just so many centuries in the history of, of humanity even, and especially if you want to think about it on galactic time scales. And even once you narrow it down to 80, it's just much way less weird. It's just like, well, if, if I've already convinced you using kind of

• 33:20Speaker 00

  non-controversial reasoning, that we're one of the 80 most important centuries, it shouldn't take me nearly as much further evidence to say, actually, this one might be number one out of 80, because your, your starting odds are more than 1%. So to get you up to 10% or 20% or 30%, it's, it's, it doesn't necessarily require a massive update the way that it would if we're just starting from nowhere.

• 33:39Dwarkesh Patel?

  But I guess I'm still not convinced that just because we, uh, this is a weird century, that has any implications for why or whether we should see Transformer to be I this century? So if we have a model about when Transformer to be I happens, and if one of the variables that goes into that is what is the growth rate in 0 AD and what is the growth rate in 2080? That just feels weird to have as a parameter and like when the specific technological development is going to happen.

• 34:02Speaker 00

  It's just one argument in the series. It's just kind of, you know, I think, I think the way that I would come at it is I would just say, hey, look at, look at AI systems. Look at what they're doing. Look at how fast the rate of progress is. Look at these like five different angles on imagining when AI might be able to do all the things humans do to advance science and technology. And, and just imagine that we get there this century, wouldn't it be crazy to have AI that could do all the things humans do to advance science and technology? Wouldn't that lead to just a lot of crazy stuff happening? Like, there's only ever been one species in the history of the universe that we know of that can do the kinds of things humans do. Wouldn't it be weird if there were two? That would be crazy. And, and one of the, one of them was a new one we built that could be like copied at will, run at different speeds, run on any, you know, any hardware you have.

• 34:44Speaker 00

  That would be crazy. And then you might come back and say, yeah, that would be crazy. This is too crazy. Like, I'm ruling this out because this is too crazy. And then I would say, okay, well, we have a bunch of evidence that we live in an unusual, crazy time. And you actually should think that there's like a lot of signs that this, this century is not just a random century picked from a sample of millions of centuries. So that's the basic structure of the argument. As, as far as the growth rate in zero AD, I mean, I think it matters. I think you're asking the question, well, why do the dynamics of, of growth in zero AD matter at all for this argument? And I think it's because it just, it's a question of like, how does economic growth work generally and how has it worked? And what is the trend that we're on? And what happens if that trend continues? So if around zero AD, growth was very low but accelerating, and if that was also true at 100 AD and 1000 AD and negative 1000,

• 35:30Speaker 00

  or, you know, 1000 BC, however you want to put it, if all those things are true, then it starts to point to a general pattern. The growth is accelerating and may be accelerating for a particular reason, and therefore you might expect more acceleration.

• 35:40Dwarkesh Patel?

  Gotcha, gotcha. Okay, so let's just talk about transformative AI then.

• 35:44Speaker 00

  Yeah.

• 35:44Dwarkesh Patel?

  Can you describe what success looks like concretely? Are humans part of the post transformative AI world? Are we hoping that these EIs become enslaved gods that help us create a utopia? What is the concrete success scenario look like?

• 36:00Speaker 00

  I mean, I think we've talked a lot about the difficulty of predicting the future. And I think I do want to emphasize that I really do believe in that. So my attitude to the most important century is not at all, hey, I know exactly what's going to happen and I'm making a plan to get us through it. It's much more like there's a general fuzzy outline of a big thing that might be approaching us. There's maybe like two or three things we can come up with that seem good to do. Everything else we think about, we're not going to know if it's good to do or bad to do. And so I'm just trying to find the things that are good to do so that I can make things go a little bit better or help things go a little bit better. That is my general attitude. So it's, um, you know, it's, I don't know. It's like if you were, if you were on a, on a ship in a storm and you saw some like very large fuzzy object obscured by the clouds, you might want to steer away from it.

• 36:44Speaker 00

  You, you might not want to say, well, what I think that is, is it's an island and I think there's probably, you know, a tiger on it. And if we go and train the tiger in the right way, blah, blah, blah, blah, blah. You don't want to get into that. Right? So that, that is the general attitude I'm taking. So what does success look like to me? I mean, success could look like a lot of things, but one thing success would look like to me would frankly, just be that we get something not too different from the trajectory we're already on. So in other words, if we could have AI systems that behaved as intended, acted as tools and amplifiers of humans, did the things they're supposed to do. And if we could avoid a world where those AI systems got sort of like, I don't know, all controlled by one government or one person, avoid a world where that caused a huge concentration of power.

• 37:28Speaker 00

  If we could just have a world where AI systems are just another technology, they help us do a lot of stuff. We invent lots of other technologies and everything is like relatively broadly distributed and everything works roughly as it's supposed to work. Then you might be in a world where we continue the trend we've seen over the last couple of hundred years, which is that we're all getting richer, we're all getting more tools. We all hopefully get increasing ability to kind of understand ourselves, study ourselves, understand what makes us happy, what makes us thrive. And hopefully the world just gets better over time and we have more and more new ideas. The ideas make us hopefully wiser. And, you know, I do think that in most respects, the world of today is just like a heck of a lot better than the world of 200 years ago. I don't think the only reason for that is wealth and technology, but I think they played a role.

• 38:13Speaker 00

  And I think that like, yeah, if you'd gone back to 200 years ago and said, Holden, you know, how would you like the world to develop a bunch of new technologies? As long as they're like sort of evenly distributed and they behave roughly as intended. And people mostly just get richer and discover new stuff. I'd be like, that sounds great. I don't know exactly where we're going to land. I can't predict in advance whether we're going to decide that that we want to treat our technologies as having their own rights. That's stuff that the world will figure out. But I'd like to avoid massive disasters that are identifiable. Because I think if we can, we might end up in a world where where the future is wiser than we are and is able to do better things.

• 38:46Dwarkesh Patel?

  Okay. The way you put it with AI enabling humans, that doesn't sound like something that could last for thousands of years. It almost sounds as

• 38:54Speaker 00

  Sure.

• 38:55Dwarkesh Patel?

  weird as like chimps saying, you know, what we would like as as humans to be our tools. At best, maybe they could hope we would give them nice zoos. But like, what is the world for humans in this in this future?

• 39:06Speaker 00

  I mean, a world I could easily imagine, although that doesn't mean it's realistic at all, is is a world where we do build these AI systems. They do what they're supposed to do. And we kind of use them to gain, you know, more more intelligence and wisdom. I've talked a little bit about this this hypothetical idea of digital people. Maybe we develop something like that. And then, you know, after 100 years of this, we've been around and we've been, people have been having discussions in the public sphere. And people kind of start to talk about whether the AIs themselves do have rights of their own and should be sharing the world with us. And then maybe they do get rights and maybe, you know, maybe some AI systems end up voting. Or maybe we decide they shouldn't and they don't. And either way, you have this kind of world where there's a bunch of different beings that all have rights and interests that matter. And they vote on how to set up the world so that we can all hopefully thrive and have

• 39:51Speaker 00

  a good time. We have less and less material scarcity, so fewer and fewer trade offs need to be made. That would be great. I don't know exactly where it ends or what it looks like. But that does, I don't know. I mean, what does anything strike you as like as as unimaginable about that?

• 40:04Dwarkesh Patel?

  Yeah, the fact that you can have beings that can be copied at will, but also there's some method of voting that

• 40:10Speaker 00

  Oh, yeah.

• 40:10Dwarkesh Patel?

  Yeah, so

• 40:11Speaker 00

  Yeah, that's a problem that would have to be solved. I mean, we have a lot of today, we have a lot of attention paid to, you know, how voting system works, who gets to vote and how we avoid things being unfair. And yeah, I mean, it's definitely true that if we had We decided there was some kind of digital entity that it should have the right to vote, and that digital entity was able to copy itself. Well, you could get some havoc right there. So you'd want to come up with some system that maybe restricts how many copies you can make of yourself, or restricts how many of those copies can vote. These are problems that I am hoping can be handled in a way that, while not perfect, could be non-catastrophic by a society that hasn't been derailed by some huge concentration of power or misaligned AI systems.

• 40:51Dwarkesh Patel?

  So that sounds like that might take time, but let's say you didn't have time. So let's say you get a call, and if somebody says, "Hold on, next month, my company is deploying a model that might plausibly lead to AGI." What does Open Philanthropy do? What do you do?

• 41:04Speaker 00

  Well, I need to distinguish. I mean, you may not have time to avoid some of these catastrophes, like a huge concentration of power, or AI systems that don't behave as intended and have their own goals. If you can prevent those catastrophes from happening, you might then get more time after you build the AIs to have these tools that help us, you know, help us invent new technologies and help us perhaps figure things out better and ask better questions. And then you could have a lot of time, or you could figure out a lot in a little time, if you had those things. But if someone said, "How long did you give me?"

• 41:31Dwarkesh Patel?

  A month.

• 41:33Speaker 00

  A month?

• 41:33Dwarkesh Patel?

  Let's say three months. So it's a little bit more...

• 41:35Speaker 00

  Yeah. I would find that extremely scary, and I think I would... I think that would do... I kind of feel like that's one of the worlds in which I might not even be able to offer an enormous amount. I think... So my job is in philanthropy, and a lot of what philanthropists do historically, or have done well historically, is we help fields grow. We help do things that operate on very long time scales. So an example of something Open Philanthropy does a lot of right now is we fund people who do research on AI alignment, and we fund people who are thinking about what it would look like to get through the most important centuries successfully. And a lot of these people right now are like very early in their career and just figuring stuff out. And so a lot of the world I picture is like it's 10 years from now,

• 42:19Speaker 00

  it's 20 years from now, it's 50 years from now. And there's this whole field of expertise that got support when traditional institutions wouldn't support it, and now is because of us. And then you come to me and you say, "We've got one week left. What do we do?" And I'm like, "I don't know. We did what we could do." You know, like go back in time and like try to prepare for this better. So that would be a lot of my answer. I mean, I could say more specific things about what I'd say in the one- to three-month timeframe, but a lot of it would be like flailing around and freaking out, frankly.

• 42:46Dwarkesh Patel?

  Gosh, okay. So maybe we can reverse the question.

• 42:48Speaker 00

  Yeah.

• 42:49Dwarkesh Patel?

  Let's say you found out that AI is going to take much longer than you thought. And you have like more than five decades.

• 42:56Speaker 00

  Yeah.

• 42:56Dwarkesh Patel?

  What changes? What are you able to do that you might not otherwise be able to do?

• 43:00Speaker 00

  Yeah, I mean, I think the further out things are, the more I think it's valid to say that humans have trouble making predictions on long timeframes. And the more I'm like interested in focusing more on other causes or on very broad things we do, such as trying to grow the set of people who thinks about issues like this. Rather than trying to specifically study, for example, how to get AI systems like today's to behave as intended. So I think that's a general shift, but I but I would say that, you know. Yeah, I tend to feel a bit more optimistic on longer timeframes because I do think that the world just isn't ready for this and isn't thinking seriously about this. And a lot of what we're trying to do at Open Philanthropy is create support that doesn't exist in traditional institutions for people to think about these topics.

• 43:43Speaker 00

  And, you know, that includes doing the AI alignment research that also includes like thinking through how we want politically, what regulations we might want to prevent disaster. I think those are a lot of the things. So it's kind of a spectrum. I would say, you know, if it's in. If it's in three months, I would probably be trying to hammer out a reasonable test of whether we can demonstrate that the AI system is either safe or dangerous. And if we can demonstrate it's dangerous, use that demonstration to really advocate for a broad slowing of AI research to buy more time to figure out how to make it less dangerous. But I don't know that I feel that much optimism. If AI is like five, you know, this kind of AI is 500 years off. Then I I'm kind of inclined to just ignore it and just try and make the world better and more robust and wiser.

• 44:26Speaker 00

  But I think if we've got, you know, 10 years, 20 years, 50 years, 80 years, something in that range. I think that is kind of the place where, you know, supporting early careers and supporting people who are going to spend their lives thinking about this. Then we flash forward to this crucial time and there's a lot more people who've spent their lives thinking about it. And I think that would be a big deal.

• 44:44Dwarkesh Patel?

  Let's talk about the question of whether we can expect AIs to be smart enough to disempower humanity, but, you know, dumb enough to have that kind of goal. You know, when I look out at smart people in the world, it seems like a lot of them have very complex nuanced goals that they've thought a lot about what is good and how to do good.

• 45:02Speaker 00

  A lot of them don't.

• 45:03Dwarkesh Patel?

  Yeah, okay.

• 45:04Speaker 00

  Yeah.

• 45:04Dwarkesh Patel?

  But does that overall make you more optimistic about AIs?

• 45:08Speaker 00

  I am not that comforted by that. I, you know, I pretty much, this is a very, very old debate in the world of AI alignment. Eli Ezeriudkowski has something called the orthogonality thesis. I don't remember exactly what it says, but it's something like you could be very intelligent about any goal. You could have the stupidest goal and be very intelligent about how to, how to get it. And in many ways, a lot of human goals are pretty silly. Like a lot of the things that make me happy are not things that are, you know, profound or wonderful. They're just things that happen to make me happy and you could very intelligently try to get those things. So it doesn't give me a lot of comfort. I think basically my picture of how modern AI works is that you're basically training these systems by trial and error. And so you're basically taking, you know, taking an AI system, you're encouraging some

• 45:52Speaker 00

  behaviors, discouraging other behaviors, and you might end up with a system that's like been encouraged to pursue something that you didn't mean to encourage. And it does it very intelligently. And I don't see any contradiction there. I think that just, just because you, you know, if you, if you were to design an AI system and you were, you're kind of giving it encouragement every time it was, you know, getting more money into your bank account. If you were encouraging it to do that, you might get something that's very, very good at getting money into your bank account to the point where it's going to disrupt the whole world to do that. And you will not automatically get something that thinks, gosh, is this a good thing to do? And I think a lot of, you know, a lot of human goals are just like, there's not really a right answer about whether our goals actually make sense. They're just the goals we have.

• 46:33Dwarkesh Patel?

  You've written elsewhere about how moral progress is something that's real, that's historically happened and it corresponds to like what actually counts as moral progress. I wonder, do you think there's reason to think the same thing might happen with AI? Like whatever the process, whatever the processes that creates is moral progress?

• 46:48Speaker 00

  I kind of don't in particular. So I've, I've kind of used the term moral progress as a, as just a term to refer to changes in morality that are good. I think there has been moral progress, but I don't think that means moral progress is something inevitable or something that happens every time you are intelligent or something like that. So I just, you know, an example I use a lot is just like attitude towards homosexuality. It's a lot more accepted today than it used to be. I call that moral progress because I think it's good. And some people will say, well, you know, I don't believe that morality is objectively good or bad. I don't believe there is any such thing as moral progress. I just think things change randomly. And that will often be an example I'll pull out and I'll say, but do you, do you think that was a neutral change? I just think it was good. I think it was good, but that's not because I believe there's some underlying

• 47:30Speaker 00

  objective reality. It's just my way of tagging or using language to talk about moral changes that seem like they were positive to me. I don't particularly expect that an AI system would have the same sort of, would have the same evolution that I've had in reflecting on morality, or would come to the same conclusions I've come to, or would come up with moralities that seem good to me. I don't have any reason to think any of that. I do think historically there have been some cases of moral progress.

• 47:56Dwarkesh Patel?

  All right. What do you think is the explanation for that historical progress?

• 47:59Speaker 00

  Well, I mean, one, one thing that I would say is that, you know, humans have a lot in common with each other. So I think some of history contains cases of humans just kind of learning more about the world, learning more about themselves, debating each other. Maybe, I think a lot of moral progress has just come from humans like getting to know other humans who they previously were stereotyping and judging negatively and afraid of. So I think there's, there's some way in which humans learning about the world and learning about themselves leads them to have kind of conclusions that are, that are more reflective and more intelligent for their own goals. But if you brought in something into the picture that was not a human at all, it might be very intelligent and reflective about its goals, and those goals might have zero value from our point of view.

• 48:39Dwarkesh Patel?

  Recent developments in AI have made many people think that AI could happen much sooner than they otherwise thought. Have, have the release of these new models impacted your timelines?

• 48:47Speaker 00

  Yeah, I, I definitely think that recent developments in AI have made me a bit more freaked out. And even since I wrote the most important century series and before that, I mean, there were years when Open Philanthropy was very interested in AI risk, but it's become more so as we've seen the progress in AI. So I think, you know, what we're seeing is we're seeing these very simple systems. It's kind of the same type of system over and over again, very general, and is able to do a lot of different tasks. One of the most interesting, so I think, I think people are interested in this. I think you should maybe Google GPT-3. There's a lot of compilations of what this very simple language model that, by the way, my, my wife and brother-in-law both worked on, so that's a disclosure. But this very simple language model that just predicts the next word it's going to see in

• 49:29Speaker 00

  a stream of text. People have gotten it to tell stories. People have got it, similar, though not identical, models to analyze and explain jokes. People have gotten it to play kind of role-playing games. Write poetry, write lyrics, answer multiple choice questions, answer trivia questions. And, you know, one of the, one of the results that I found most kind of ridiculous and strange and weird was this thing called Minerva, where people took one of these language models and with very little intervention, very little special intervention, they got it to do these like difficult math problems and explain its reasoning and get them right about half the time. And this is just not like, it wasn't really trained in a way that was very specialized for these math problems. And so I think that's like, we just see AI systems kind of having all these unpredictable, human-like abilities

• 50:10Speaker 00

  just from having this very simple training procedure. And that is something I find kind of wild and kind of scary. I don't know exactly where it's going or how fast.

• 50:17Dwarkesh Patel?

  So if you think transformative AI might happen this century, what implications does that have for the traditional global health and all being stuff that Open Philanthropy does? Yeah, I mean, will that have persistent effects of AI, if it gets aligned, will create a utopia for us anyway?

• 50:31Speaker 00

  I mean, I don't know about utopia. I mean, my, my general take is that anything could happen. And I think like my general take on, on this most important century stuff, the reason it's so important is because it's easy to imagine a world that is really awesome and is free from scarcity and that we end up in a really, you know, we see more of the progress we've seen over the last 200 years and end up in a really great place. It's also easy to imagine like a horrible dystopia. But I do think it's, you know, my, my take is that the more likely you think all of this is, the more likely you think transformative AI is, the more you should think that that should be the top priority, that we should be trying to make that go well, instead of trying to solve more direct problems that are more short-term. I'm not an extremist on this, so, you know, Open Philanthropy does both.

• 51:09Speaker 00

  Open Philanthropy works on speculative far-off future risks and Open Phil also does a bunch of, you know, more direct work. You know, again, we, we do direct and recommend a lot of money to give us top charities, which do things like distributing bed nets in Africa to help prevent malaria, treating children for intestinal parasites. You know, Open Philanthropy does a lot of advocacy for more money going to foreign aid or for better land use policies to, you know, to have a stronger economy. We do a bunch of like scientific research work that is more aimed at just like direct medical applications, especially in poor countries. So I support all that stuff. I'm glad we're doing it. And it's just a matter of like, how real and how imminent do you think this transformative AI stuff is?

• 51:52Speaker 00

  The more real and more imminent, the more relatively of our resources should go into it.

• 51:55Dwarkesh Patel?

  Yeah, no, that makes sense to me.

• 51:57Speaker 00

  Yeah.

• 51:57Dwarkesh Patel?

  But I'm curious, whatever work you do elsewhere, does that still have persistent effects after transformative AI comes? Or do you think that those will all wash out in comparison to the really big stuff?

• 52:07Speaker 00

  I mean, I think in some sense, the effects are permanent in that if you if you cause someone to live a healthier, better life, that's a thing that happened and and and nothing will ever erase that life or make that life unimportant. But I think in terms of the effects on the future, I do expect it mostly to wash out. I expect mostly mostly whatever we do to make the world better in that way, it will not persist in any kind of systematic predictable way past these kind of crazy changes. And I think that's probably how things look pre and post industrial revolution. There's probably some exceptions, but that's my guess.

• 52:36Dwarkesh Patel?

  You've expressed skepticism towards the competition frame around AI, or you know, you try to make capabilities go faster for the countries or companies you favor most.

• 52:45Speaker 00

  Yeah.

• 52:46Dwarkesh Patel?

  But you know, elsewhere, you've used the innovation as mining metaphor, and maybe you can explain that when you're giving the answer. But it seems like this frame should imply that actually, the second most powerful AI company is probably right on the heels of the first most powerful. And then so just actually, if you think the first most powerful is going to take safety more seriously, you should try to boost them. How do you think about how these two different frames interact?

• 53:10Speaker 00

  I think it's it's common for people who become convinced that AI could be really important to just jump straight to, well, I want to make sure that people I trust build it first. And that could mean my country, that could mean my friends, people I'm investing in. And I have generally, you know, called that the competition frame, want to win a competition to develop AI. And I've contrasted it with a frame that that I also think is important, which is the caution frame, which is that we need to all work together to be careful to not build something that spins out of control and has all these properties and behaves in all these ways we didn't intend. I do think we're likely, if we do develop these very powerful AI systems, I do think we're likely to end up in a world where there's multiple players trying to develop it, and they're all hot on each other's heels. And

• 53:51Speaker 00

  I am very interested in ways to find ways for us all to work together to avoid disaster as we're doing that. And I am maybe less excited than the average person who first learns about this is about like picking the one I like best and helping them race ahead. Although I am someone interested in both.

• 54:07Dwarkesh Patel?

  But if you take the innovation is mining metaphor seriously,

• 54:09Speaker 00

  Yeah.

• 54:10Dwarkesh Patel?

  Doesn't that imply that actually the competition is really a big factor here? Because.

• 54:14Speaker 00

  Oh, so the innovation mining metaphor is from another bit of cold takes. And it's an argument I make that you should think of ideas as being somewhat like natural resources in the sense of once someone discovers a scientific hypothesis, or even once, you know, once someone writes a certain great symphony, that's something that can only be done once. That's an innovation that can only be done once. And so it gets harder and harder over time to have revolutionary ideas because the most revolutionary, easiest to find ideas have already been found. So there's an analogy to mining. I don't think it applies super importantly to the AI thing, because all I'm saying is that success by person one makes success by person two harder. I'm not saying that has no impact or that it doesn't speed things up. So just to use a literal mining metaphor, let's say there's

• 54:57Speaker 00

  like a bunch of a bunch of gold in the ground. It is true that if you rush and go get all that gold, it'll be harder for me to now come in and find a bunch of gold. That is true. What's not true is that it doesn't matter if you do it. I mean, you might do it a lot faster than me. You might do it a lot ahead of me. That's a totally thing that

• 55:12Dwarkesh Patel?

  Fair enough, fair enough. So maybe one piece of skepticism that somebody could have about transformative AI is that all this is going to be bottlenecked by the non-automatable steps in the innovation sequence. So there won't be these feedback loops that speed up. Well, what is your reaction?

• 55:29Speaker 00

  Yeah, I think the single best criticism and my biggest point of skepticism on this most important century stuff is the idea that you could build an AI system that's very impressive, that could do pretty much everything humans can do, but there might be like one step that you still have to have humans do, and that could bottleneck everything. And then you could have the world not speed up that much and science and technology not advance that fast because AIs are doing almost everything, but humans are still like slowing down this one step. Or the real world is slowing down one step. So you have to do, you know, let's say, real world experiments to invent new technologies and they just take how long they take. I think this is the best, the best objection to this whole thing and the one that I'd most like to look into more. But I ultimately think that like there's enough.

• 56:11Speaker 00

  There's enough reason to think that if you had AI systems that had kind of like human-like reasoning and analysis capabilities, I think you shouldn't count on this kind of bottleneck causing everything to go really slow. And a lot of that, I mean, I write about that in this in this piece called "Weak Point in the Most Important Century: Full Automation." Part of this is just like, you don't need to automate the entire economy to get this crazy growth loop. You can automate just a part of it, that specifically parts that have to do with like, very important tech like energy and AI itself. And those actually seem in many ways just like, less bottlenecked than a lot of other parts of the economy. So you could have, you could be like, developing better AI algorithms and AI chips, manufacturing them, mostly using

• 56:52Speaker 00

  robots, using those to come up with even better designs. And then you could also be designing like, more and more efficient solar panels, using those to collect more and more energy to power your AIs. So you could, like, a lot of the crucial pieces here just actually don't seem all that likely to be bottlenecked. And then also, I mean, at the point where you have something that has the ability to have kind of like, creative new scientific hypotheses the way human does, which is a debate over whether we should ever expect that and when. But once you have that, I think you should figure there's just a lot of ways to get around all your other bottlenecks, because you have this potentially massive population of thinkers looking for them. And so an example is, you know, you could, with enough firepower, enough

• 57:32Speaker 00

  energy, enough AI, enough analysis, you could probably find a way to simulate a lot of the experiments you need to run, for example.

• 57:39Dwarkesh Patel?

  Gotcha. Now, it seems like the specific examples you used of energy and AI innovations, it seems like those are probably like, the hardest things to automate, given the fact that those are the ones that humanity's only gotten around to advancing most recently. Can you, can you talk through the intuition that those might be easier?

• 57:57Speaker 00

  I think some of the stuff that might be hardest to automate would just be stuff that, in some sense, doesn't have anything to do with software or capabilities. So, an example of something that might just be like, extremely hard to automate, might be like... Trust, like, like, you know, making, making a business deal or providing care for someone who's sick, it might just be that like, even if an AI system has all the same intellectual capabilities as a human, you know, can write poetry just as well, can have just as many ideas, can have just as good a conversation, it just, it doesn't look like a human, so people don't want that. Maybe you can create a perfect representation of a human on a screen, but it's still on a screen. And in general, I see the progress in AI as being mostly on the software front,

• 58:39Speaker 00

  not the hardware front. So AIs are able to do a lot of incredible things with language, things with math, things with board games. I wouldn't be surprised if they could write hit music in the next decade or two, but they, people really are not making the same kind of progress on like robotics, like, so, so weirdly, a task that might be among the hardest, harder ones to automate, or especially if things go fast, it might be hard to automate the task of like, taking this bottle of water and taking off the cap. Because I have this like, you know, I have this hand that is just like well-designed for that. Um, well, it's clearly not designed for that, but it's, I have these like, these hands that can do a lot of stuff. And we aren't seeing the same kind of progress there. So I think, I think there are a lot of places where like, AI systems might have kind of,

• 59:22Speaker 00

  their brains can do roughly everything human brains can, but there's some other reason they can't do some key economic tasks. And I think these are not the tasks I see likely to bother like the R and D as much.

• 59:31Dwarkesh Patel?

  Gotcha.

• 59:32Speaker 00

  This is, this is an argument I make in, in one of my more obscure cold takes posts, but I say like, you know, AI that could actually take everyone's job, like, like every human's job, might be like a lot harder than AI that could radically transform the galaxy via new technology. Cause it might be, in some ways, like, it might be easier to take a scientist's job than like a teacher's job or a doctor's job, because the teachers and the doctors are regulated. And people might just say, I want human teachers, I don't want an AI teacher. Whereas you can sit there in your lab with your AI scientists and find new theories that change the world. So, some of this stuff I think is very counterintuitive, but, but I could, I can imagine worlds where you get, you know, you get really wacky stuff before you get self-driving cars out on the road, just because of the way the regulation's working.

• 1:00:11Dwarkesh Patel?

  Gotcha, okay. Let's talk about another weak point, uh, or the one you identify as a weak point, lock-in. What do you think are the odds of lock-in given transformative AI?

• 1:00:18Speaker 00

  So lock-in is my term, or I don't know if it's my term, but it's a term I use. To talk about the possibility that we could end up with a very stable civilization. And so I talk about that as it's another post, it's called weak point in the most important century lock in, I wrote posts about the weakest points in the series. And the idea is basically like throughout history so far, let's say someone becomes in charge of a government and they're very powerful and they're very bad. Usually this is like generally considered to be temporary in at least some sense, like the kind of thing that's not going to go on forever. There's a lot of reasons the world is just dynamic and the ways the world is tend to just like not stay that way completely. The world just has changed a lot throughout history. It's kind of a dumb thing to say, but I'll get to why, why this might be important.

• 1:01:03Speaker 00

  So, you know, if someone is running a country in a really cruel corrupt way, I mean, for one thing, at some point they're going to get old and die and someone else is going to take over and that person will probably be different from them. For another thing, the world is changing all the time. There's new technologies, new things are possible, there's new ideas. And so, you know, the most powerful country today might not be the most powerful tomorrow. The people in power today might not be the ones in power tomorrow. And I think this gets us used to the idea that everything is temporary, everything changes. And a point I make in the most important century series is that you can imagine a level of technological development where there just aren't new things to find and there isn't a lot of new growth to have and people aren't dying because we've, you know, for whatever reason, we've, you know, that seems like it should be medically possible

• 1:01:47Speaker 00

  for people not to age or die. And so you could imagine a lot of the sources of dynamism in the world actually going away if we had enough technology. You could imagine a government that was able to actually surveil everyone, which is not something you can do now with a dictator who actually doesn't age or die, who knows everything going on, who's able to respond to everything. And then you could imagine that world just being completely stable. I think this is a very scary thought and it's something we have to be mindful of that if the rate of technological progress speeds up a lot, we could quickly get to a world that doesn't have a lot more dynamism and is a lot more stable. What do I think are the odds of this? I don't know. It's very hard to put a probability on it. I think when you think about, if you imagine that we're going to get this explosion in

• 1:02:30Speaker 00

  scientific and technological advancement, I think you have to take pretty Seriously, the idea that that could end by hitting a wall, that there could not be a lot of room for more dynamism, and that we could have these kind of very stable societies. What does seriously mean? I don't know, a quarter, a third, a half, something like that. I don't know, I'm making up numbers. I think it's serious enough to to think about it and think about it as something that affects the stakes of what we're talking about.

• 1:02:52Dwarkesh Patel?

  Gotcha. So I I I'm curious if you're concerned about lock-in just from the perspective of locking in a negative future, or if you think that might intrinsically be that to lock in any kind of future. If you could right now press a button and lock in a reasonably positive future, but that won't have any dynamism, or one where dynamism is guaranteed, but net expected positive is not. How would you make that determination?

• 1:03:17Speaker 00

  Well, I don't, I mean, I don't think a lot about what I would do with with unrealistic buttons where I have crazy amounts of power that I'll never have and shouldn't have. You know, I think I think of lock-in by default as like mostly a bad thing. I mean, I feel like mostly we want to at least at least kind of preserve optionality and, you know, have a world where it's not just like one person running the show with their values set up the way they want forever. I think of it mostly that way. I can imagine some future world where the civilization's been around long enough. We've learned what there is to learn and we know what a good world looks like. And most people feel pretty confident about that and they're right to feel confident. And maybe lock-in's not so bad. But I mostly think of lock-in as a bad thing. I also I also imagine that you could sort of like lock in some things about the world in order to avoid locking in others.

• 1:04:02Speaker 00

  So I can imagine if you had this enormous amount of power over how the world works. And again, this this comes down, some of this is more explained in my digital people series, how this could work. But if you had this kind of world where you completely control the environment, you might want to like lock in the fact that you never have one person with all the power. That might be a thing you might want to lock in, and that prevents other kinds of lock-in.

• 1:04:23Dwarkesh Patel?

  Do you worry about AI alignment as being a form of lock-in? So in some sense, if the goal of the research is to prevent drift from human values, then you might be just be locking in values that aren't are suboptimal.

• 1:04:37Speaker 00

  Yeah, I mostly think of AI alignment as just trying to avoid a really bad thing happening. So I mostly think of it like the thing we don't want to happen is we have some AI system. We thought we were designing it to help us. We're actually designing it to do something extremely random thing. It's just like, again, these systems work by trial and error, by encouragement, discouragement, or positive and negative reinforcement. And so we might have not even noticed that through the pattern of reinforcement we were giving, we trained some system to want to put as much money as possible into one bank account, or gain as much power as possible, or control as much energy as possible, or something like that, or just set its own reward number, its own score to the highest possible number. I think that would be a form of lock-in if we had systems more powerful than humans

• 1:05:22Speaker 00

  that had these kind of random goals. That would be like locking in kind of a future that is not related to the things that humans value and care about. That's an example of a thing I think would be really bad. Now, if we got these systems to behave as intended, we still might have problems because we might have humans doing really stupid things and locking in really bad futures. I think that's an issue, too. I feel reasonably comfortable, though not 100% confident, saying that we'd like to avoid that just like slip up. We'd like to avoid having these systems that have these random goals we gave them by accident, and they're very powerful, and they're better at kind of like setting up the world than we are. And so we get this world that's just like doing this random thing that we did by accident. I think that's a thing worth avoiding.

• 1:06:01Dwarkesh Patel?

  What is your biggest disagreement with Will MacAskill's new book on long-termism?

• 1:06:05Speaker 00

  I like Will's book. I mean, I think it's worth reading Will MacAskill's book about how the future could be very large and very important. I think he presents a picture, like, in my opinion, if you want to talk about the long-run future and how to make the long-run future go well, you're starting from a place of like, by default, almost nothing I can do will actually make sense. Like, I do really believe it's hard to understand the long-run future, and it's hard to make specific plans about it. So I would say compared to Will... I am very picky about which issues I think are like big enough and serious enough to actually pay attention to. And so I feel the issue of AI, AI would be transformative enough. It looks likely enough that it'll be soon.

• 1:06:48Speaker 00

  If it's soon, that means there might actually be things we can do that have predictable effects. I think this misaligned AI thing is a real threat. The way people design AI systems today would be really bad. I am ready to like, put some resources into preventing it. But like, that's kind of, you know, that's crossing my threshold. Most things don't. And so if you make a list of ways to make the next million years go well, I'll look at most of them and be like, I don't know. I don't really believe in this. I wouldn't really invest in this. So I think Will is a bit broader than I am in a sense. He's interested in more things and I am pickier than he is because I think it's so hard to know what's really going to affect the long run future. That I'm just looking for like a really short list of things that are worth paying special attention to.

• 1:07:26Dwarkesh Patel?

  Gotcha, gotcha. Is there a specific thing that he points out in the book that you think it'd be hard to grapple with?

• 1:07:31Speaker 00

  I don't remember super well. I mean, the book is like a really broad survey of lots of stuff. And I think, you know, an example I might give is he, he talks about, like, he talks about, it's like, he talks about the risk of stagnation, for example. So the risk that like growth might just stop or something, or growth might slow to very low levels. And then that kind of implies that what we should be trying to do is make sure we continue to innovate and continue to have growth. But then there's like other parts of the book that kind of make it sound like we shouldn't move too fast and we shouldn't innovate too much. Because we don't want to get to our future before we've like, kind of achieved some sort of civilizational maturity beyond what we have now. To decide what we want that future to look like, like we don't want to build these powers before we like, have a better idea of what to do with them. So I think these are examples of things where I'm just like, gosh, I don't know.

• 1:08:15Speaker 00

  Like it could be, it could be good to have more growth, could be bad, good to have less growth. It could be that stagnation is a big threat. It could be that like, building powerful technologies too fast is a big threat. Like, I just don't really know. I'll tell you what I'm thinking about. I'm thinking about AI because I think it's, I think it's a big enough deal and likely enough. And that we've got enough traction on some of the major risks.

• 1:08:35Dwarkesh Patel?

  Right, right. When I look throughout history, it often seems like people who are predicting long-term trends are too pessimistic. So in the seventies, you might have been too pessimistic about the ability to find more oil or feed a growing population. And because he couldn't have predicted the technological breakthroughs that might have made these things possible. Does this inform some sort of vague optimism about the future for you with regards to AI? Or no?

• 1:08:59Speaker 00

  Yeah, I think I think historically, people have been overly pessimistic about future technologies. And I think by default, the picture with AI looks just like really scary. It just looks like it would be really easy to get a bad future in a lot of different ways. If we just didn't move cautiously. So these two considerations kind of balance each other out a little bit for me. I know I know a lot of people who just who believe that we're in just deep, deep, enormous trouble. That this outcome where you get AI with its own goals, wiping humanity off the map is just like almost surely going to happen. I don't believe that. And this is part of the reason I don't believe it. It's just like, I actually think this situation looks very challenging, very scary by default. And I think we're tending to overestimate how bad and how dire things are. And so they balance out a little bit for me.

• 1:09:44Dwarkesh Patel?

  Okay, gotcha. But in many of these cases, it seems like it would be impossible to see the positive scenario come about. Like, for example, if you were forecasting population in the 70s.

• 1:09:54Speaker 00

  Sure.

• 1:09:55Dwarkesh Patel?

  Is there some reasonable method by which you would have predicted this is not going to lead to some massive famine that kills a billion people? Or would that have been your focus in the 70s if the open field was a thing back then?

• 1:10:04Speaker 00

  I think it's really hard to know how noble the future was in the past. And what that means for today. I do think that like, when you look back, and again, there's something I've looked at a fair amount. When you look back at people trying to predict the future in the past, it just looks like deeply unserious. You could say that future people will say the same about us. I'm sure they'll think we look less serious than they are. But I think there's a difference. I really do think that that attempts to attempts to kind of like rigorously make predictions about the future. There haven't been a lot of them historically, and I don't think it's obvious that people were doing the best they can and that we can't do better today. So, you know, this population is an example. Like, I'm just, I'm just.

• 1:10:46Speaker 00

  It doesn't seem necessarily true to me that you couldn't have been at that time and said, "Gosh, you know, population's been going up for a while now and people keep inventing new ways to come up with more resources. Maybe that'll keep happening." I'm just like really not convinced you couldn't have said that. Definitely not convinced no one did say it. I think some people did say it. And so I think, yeah, I'm hesitant to get too defeatist just from the fact that some people were wrong about the future in the past. I think it's hard to know if there was really no way to know or if they just weren't trying very hard.

• 1:11:19Dwarkesh Patel?

  Okay. Well, one thing you just said a minute ago was that we are better at making predictions than people in the past were. So that alone should make us more optimistic about predicting the future.

• 1:11:28Speaker 00

  It's just a guess, but I do, I do think that there's been a lot of, you know, there's been a lot of progress. I mean, this is, this is what society is, is we have a lot of progress on all kinds of intellectual fronts. And I think there has been a lot of progress on like what it looks like to make good reasonable predictions about the future. I think that's something that's happened. So I think we should be like expecting ourselves to do a bit better than people did in the past and future people will probably do better than we do.

• 1:11:51Dwarkesh Patel?

  Right. But, you know, I often, I kind of wonder when I look at a report like Biological Anchors, whether, you know, Asimov's just shooting the shit about screens and what you're able to do with them, whether he might have had less sources of error than this eight-step methodology, where you might not even be aware of there's a ninth or 10th missing step that might make the whole thing invalid and where many of the inputs have multiple orders of magnitude wide confidence intervals. What do you think of that general like skepticism?

• 1:12:19Speaker 00

  I mean, Biological Anchors, I think, is a very important input into my thinking. It's not the only input. So I think it's, you know, I think my views on AI timelines are kind of a mix of just kind of, A, looking at AI systems today, looking at what they did 50 years ago, looking at what they did 10 years ago, and just kind of being like, "Well, gosh, it sure looks plausible that these will be able to do all the things humans can do to advance science and technology pretty soon." That's one put into my thinking. Another input into my thinking is what we call the semi-informative priors analysis, which is, you know, it's a complex report because it looks from a lot of different angles, but I think you can summarize the highlights of the report as just saying, Most of the effort that has ever in the history of humanity gone into making AI will be in this century, ever.

• 1:13:04Speaker 00

  Because the field of AI is not very old, and the economy and the amount of effort invested have gone up dramatically. And I think that's a data point in favor of not being too skeptical that we could be on the cusp of transformative AIs. Like, in some sense, the world has not been trying very hard for very long. So that's an input. Another input is just like expert surveys, when people just ask AI researchers when they think AI will be able to do everything humans can. They tend to come out saying it's a few decades. Now, that can be biased and unreliable in all kinds of ways. All these things have their problems. That's a data point. And then there's biological anchors. And biological anchors is a report that kind of, you know, I would summarize it, to summarize it on a podcast, it's a very complex report. There's a lot of different angles it uses, there's a lot of different questions it asks, there's a lot of different numbers.

• 1:13:49Speaker 00

  But I think you can boil it down to some fairly simple observations. You can say that, in some important sense, which could be debated and analyzed, but seems true most ways you look at it, in some important sense, we've never built AI systems before that do as much computation per second as like a human brain does. So it shouldn't be surprising that we don't have AI systems that can do everything humans do, because humans are actually doing more work in their brains than a normal AI system is doing. However, it also looks like, within this century, we probably will have AI systems that are that big. And if we estimate how much it would take to train them, how much it would cost to build them, that looks like it will probably be affordable this century. And then you can just talk about all the different ways you could define this, and

• 1:14:34Speaker 00

  all the different ways you could quantify that, and all the different assumptions you could put in. But my bottom line is like, almost any way you slice it, it looks pretty, whatever, however you want to think about it, however you want to define what it means for an AI brain to be as big as a human's, and what it would mean to get that brain sort of trained, most angles on it suggest that it looks reasonably likely it'll happen this century. That's a data point for me. That matters.

• 1:14:56Dwarkesh Patel?

  Yeah, yeah.

• 1:14:57Speaker 00

  So all those are data points feeding into my view.

• 1:14:58Dwarkesh Patel?

  Okay, so I'm stealing this from Elliot Zer, who asked on Twitter, "Has there literally ever, in the entire history of AI, been any case of anybody successfully calling the development timing of literally any novel AI capability?" He has very complex sentences.

• 1:15:13Speaker 00

  Yeah, yeah, yeah.

• 1:15:14Dwarkesh Patel?

  Using a bio-anchored or bio-inspired calculation.

• 1:15:17Speaker 00

  Uh, I saw some discussion of this on this Facebook and I, I think the answer might be yes, but I mostly want to attack the premise. I mean, I mostly just want to say, like, there haven't been a lot of cases of people predicting AI milestones with great precision, and that's also not what I'm trying to do. Um, a lot of what I'm trying to say is I'm trying to say, gosh, this century, it looks like more likely than not that we'll get something hugely transformative. That, you know, he's, he's asking about some history of AI that's like, you know, that's like a few decades old and, and there, there haven't even been a lot of people trying to make predictions and a lot of the predictions have been way more narrow and specific than that. So I, I'm mostly just like, this isn't a very important or informative question, you know, but if, and, and I think all the work he's doing, all the things, has there ever been an example of someone using the

• 1:16:02Speaker 00

  kind of reasoning Eli Ezer is using to predict the end of the world or something, which is what he's predicting. Um, so I, I mostly just want to challenge the premise and just say like, look, like, we're not working with a lot of sample size here. Um, it's not like this is some big, well-developed field where people have tried to do the exact same thing I'm trying to do 25 times and failed each time. This is like, mostly people in academia try to advance AI systems. They don't try to predict when AI systems can do much and we're, do what? We're not working with a lot here. We have to do the best we can, make our best guess, use our common sense, use our judgment, use the angles we've got. That's my main answer. Um, but another answer I would give is I would just say that, I don't know, like, like Moravec, Hans Moravec was kind

• 1:16:44Speaker 00

  of the original biological anchors person and he, you know, I think he predicted like, artificial general intelligence like around 2020 or something. And I'm kind of like, on Eli Ezer's own views, it's going to look like he was like unbelievably close. Like that was like, maybe, you know, sure, Eli Ezer believes we'll see it by 2030. So if that's what you, and I think that's plausible. So if that's what you believe, then it's going to be, I don't know. I think we'll look back on that as like the greatest technology prediction ever. Yeah. Uh, by a lot. Yeah. I think, I think the answer is maybe. There was also some discussion in the comments about whether Moravec called like big progress on AI is doing, on AI is doing well at vision by examining the retina. There was some debate about that. I think it's all very muddy. I don't, I don't think this is much of a, a knockdown argument against thinking about biological anchors. I mean, it is only one input.

• 1:17:28Speaker 00

  Into my thinking, I do think it looks kind of good for biological anchors that like, we have seen this deep learning revolution and we have seen these brute force AI training methods working really well when they didn't used to work well. And it is when AI systems started to be about the size of like insect or small animal brains or within range, within a few orders of magnitude of human brains. You could call that a wild coincidence, but it doesn't feel, you know, probably these numbers are all off. They're probably all off by 10x, 100x, 1000x. But I mean, we're talking about very important things and trying to get our best handle and our best guess. And I think biological anchors looks fine so far. I don't know, it doesn't look amazing, looks fine.

• 1:18:04Dwarkesh Patel?

  Now, as I'm sure where many people have proposed that increasing progress in science technology and increasing academic growth are the most compelling things to be doing instead of working on transformative AI. I just want to get your broad reaction to that first.

• 1:18:20Speaker 00

  Sure. So I think we're kind of talking about, you know, maybe the progress studies crowd. Or yeah, I wrote a piece about this on Cold Taste called Rowing Steering, Anchoring Equity and Mutiny, where I, I discuss these like different ways of thinking about what it means to make the world good. And I do have some sympathy for the idea that, you know, a lot of the way the world has gotten better over the last couple hundred years is just we've, we've gotten richer, we've had more technological capabilities. Maybe we should try and do more of that. I don't think this is like a nutty thing to think. I think this is somewhat reasonable, but I overall feel that we, you know, even a couple hundred years is not that big a percentage of the history of humanity. I wrote a series called Has Life Gotten Better? That kind of asks what the whole graph of quality of life looks like over the course of humanity.

• 1:19:04Speaker 00

  And there is precedent for like technological development seeming to make things worse. That's kind of what it looks like happened in the agricultural revolution. So I have, I have some sympathy for saying, hey, this thing has been good for 200 years. Let's do more of it. But I don't think it's like the tightest, most conclusive argument in the world. And I think we do have some specific reasons to believe that developing some particular new technologies, not only AI, but also potentially like bioweapons, could just be like so catastrophically dangerous. I think it's it's something like where you could think of us, I think Toby Ord uses the analogy of humanity being like a child who's becoming an adolescent. And it's like, it's great to become stronger up to a certain point. That's fun. That that feels good. And then at a certain point, you're like, strong enough

• 1:19:47Speaker 00

  to really hurt yourself and really get yourself in trouble. Or maybe strong enough that you don't know your own strength. I think there's a pretty good case that humanity is reaching that point. I think we're reaching the point where we could have a nuclear war, or a bioweapon, or AI systems that really changed the world forever. And so it might have made sense 300 years ago, and we're all struggling to feed ourselves, to say, hey, we want more power, we want more technology, we want more abilities. Today, I think we're starting to enter the gray area. We're starting to enter the gray zone, or maybe we should slow down a little bit, be a little bit more careful. I'm not saying to literally slow down, but I'm talking about priorities. I would rather look at what I think are dramatically neglected issues that might affect all of humanity's future, and at least do the best we can to have a handle on what we want to do about them, than put my effort into throwing more

• 1:20:31Speaker 00

  juice and more gas behind this ongoing technological progress, which I think is a good thing. It's just a matter of priority.

• 1:20:37Dwarkesh Patel?

  Okay. Do you think that the entire vision of increasing progress is kind of doomed if ideas get harder to find?

• 1:20:44Speaker 00

  I do think there's eventually limits to how much, like, I think that the rate, like I said, with the atoms of the galaxy argument, but I think just a broader, you know, a broader common sense take would be that the world over the last couple of hundred years has changed incredibly dramatically. We've had just like, new, exciting technologies and capabilities every year. And I think just a good guess would be that that hits a wall at some point. I think that just, you know, it might be the atoms of the galaxy, it might just be something much more boring. I don't, it might just be like a gradually slowing, you know, actually, what we seem to observe when we look at the, when we look at the numbers, is that we are seeing a bit of stagnation, a bit of slowing down, and probably will keep slowing down by default. So, yeah, I think, I think probably a good

• 1:21:27Speaker 00

  guess is that we are, the world is changing at incredible pace that has not been the case for most of history, and it probably won't be the case for the whole future.

• 1:21:34Dwarkesh Patel?

  Okay, gotcha. I guess there's several reactions somebody could have to the idea that ideas are getting harder to find and therefore that this makes progress studies less relevant. One is just if you look at your own blog, right?

• 1:21:44Speaker 00

  Yeah.

• 1:21:44Dwarkesh Patel?

  The entire thing is just you complaining about all this low hanging fruit that people are not plucking. Nobody's thinking seriously about transformative AI. Nobody's thinking seriously about utopia. Nobody's thinking seriously about ideal governance.

• 1:21:54Speaker 00

  Sure.

• 1:21:54Dwarkesh Patel?

  How do you square this with the general thing about ideas getting harder to find?

• 1:21:59Speaker 00

  Oh, I think there's, I think there's just a ton of really important stuff today that not enough people are paying attention to. I just expect, I mean, that was also true 50 years ago. That was also true 100 years ago. It was probably more true 50 years ago than it is today. It was probably more true 100 years ago than 50 years ago. It's gradually the supply of amazingly important ideas that are not getting any attention. It's probably getting harder to do that. Harder doesn't mean impossible. You know, I, I do actually think that, that if people want to do something that's really new and world changing and dramatic and revolutionary. I think maybe the worst way to do that is to kind of go into some well-established scientific field and try to revolutionize that.

• 1:22:37Speaker 00

  I think it's better to just, just use your common sense and ask an important question about the world that no one's working on because it isn't a scientific field. Because it isn't a field of academia, because it doesn't have institutions and work on that. And a lot of my blog does advocate that. So, you know, for example, AI itself is a very well-established field, but AI alignment is a weird field that doesn't really have like academic departments right now. A lot of the stuff I'm talking about, trying to predict what the future is going to look like. It's kind of a weird low prestige thing that you can't easily explain to your extended family. And I do think that's probably the best place to look. If you want to do something that's going to be, you know, super significant, super revolutionary. That is why I've kind of professionally been drawn to it, looking for potential big wins that philanthropy could get.

• 1:23:18Dwarkesh Patel?

  But isn't another way to just read what you said, which is that we shouldn't follow in the footsteps of the greats to be a great person, you know, have great achievements yourself. Is another way to think about that, that you should probably also ignore the advice that the optical advice you're giving or 80,000 hours gives. Because those specific things are, that's not what's going to make you the next Einstein?

• 1:23:37Speaker 00

  I mean, I, a fair number of your questions are like, a fair number of like dialogue I see in, in some of the like skeptical of futurism world. It's like, it feels to me like it's almost just getting. Unnecessarily fancy or something. I kind of just want to say, who's someone who really revolutionized the way the world thinks about stuff? Well, I don't know, like Darwin. Now, what was Darwin doing? Was Darwin saying like, well, you know, I really don't want to think about this thing because like, I don't believe humans are capable of thinking about that thing. And I don't want to like, I don't want to think about, you know, this topic because like, I think it's too hard to know the future and blah, blah, blah. Was he, was he doing all that stuff? Or was he just, was he just asking an interesting question? You know, was he, was he just kind of, was he just kind of saying, hey, you know, this thing,

• 1:24:20Speaker 00

  this thing seems important. I'm going to use my common sense and judgment to figure out how it works and I'm going to write about it. And I think some of this stuff gets too fancy. So I think, you know, I think today what's going on is like, if I just look at the world and I say, what are the most important things that could matter for the world being a good or bad place in the future? I've looked at a lot of possibilities. I think that AI is one of the leading examples and I don't see a lot of people paying attention to it. And so that's what I want to work on. And I think a lot of the people who have done, you know, revolutionary work that we now look back on and a lot of people try to imitate, they weren't really doing all this stuff. They weren't trying to imitate the stuff that usually works and stay away from the stuff that wasn't. They were just asking interesting, important questions and working on them. And, you know, as far as myself and 80,000 hours, I mean, I just, I guess I just don't feel that we're well-known enough or

• 1:25:03Speaker 00

  influential enough that our advice, that that stuff we're interested in is obviously is automatically therefore not neglected. I think the stuff we're talking about is very neglected. But if you find something that's even more neglected and more important, I mean, more power to you.

• 1:25:15Dwarkesh Patel?

  Gotcha. Let's say the total amount of money given to EA just increased by an order of magnitude or something. What could be possible at that point that's not possible now?

• 1:25:25Speaker 00

  I don't know. I think, I think even then, I think the amount of money we'd be working with would be like really small by the standards of any kind of like government budget. In general with philanthropy, I'm always, I'm always looking for things where it's like, can we, can we seed the creation of a field? Can we, you know, fund people to introduce new ideas, but you know, still we're very small compared to the overall economy and the overall government. I think even multiplying everything by 10, that would still be true. I'm not sure exactly what we'd do with 10x as much money. I'm not even sure what we're going to do with, with the money that already exists. Yeah.

• 1:25:55Dwarkesh Patel?

  But do you think there will be more yea billionaires in the coming future?

• 1:25:58Speaker 00

  I would guess so.

• 1:26:00Dwarkesh Patel?

  Does that imply you should be spending money faster now if you're?

• 1:26:03Speaker 00

  Yeah, I mean, we're trying to model it, right? I mean, it's like we have, I mean, in theory, we have all these models that say like, here's our guess at how much money is eventually going to be available. And here's our guess at how many giving opportunities will eventually be there to fund. And, you know, this is our guess at like, what's good enough to fund and what's not. And we, and that's a very tentative guess. And a lot of it is just like really, you know, really, really imprecise stuff, but we have to have some view on it. Anyone who's spending money does. So, I mean, I, yeah, I do, I do tend to assume that that Sam Bankman Fried and Dustin Moskowitz are not, and Kerry Toone are not the last billionaires who are interested in doing as much good as possible. But it is really hard to model this stuff. And frankly, a lot of it, you know, we have, we have like various rough models we've made over the years.

• 1:26:45Speaker 00

  We also just will sometimes use our intuition and just say, we fund the stuff that seems like quite good and exciting, and we don't fund stuff that doesn't. And that's an input into our thinking, too.

• 1:26:54Dwarkesh Patel?

  I got you. And then how do you think about the risk that some of your giving might have negative impacts? So people have brought this up in the context of your $30 million investment in OpenAI. But, you know, in all sorts of contexts, especially when you're talking about political advocacy, people might think that the thing you do has negative side effects that counteract the positive effects. How do you think, is it just a straight EV calculation? Or how do you think about this?

• 1:27:16Speaker 00

  I mean, I think in theory, what we want is we want to make grants that have like more upside than downside, or have expected net positive effects. I think we tend to be just, you know, in a, in a common sense way, a little bit conservative with the, with the negative effects in that the thing we don't want to do is we don't want to like, enter some field on a theory that's just like totally messed up and wrong in a way that we could have known if we'd just done a little bit more homework. I think that there's just like something kind of irresponsible and uncooperative about that. And so in general, like when we are making big decisions, like big dollar decisions or going into a new cause, we often will try really hard to like, Do everything we can to understand the downsides. And if after we've done everything, you know, roughly everything we can up to some reasonable diminishing returns.

• 1:28:00Speaker 00

  And once we've done that, you know, if we believe as far as we can tell, the upsides outweigh the downsides. We're generally going to go for it. Our goal is not to avoid harm at all costs. Our although, although, you know, our goal is to kind of operate in a cooperative, high integrity way. Always doing our best, always trying to anticipate the downsides. But recognize we're going to have unintended side effects sometimes. And, and that's life and anything you do has unintended side effects. Um, I don't agree with the specific example you gave as an example of, of something that was net negative, but I don't, I don't know.

• 1:28:29Dwarkesh Patel?

  Are you talking about OpenAI?

• 1:28:30Speaker 00

  Yeah.

• 1:28:30Dwarkesh Patel?

  Okay. Yeah, many people on Twitter reminded me ask if you were investing in OpenAI.

• 1:28:35Speaker 00

  I mean, we, you know, you can, you can look up our $30 million grant to OpenAI. I think it was back in 2016 and we wrote about some of the thinking behind it. We wrote that, you know, part of, part of that grant was, was getting a board seat for Open Philanthropy for a few years. That we could help with their governance at a crucial early time in their development. You know, I think that, um, I think you could have, I mean, I think some people believe that, that OpenAI has been net negative for the world. Because of the fact that they have contributed a lot to AI advancing and to AI being sort of hyped.

• 1:29:05Dwarkesh Patel?

  Yeah.

• 1:29:05Speaker 00

  And they think that gives us less time to prepare for it. And I do think that all else equal, AI advancing faster gives us less time to prepare. It is a bad thing, but I don't think it's the only consideration. I think OpenAI has done a number of good things too, has set some important precedents. I think is probably much more interested in a lot of the issues I'm talking about and risks from advanced AI. Then like the company that I would guess would exist if they didn't, that would be doing similar things. I don't really accept that, that even, even the idea that OpenAI is, is a, you know, is a negative force. I think it's like highly debatable. We could talk about it all day. And then if you look at our specific grant, it's just that's, that's even a completely different thing because, because. A lot of that was not just about boosting them, but was about getting to be

• 1:29:47Speaker 00

  part of their early decision making. And I think that was something that there were benefits and was important. And so, yeah, I mean, my, my overall view is that that I don't, I, I look back on that grant as, as one of the better grants we've made, not one of the worst ones.

• 1:29:58Dwarkesh Patel?

  Oh, wow. Okay.

• 1:29:59Speaker 00

  But certainly we've done a lot of things that have had, you know, that have not worked out. And I think there are sometimes surely when we've done things that have consequences we didn't intend. And, you know, no philanthropist can, can be free of that. What we can try and do is be responsible. Seriously do our homework to try to understand things beforehand, see the risks that we're able to see and think about how to minimize them.

• 1:30:19Dwarkesh Patel?

  Uh, so let's talk about ethics. I think you have a very interesting series of blog posts about future proof ethics.

• 1:30:24Speaker 00

  Sure.

• 1:30:24Dwarkesh Patel?

  You want to explain what this is first?

• 1:30:26Speaker 00

  Uh, sure. I mean, I, I wrote, I wrote a, a short blog post series, just kind of try, trying to explain some of the philosophical views and ethical views that are common among people who call themselves effective altruists. And so one of the ideas I appealed to is I'm like, oh, what, what I think, and I'm not sure I'm getting this right, but what I think a lot of people I know are trying to do is they're trying to come up with a system of morality and a system of ethics that would survive a lot of moral progress or something that, that, you know, if they later became a lot wiser and learned a lot more and reflected on their morality, they wouldn't look back on their earlier actions and think they were doing horrible, monstrous mistakes. And a lot of history has just like people doing things they thought were fine and right at the time. And now we look back and we're horrified.

• 1:31:08Speaker 00

  So you could think of yourself as like, what morality could I have that would make it not so likely that, you know, if there is a bunch more moral progress and if people do learn a lot more, that the future won't look back on me and be horrified at what I did. So I wrote, I wrote a bit of a series about what it might look like to try to do that. Laid out a few principles of it and kind of tried to use this to explain the moral systems a lot of effective altruists tend to use, which tends to be some flavor of utilitarianism. Is often very expansive about like whose rights count. So effective altruists are very interested in like future generations that don't exist yet. They're interested in animals being mistreated on factory farms. They're interested in like, Various populations that a lot of people don't care about today, but that there are large numbers of. So I tried to kind of explain that. A thing that's important is I laid this

• 1:31:53Speaker 00

  view out partly so I could argue against it later, and I haven't done the latter yet. So I have a lot of reservations, too, about the ethical systems that are common with effective altruism.

• 1:32:01Dwarkesh Patel?

  Okay, okay. So let's talk about some of the pillars you laid out in this piece. So sentientism, that seems pretty reasonable to me.

• 1:32:08Speaker 00

  So sentientism, sentientism is the idea that, well, so there's, let me, let me back up. There's, there's three principles that I kind of like roughly outlined that you might want for a morality that, you know, that is going to stand up to scrutiny, or that you won't be so likely to change your mind about if you learn more and get better. So one principle is just systemization. It's like, it's better to have morality based on simple, general principles that you apply everywhere that have a morality that's just always you just deciding what feels right in the moment, because the latter could be subject to a lot of the biases of your time, and the former lets you kind of stress test the core ideas. And then two of the core ideas I propose are what I call thin utilitarianism, which is basically the greatest good for the greatest number, and sentientism, which

• 1:32:48Speaker 00

  is basically saying that someone counts or someone matters if they're able to suffer or have pleasure. And I think you just said sentientism seems reasonable to you. I think sentientism might be the weakest part of the picture to me. I think you, if you have a morality where you are insistent on saying that everyone counts equally in proportion to the amount of pain or pleasure they're able to have, you run into a lot of like weird dilemmas that you wouldn't have to run into if you didn't have that view. So I think it's very strange, but I think it is like, actually, one of the more questionable parts of the view. It's kind of saying, you know, when I'm deciding whether I care about someone, it doesn't matter at all if they're way in the future, if they're way far away, if they're totally different from me, if they're not human, if I've never met them,

• 1:33:30Speaker 00

  all that matters is if they can have pain or pleasure. I think it sounds great, and I completely get why someone listening to it would be like, how could you ever disagree with that? But I do think, I do think there's various challenges with it, which I have not had the chance to write about yet, and I doubt I can be very convincing on this podcast yet, because I haven't thought about it.

• 1:33:46Dwarkesh Patel?

  Yeah, all right. Sounds good. Let's talk about the other two. So

• 1:33:48Speaker 00

  Yeah.

• 1:33:49Dwarkesh Patel?

  Systemization.

• 1:33:50Speaker 00

  Yeah.

• 1:33:51Dwarkesh Patel?

  Doesn't the fact that you have lots of, you know, complex and sometimes contradictory moral intuitions, does that suggest that maybe the whole goal of having some fundamental principles you extrapolate the rest of morality from, that's a kind of doomed project?

• 1:34:03Speaker 00

  I think it does somewhat suggest that. I mean, I am somewhat partial to that view, and that's something I may be writing in my rebuttal. But I also think it's possible to just be confused. And I think it's possible to just have lots of stuff going on in your brain, and some of it might be based on like, really good, really good intentions of treating other people fairly and being good to other people. And some of it might be based on just like, other weird stuff about like, wanting to stand up for people who look like you, or help people who look like you, and things like that. And so I think it's, I think it's, I do have some sympathy for the project of trying to say, you know, my intuitions contradict each other, but some of them are coming from a good place, some of them are coming from a bad place. If I thought more about it, I would realize which ones are which, and I want to try and do that.

• 1:34:44Dwarkesh Patel?

  Yeah, yeah. Let's talk about the utilitarianism. So, there's this question from an old Scott Alexander post, where he asks, "Would you rather the medieval church spent all of its money helping the poor, rather than supporting the arts?" So that maybe there were fewer poor people back in the medieval times, but you wouldn't have any cathedrals, or you wouldn't have the Sistine Chapel. I don't know how you would answer that if you were in medieval times.

• 1:35:05Speaker 00

  It doesn't sound like the strongest version of this argument to me, to be honest. Like, I think maybe that would be fine, or good. I don't know. Maybe if I was a bigger fan of, I mean, I don't know, my wife really loves these like, old churches, so maybe if I had more of her attitude, I would be more horrified by this idea. I think that, uh, yeah, I mean, like, low-income people had a rough time in the past, and having better lives seems pretty appealing. So I don't really know if that's the best version of, I don't know if this is the best version of this argument.

• 1:35:33Dwarkesh Patel?

  All right. So, how much of future of ethics is basically, you're very confident that a future Holden will have a much more developed and better set of ethics? Or how much do you think people in general, or humanity in general, will get better ethics over time?

• 1:35:45Speaker 00

  Yeah, this has been definitely a point of confusion in this series and partly just something I think I didn't communicate well about and makes the series like not that amazing. Is just that I use the term moral progress and I just use it to refer to like things getting better. I think sometimes there is such a thing as thinking more about your morality, gaining some insight and ending up in a better place as a result. I think that is a thing that is real. There are, there are some people who believe morality is an objective truth, but I'm not one of those people. But even though I believe morality is not objective, I still think there's a meaningful notion of moral progress. There's such a thing as having more reasonable moral views than I used to. What I didn't mean to say is that moral progress has any inevitability about it. I didn't mean to say that moral progress necessarily happens just because time goes on. I don't think that. I just

• 1:36:30Speaker 00

  think it's a thing that can happen. So I do think a future Holden will probably be better at morality, just because I'm really interested in the topic and I'm going to keep trying to improve at it. And I think that, you know, we have some reason to think that actually just like does help a bit, a really tiny bit. But I'm not confident in that at all. And I certainly don't think that society is going to have moral progress just necessarily. But I do think we've had some in the past.

• 1:36:52Dwarkesh Patel?

  Okay, but then, I don't know, then it seems weird to like label the system of ethics future-proof ethics, right? Maybe it should be future Holden-proof ethics.

• 1:37:00Speaker 00

  Yeah, possible. I mean, I talk about this a bunch in the series and I think I just didn't do a great job with this. Yeah, I mean, I think what I was trying to do is I was trying to use a term that you didn't have to be a moral realist to, to get behind. And what I was really trying to capture is like, can I think now to reduce the odds that if later I improve, that I'll be horrified by my early actions? That was what I was trying to capture the concept of. I'm not sure I really did it successfully.

• 1:37:26Dwarkesh Patel?

  Gotcha. Okay, so you had a recent post on EA Forum that I thought was really interesting. And a quote from that is, "My view is that for the most part, people who identify as EA's tend to have unusually high integrity, but my guess is that this is more despite utilitarianism than because of it." So what do you think is the explanation for this coincidence where a group of reasonable, non-fanatical, high-integrity people also happen to be a community of utilitarians?

• 1:37:53Speaker 00

  You might have a set of people who are Who think of themselves as trying really hard to be like the kind of person they should be, or really hard to bring their actions into line with their beliefs and their statements. And so that drives them to be kind of like honest a lot of the time and follow a lot of our common sense rules of morality, but it also drives them to like really try to get that ethics right and land on ideas like utilitarianism that are very like systematic and pure and like give you sort of this like clear theoretical guidance. So it could drive both those things. Whereas I believe that if you're, if you're a utilitarian, it's really unclear whether utilitarianism actually tells you to do things like avoiding lying. Some people think it does. Some people think it doesn't. I think it's very unclear.

• 1:38:35Dwarkesh Patel?

  You've advocated for the moral parliament's approach when you're trying to make decisions. What is the right level of organization at which to use that approach? Is it, should individuals be making decisions based on having multiple different moral parties inside them? Is that the right approach for entire movements, but individuals should be specializing? What is the right level to be applying this approach at?

• 1:38:57Speaker 00

  I mean, this is something I hope to write about in the future is this topic of moral uncertainty. And the basic idea is like, there might be a bunch of different ways about thinking about what the right thing is to do in the world. And one, you know, you might look at the world from one angle and say, well, what matters is like the total sum of all the pleasures. So therefore a bigger world would be better. So therefore I should be like really obsessed with like getting the world to be as big as possible. There might be another perspective that says that what really matters is suffering. We should minimize suffering if we want the world to be small. There might be another perspective that says, you know, it doesn't matter what happens to the world. It matters how I act. What matters is that I act with integrity, that I tell the truth, things like that. And, you know, there's these interesting debates about what, what do you do when you think you have some sympathy for all these views? And then how do you choose an action that, that some perspectives would say is the best thing you've ever done and some would say is the worst thing you've ever done?

• 1:39:42Speaker 00

  And, you know, the moral parliament idea is an idea that was laid out by Nick Bostrom in an overcoming bias post, like something like a decade ago that I like. I think about it as if I'm just multiple people. I just think about it as if I'm just, there's multiple people all living inside my head arguing about what to do. And they all are friends and they all care about each other and they want to get along. And so they're trying to reach a deal that all them can feel fairly good about. That is how I tend to think about dealing with different moral views. So I tend to want to do things that are like really good according to one and not too bad according to the rest and, and try to, you know, have, have the kind of Have the kind of different parts of myself making deals with each other. So that is, you know, that relates to something I said at the beginning about not being into ends justified the means like I put a lot of effort into doing things that would be like really,

• 1:40:26Speaker 00

  really good if this most important century stuff came out true, but also not too catastrophic if it didn't like there's, there's lines I'm not willing to cross. There's behaviors I'm not willing to engage in to promote the kind of, you know, goals of people who worry about AI safety. So it's, it's a, it's a moderating approach, I think.

• 1:40:43Dwarkesh Patel?

  Gotcha. Okay.

• 1:40:44Speaker 00

  Yeah.

• 1:40:44Dwarkesh Patel?

  That, that makes a lot of sense for somebody who's the CEO of Open Philanthropy that you would want the decisions you make to reflect uncertainties about your decisions. But if it's just somebody like me where I, I, I'm not in some sort of leadership position where I have a large amount of resources dedicated to, should I just specialize in that particular moral view I have, or should I also be trying to allocate my time and resources according to different moral views?

• 1:41:07Speaker 00

  I mean, I think no matter what position I was in in the world and however many resources I had, I would feel that my decisions were significant in some sense and affected people and were important in that way that they, that they affect those around me. So I think it's just very natural to me to think, you know, there's a lot of different perspectives on what it means to be a good person. And rather than trying to turn them into a single unifying mathematical equation and take the expected value, which is another approach that I think is interesting. But I think the approach I do tend to prefer is to imagine the different perspectives as different people trying to get along and make a deal with each other.

• 1:41:38Dwarkesh Patel?

  Well, let's talk about governance and management. So in software, as I'm sure you're aware, there's a concept of a 10x engineer.

• 1:41:44Speaker 00

  Yeah.

• 1:41:45Dwarkesh Patel?

  Is there something similar in the kinds of work a research analyst at Open Philanthropy does? Is there somebody, i- i- is it meaningful to say that two people doing the same job, one can be orders of magnitude more effective than another?

• 1:41:56Speaker 00

  Yeah, I think some people are much, any given thing in Open Philanthropy, some people are much better at it than others. I don't think that's very surprising. I think it's true in many jobs and I don't, I don't really know the reasons for it. Could be any combination of talent, interest, how hard someone works at it. So yeah, but I, I certainly think there's a lot of variance in, in hiring people who can do a great job at the work Open Phil does has been a, you know, a lifelong challenge.

• 1:42:19Dwarkesh Patel?

  You've written about the Bayesian mindset. You know, many billionaires, many of them are donors to Open Philanthropy. In your experience, do these startup founders who end up becoming very successful, do they have a Bayesian mindset, or is that the wrong way to characterize their?

• 1:42:33Speaker 00

  Yeah, I wrote about this idea of Bayesian mindset, which is basically like being willing to kind of put a probability on anything and use your probabilities and say your probabilities as a way of discovering why it is you think what you think, and using expected value calculations similarly. I think this is like much more common among successful tech founders than it is among like the general population. But there's plenty of tech founders who don't think this way at all. I don't think, you know, I say in the Bayesian mindset, I don't think it's like a super well-tested, well-proven social technology that does amazing things. I think it's more like an interesting thing to be experimenting with.

• 1:43:07Dwarkesh Patel?

  Okay, that seems...

• 1:43:09Speaker 00

  Well, the general population, I mean, it's practically unheard of.

• 1:43:12Dwarkesh Patel?

  Right.

• 1:43:12Speaker 00

  Bayesian mindset, yeah. I mean, not even just the name. I mean, I just, just the idea, this whole idea of like thinking expected value and subjective probabilities all the time. It's just like, almost no one does that. I think tech founders probably do it more than like the average person.

• 1:43:25Dwarkesh Patel?

  No, that makes sense. Do you think that adopting more Bayesian mindset would help somebody at the top levels be more successful?

• 1:43:31Speaker 00

  It's really TBD and unclear. I think, I just think the Bayesian mindset is a, is a cool thing to experiment with. I experiment with it a lot. I feel like it helps me sometimes. Like most things, it's good in moderation and with taste and not using it for every single thing. And maybe 10 years from now, as it gets more popular, or a better sense of where the actual applied strengths and weaknesses are.

• 1:43:48Dwarkesh Patel?

  As I'm sure where there's many prizes floating around for all kinds of intellectual work in effective altruism, and some of them have come from Open Philanthropy.

• 1:43:55Speaker 00

  Yeah.

• 1:43:56Dwarkesh Patel?

  Are you optimistic about their ability to resurface or surface new ideas?

• 1:44:01Speaker 00

  I would say I'm like medium optimistic about the impact of all these prizes. I mean, I've, I've been part of designing some of them. I've just seen some other ones, like people will say, you know, "Hey, we'll pay you X dollars if you can give us a good critique of our..." Like GiveWell will pay people to give them a good critique of their, of their reasoning about, about what the best charities already give to. Open Philanthropy has a prize for like showing us a cause we should be looking at that we're not. I don't know. I think I'm like medium optimistic. I mean, I think, I think it will get some interest and it will get some people to pay attention who weren't otherwise, and some of those people might have good ideas. I don't think it's like the only way to, to solve these problems or that it will automatically solve them. And that's generally how the people designing the prizes think about them too.

• 1:44:38Dwarkesh Patel?

  You have an interesting post about stakeholder management that over time institutions have to take into account the interest of more and more stakeholders. Do you expect that this will be something that will be a major factor in how open philanthropy acts in the future? And do you think, what will be the impact on how open philanthropy runs overall?

• 1:44:54Speaker 00

  Yeah, I think in general, the bigger your organization is, the bigger your city is, the bigger your society is, there's more people who are, you know, if you want everyone to be happy, there's more people you're going to have to make happy. And I think this does mean, you know, in general, by default, as a company grows, it gets less able to make a lot of disruptive, quick changes. A lot of people would use the term nimble. A lot of people in the tech world like to just use these like very negative terms for big company properties and very positive terms for small company properties. So, small companies are like nimble and quick and practical and adaptive and dynamic and high productivity and big companies are like bureaucratic and slow and non-adaptive. I think that's all fair. I also think that big companies often just like, at the end of the day, just like produce more stuff than they could if they were small.

• 1:45:37Speaker 00

  I mean, like, you know, I think, I think if, if Apple were still 10 people, it might be a more exciting place to work, but they wouldn't be able to make all those iPhones. You know, there's a lot of iPhones going out to a lot of people, serving a lot of different people's needs, abiding by a lot of like regulatory requirements. There's a lot of work to be done. So, I don't think, you know, I don't think it's necessarily a bad thing, but I think it's a trade-off for a company to grow. I do think open philanthropy is, you know, in the business of doing kind of unconventional giving and using a lot of judgment calls to do it. So, I tend to think we benefit a lot from staying as small as we can and I generally have thought for us to small, stay as small as we can while doing our work, but we still have to grow from where we are.

• 1:46:17Dwarkesh Patel?

  Gosh, do you mean stay small in terms of funds or do you mean stay, okay, people? Yeah, yeah, yeah. But it seems odd to say that the organization you have the most experience with, your inside view is that more stakeholders would be bad, but overall, it's been a net zero positive.

• 1:46:30Speaker 00

  It's not clear, like, we're, we're growing. I mean, we're bigger than we were a year ago. We'll be bigger in a year. So, I, it's definitely not true that I'm just like minimize the size of the company. I mean, we're growing. But I think we wanna, you know, I think we wanna watch it. I think we wanna treat each hire as as something that we only do because we had a really good reason to. And I think there's some companies that may have more more to gain from being 10,000 people. I don't think we'll ever be 10,000 people.

• 1:46:52Dwarkesh Patel?

  Right, right, right. Now, your written career advice emphasizes building aptitudes and specializing. But when I look at your career, you know, it's all over the place, right? You started, we were just talking about at the beginning of the interview, you started off in GiveWell, then you were working at Open Philanthropy, and now you're forecasting AI. Like, so how do you think about this kind of thing? Where are you specializing? Or what's going on here?

• 1:47:15Speaker 00

  I don't know if I really forecast AI. I mostly I mostly distill and bring together analyses that others have done, and I manage people who work on that sort of thing. Yeah, I mean, I don't know. I think it's I think it's really good. I mean, the career advice I often give is just like, it's really good to just have something you're very focused on, that you're specializing in, that you're trying to be the best in the world at. You know, the way that my career went is like, first off, the the general theme of my career is just taking questions, especially questions about how to give effectively, where just like, no one's really gotten started on this question. So even doing a pretty crappy analysis can be better than what already exists. Often, what I what I have done in my career, what I consider myself, you know, to have kind of specialized in, in a sense, is I do the first cut crappy analysis of some question that has not been analyzed much and is very

• 1:47:59Speaker 00

  important. And then I build a team to do better analysis of that question. So that's that's been my general pattern. I think that's the most generalizable skill I've had. But I have switched around, because I do think that, you know, that I've I've kind of, at various points in my career, just said, hey, here's something that's getting very little attention, and it's very important, and it's worth the sacrifice of the specialized knowledge I've built up in one area to switch into this other area that I think I ought to be working on.

• 1:48:22Dwarkesh Patel?

  What does the logo on the Colthay's blog mean?

• 1:48:27Speaker 00

  There is no logo. I think you're talking about the browser icon.

• 1:48:29Dwarkesh Patel?

  Yeah, yeah.

• 1:48:30Speaker 00

  So that is, that is a stuffed animal named Mora. And at some point, if I get enough subscribers, I will explain who all these stuffed animals are. But it's my my wife and I basically use a stuffed animal personality classification system, where we will like, compare someone to various stuffed animals to explain what their strengths and weaknesses are. And Mora is a pink polar bear, who's like, very creative, but also very narcissistic and loves attention. So she's kind of the mascot of the blog, because it's it's kind of this blog that's just like, very crazy, very out there, and is just like me writing in public. And so it just felt like her spirit.

• 1:49:05Dwarkesh Patel?

  Gosh, okay. So, well, let me ask, what is the goal of the blog? Having a second job as a vlogger in addition to being the CEO of a big organization?

• 1:49:11Speaker 00

  No, I think it fits into my job reasonably well. I didn't want it to be open philanthropy branded. I just wanted the freedom to write about things the way I wanted and how I wanted. But I do think that like, we make these high stakes decisions based on very unconventional views about the world. And I think it's good for us to be trying to make those views have contact for the rest of the world. So I think there would be, there would be something not ideal about being a large foundation, giving a large amounts of money. And we're just kind of quietly going around believing these enormously important of true things that no one else believes. And it is a way of kind of, you know, if we put the views out into the world, A, I think just like, all the people seeking money from us just have a better idea of where we're coming from. Just like, why is it that we're, that we're interested in funding what we're interested in funding?

• 1:49:52Speaker 00

  And I think to the extent people find the arguments compelling, or even just like, understand them and helps understand our thinking, I think that can help create more grantees for us. It can, it can help, you know, cause the world to be a place where there's more good stuff for us to fund, because more people see where we're coming from, hopefully agree with it, and are trying to work on the things we consider important. And then to the extent that my stuff is actually just like screwed up and wrong, and I've got mistakes in there, and I've thought it all through wrong, this is also like the best way I know of to discover that. It's just, um, you know, I don't know how else I'm, how else I'm going to get people to critique it, except by putting it out there and maybe getting some attention for it. So it's, it's, uh, that's how I kind of think of it, is it's, it's taking views that are very important to the decisions we're making,

• 1:50:33Speaker 00

  and trying to express them so that we can either get more people agreeing with us who we're able to fund and support and work with, or, or learn more about what we're getting wrong.

• 1:50:41Dwarkesh Patel?

  All right, so let me actually ask you, has that happened? Like, has there been an important view that are expressed on the blog that because of feedback you change your mind on? Or, or is it mostly about the communication part?

• 1:50:50Speaker 00

  Let me think. I mean, there's definitely been like a lot of interesting stuff. I mean, like, an example is I put up this post on the track record of futurists. And then there was a post by, by I think Dan Lu recently that I haven't read yet. But there just has its own analysis of the track record of futurists, and I need to compare them. You know, think about what I really think about how good humans have historically been about predicting the future. But he has, he certainly has a ton of data in there that I was not aware of, that, um, you know, that feels like, feels like a bit of a response. May or may not have been prompted by it. Yeah, there's been, there's been like a lot of commentary. There's been a lot of like, you, you wrote some critiques of some of the stuff I've written in the most important century. There have been other critiques, I mean... I don't know, like I think it, I think a lot of the stuff I wrote about the biggest weak points in the most important century were based on the public criticism that was coming in.

• 1:51:33Speaker 00

  So I think it has like, yeah, I think I have become more aware of a lot of the like, the parts of my, parts of my thinking that are the least convincing, or the most weak, or the most in need of argument, and I have like paid more attention to those things because of that, yeah.

• 1:51:45Dwarkesh Patel?

  Right, okay. This may just be me talking, but it actually does sound like maybe you've learned about how people react to the most important century thesis, but it doesn't seem like something has surfaced which has made you change your mind on it a lot.

• 1:51:57Speaker 00

  Oh, I mean, I haven't dropped my, I mean, that would be a big, a big change, like to drop my view that we could be in the most important century for humanity, like, that's still what I believe, I'm, you know, I mean, I think I've also heard from people who just think I'm like underselling the whole thing like crazy, like people who just think that, that I should be planning on transformative AI much sooner than, than what I say, than what I kind of imply in the series. So, yeah, I mean, I wouldn't, I mean, I put out this thing, most important century, and I don't believe any of the critiques have been deep enough and strong enough to make me just like drop that whole thing, but it's, it's a big picture with a lot of moving parts, and I have kind of deepened my, you know, my understanding of many of the parts.

• 1:52:34Dwarkesh Patel?

  Yeah, yeah. One thing I find really interesting about your work is how much it involves the CEO having a deep understanding of all the issues involved. You are the one who has to like deeply understand, for example, we just have Memorial Parliaments, or whether it's like specific forecast about AI, like biological anchors, or whatever else, right? It seems maybe in other organizations, the CEO just delegates this kind of understanding and just asks for the bullet points. Is this something you think more leaders should be doing, or is there something special about your position?

• 1:53:05Speaker 00

  I know much less about any given topic than the person who specializes in the topic, and I think what I try to do is I try to know enough about the topic that I can manage them effectively. And that's like a pretty general corporate best practice, and I think it just varies a lot. So I think, for example, like, something like keeping our books, like keeping our finances, doing the financial audits, all that stuff, that's something where it's like really easy to just judge the outputs without really knowing much about finance at all. You can just say, look, was this compliant? Did we do our audit? Did we pass the audit? You know, do we still have money in the bank? Like, what's, how much money do we have in the bank? You don't need to know much about it to judge it effectively, but at any company, I mean, you, you may need You may need to know a fair amount about the top about some other topics in order to judge them very effectively. If your company is making computers or phones and design is very important,

• 1:53:49Speaker 00

  you pro the CEO would be really bad if the CEO just had like no opinions on design. It was just like, well, I'm gonna let our design person decide the design. I mean, it's it's a central thing to the company. It matters to the company and they should know some things about it. So I do, you know, things that are really central to open philanthropy. What is what does it mean to do good? How do we handle uncertainty about how to do good? What are the most important causes? If AI might be one of the most important causes, then when might we see transformative AI? What would that mean? How big is the risk of misaligned AI? I think I need to understand those issues well enough to effectively manage people who know a lot more about them than I do. Yeah, and I'm just curious, like, what do you know, what do you think about this whole most important century stuff? Does this just strike you as like crazy? Or like, you know, what do you think when you read the series?

• 1:54:29Dwarkesh Patel?

  Yeah, I've been obviously like through that entire interview, I've been trying to like nitpick small things. But when I really think about what is the main claim you're making is that this could be the most important century transformative AI could happen this century. And if it does, it's a really big deal. It's like, yeah, I don't disagree. That makes a lot of sense. Throughout, like preparing for the interview and trying to come up with objections, I've just been like a little bit more and more convinced with thinking about like, is there actually something I could do over my early career that matters? Or is that something that maybe I should just hold off on thinking about?

• 1:54:56Speaker 00

  Glad to hear it. Do you have any ideas what you might do?

• 1:54:58Dwarkesh Patel?

  Hmm, uh, no.

• 1:55:00Speaker 00

  Really? Really? Literally no ideas? You haven't been like, could I work on AI alignment?

• 1:55:04Dwarkesh Patel?

  Oh, well, yeah, in that sense, I've thought a little bit about it. Probably in like two months or three months, I'll think real hard about what actually I want to do for a career.

• 1:55:13Dwarkesh Patel?

  Hey, thanks for listening. If you enjoyed that episode, I would really appreciate it if you could share it. This is still a pretty small podcast, so it is a huge help when any one of you shares an episode that you like. Post it on Twitter, send it to friends who you think might like it, put it in your group chats, just let the word go forth. It helps out a ton. Many thanks to my amazing editor, Graham Besselou, for producing this podcast, and to Mia Ayana for creating the amazing transcripts that accompany each episode, which have helpful links, and you can find them at the link in the description below. Remember to subscribe on YouTube and

• 1:55:56Dwarkesh Patel?

  your favorite podcast platforms. Cheers. See you next time.