Robin Hanson: AI is probably the End of Humanity (and why that's OK)

Robin Hanson believes the future of AI is far stranger than most people imagine. In this episode, economist, futurist, and The Age of Em author Robin Hanson explains one of the biggest questions humanity has faced: What happens after humans? Instead of viewing AI as humanity's greatest threat, Hanson argues we should think of AI as our descendants. The conversation looks at why AI replacing humans may be part of civilization's natural evolution, why capitalism and competition aren't disappearing, and how cultural evolution, not technology, may be humanity's biggest existential challenge. We also examine the Great Filter, the future of work, AI regulation, software engineering, technological progress, fertility decline, civilization collapse, AI governance, and why Robin believes our greatest risk isn't AI taking over, but humanity losing its ability to evolve culturally. If you're interested in the future of AI, artificial intelligence, AGI, economics, civilization, technology, and what comes after humanity, this conversation will challenge many of your assumptions. Subscribe and like this video for more conversations with the world's leading thinkers on AI, technology, innovation, and digital transformation. In this episode: 00:00 Intro 01:00 AI will eventually replace humanity 01:36 Robin Hanson’s background 02:20 The future of civilization 03:22 The Age of Em explained 06:03 Why brain emulation changes everything 07:24 Capitalism isn't going away 11:18 Is humanity actually progressing? 13:10 The Great Filter explained 18:17 Why AI isn't the Great Filter 22:02 AI, technological acceleration & the next 20 years 24:24 Will AI leave people behind? 29:04 How to survive technological disruption 33:08 Why AI is humanity's descendants 40:23 Cultural evolution is breaking 47:12 Why civilization is becoming maladaptive 55:06 Is modern society quietly declining? 01:00:45 Can AI solve civilization's biggest problem? 01:07:01 AI risk vs AI progress 01:12:00 Should AI be regulated? 01:17:03 The US, China & the AI race 01:22:26 AI's economic impact 01:25:19 Advice for business leaders 01:30:45 AI's biggest opportunity for humanity 01:38:48 What gives Robin Hanson hope 01:42:24 Final thoughts Connect with Robin: X: https://x.com/robinhanson Facebook: https://www.facebook.com/robin.hanson.754/ Our links: Visit our website: https://www.infotech.com/?utm_source=youtube&utm_medium=social&utm_campaign=podcast Follow us on YouTube: https://www.youtube.com/@InfoTechRG

• 0:01Speaker 00

  AIs would eventually be more capable than non-AIs and they would therefore have to run the world and they would dominate what happened in the world and then the rest of the world would be at their mercy. And that seems a pretty robust prediction for the future because that's what's happened in the past. You know, industrials replaced farmers, farmers replaced foragers, foragers replaced other primates. Yeah, you're just going to expect that in the long run, the new form comes to dominate. And then old people should try to like switch over and join the new world as best they can.

• 0:32Speaker 01

  This is a show about the future of tech and the future of work. I'm Jeff Nielsen and today my guest is Robin Hanson. A revered economist, professor, speaker and author, I prefer to think of Robin as a thinker with one of the most fascinating minds and approaches to AI and the future of humanity. He's the author of The Age of M, a detailed look at the future of AI-driven brain emulation and has written and spoken extensively about the arc of civilization. This is a guy who thinks it's inevitable that AI replaces humans, not just in the workplace, but almost certainly as a species. Every conversation I've heard with him has been mind-expanding and changed the way I think about the world. So I couldn't

• 1:15Speaker 01

  wait to have him on the show. I want to ask him what the future he sees for us humans is, both in the next handful of months and in the next million years. Should we be excited for where we seem to be headed? Or do we need to radically alter our trajectory if we're going to have a future worth living in at all? Let's find it.

• 1:37Speaker 01

  Well, Robin, thanks so much for joining today. Really excited to have you. And you know, you're someone who has such interesting views on such a wide variety of topics and are clearly someone who has spent a lot of time reflecting on and refining your own sort of understanding of humanity, the world, the universe. And so, you know, maybe I'll start with a big question, which is, you've said before that sometimes you catch flack for talking about what comes next in the arc of civilization or the future of humanity, because what you expect to happen is not always the same as what you would like to happen or what anybody else would like to happen. So with that in mind, where do you see us as a civilization going from here?

• 2:21Speaker 00

  So I have this book from 10 years ago called The Age of M, and it describes this world of a certain kind of AI. And I think it was a pretty positive image that many other people saw it as dystopian. So that might be the kind of thing you're referring to, where I was describing what I think is a plausible, but pretty strange scenario, because I just think the future is likely to be pretty strange. That's just my default. Now, in the last few years, I've come to fear and maybe even expect that this future I wrote about in this book will be put off for longer than I had hoped. That is, I fear-

• 3:01Speaker 01

  Go ahead. Sorry, I was gonna say, let's maybe talk just briefly about the future envisioned in your book, The Age of M. And M, I understand, is short for emulated minds.

• 3:12Speaker 00

  Right.

• 3:13Speaker 01

  So can you tell us, you know, what was this future that you envisioned? And was it designed to be a sign of what's to come, or just an interesting thought experiment?

• 3:22Speaker 00

  So for a long time, many people have thought that the next really big change likely to happen would be some form of artificial intelligence. And, you know, in the last few years, that becomes an even more popular opinion, but this was 10 years ago. And a big problem with analyzing the future of AI is just there's such a vast space of possible AIs, you could imagine. And even today with LLMs, there's still a vast space of what they could become. Now, there's one kind of AI, which is much easier to predict than the others, which is where you take an individual human brain, and then you make a computer emulation of that brain, and then you run that and make copies, et cetera. The reason

• 4:05Speaker 00

  that's easier to predict is we just know so many things about human brains, and their trade-offs and their inclinations, et cetera. And so now, we can take all that things we know about humans and then lay out a much more detailed scenario of wow, that plays out. Now, that's not, obviously, the first kind of AI we're going to get. And so that means this book is somewhat speculative, assuming it's the first kind of AI gap. But it's not crazy to think that it would be the first human-level AI. I still think that's a substantial chance. And so, you know, it's both a likely scenario. That is, like, I'd say, if it's worth having 100 books on the future, it's worth having a book on a future

• 4:46Speaker 00

  that has a 1% chance. And this, I think, meets that criteria. It's got at least a 1% chance. So in that way, it's a serious attempt to actually think about the future. But in addition, it's also an attempt to think through at least one specific AI scenario in a lot more detail, and to show that it's possible to take specific technological scenarios and work a lot of detail about them. So in my life, many people have just said, it's really impossible to really say much about the future. There's so many possibilities. It's spread out so rapidly into such a weird, high-dimensional space that you really just can't say anything. Therefore, science fiction or speculation is just the best anybody can

• 5:26Speaker 00

  do. And I was trying to show, no, you can actually take a relatively specific set of assumptions and work out a lot of detail using our best understanding of lots of different, you know, areas of expertise. And that's what I attempted to do in the book. So the book, many people say correctly, that it just has more detail than they wanted. They would like something more like a, you know, a science fiction movie or something that has some detail, but, you know, full of plot and et cetera. But I crammed this book full of detail, because I was trying to prove just how much detail I could give you. And that's, I think I succeeded. I think it gave you a lot of detail.

• 6:03Speaker 01

  It's interesting. You describe the book as, you know, having optimism in it and saying that other people maybe see it as more of a state to fear. And I'm curious if you can walk us through that. I mean, to me, just sort of reflexively, the idea of being able to emulate a human brain, like, my mind very quickly goes to the matrix or some sort of simulation, or the idea that the emulation itself becomes a form of a prison for folks. But it doesn't sound like you see it that way or have described it that way. So what's the, I guess, what's the upshot of being able to do this for the average person or for society more broadly?

• 6:39Speaker 00

  Well, I mean, it is a way in which our descendants become much more capable than we are now. And their civilization becomes much larger than ours is now. The members of that civilization have a different world they live in. And it's not a version of our utopias, but that's not very plausible anyway. So I think one of the things people don't like is it's just not their idea of a utopia. And another is that there is a lot of capitalism going on. And for many people, they wanted capitalism to fade away and just, you know, not matter much anymore. Let's get over that. And this is a world full of capitalism. So I think that bugs a lot of people.

• 7:24Speaker 01

  I'll take us on a slight detour, because this is something that tends to come up in a lot of conversations I have about how much capitalism will be found in the future. And that tends to be related to whether the person I'm talking to sees capitalism as like a bug or a feature of human civilization. And I was, I was speaking with a gentleman a few weeks ago who was saying, you know, oh, well, why does money need to exist anyway? And I felt very much the need to jump in and say an awful lot about that. But it sounds like to you, Robin, capitalism and, you know, currency and the economy are very much a feature. And you expect to see, I don't know, as much of them, more of them, rather than less of them? Is that fair?

• 8:06Speaker 00

  So there are many innovations that humans have pioneered in the last 100,000 years. You know, they include, I don't know, cities, trade, law, language, norms, you know, research. These innovations, I think, are going to last. These aren't just temporary things. Humans have pioneered many big innovative concepts that will just be there into the indefinite future. We're going to use language for a long time in the future. We're going to have law for a long time in the future. And yes, we'll have trade for a long time in the future. And I think we're going to have structured, even bureaucratic organizations

• 8:50Speaker 00

  for a long time. We'll have governance. And so, yes, we'll have capitalism. This isn't like some temporary thing. This is one of the fundamental innovations that humans have accumulated. That's just going to be a powerful thing we use in a great many ways. Including money. Money, yes, money is going to be there in the future. Money is a powerful innovation that, you know, we can't do nearly as much as we do now without it. And it's not going to go away. It's, it's, value is quite robust.

• 9:19Speaker 01

  I'm inclined to agree with you. And, you know, there's a word you didn't use there, but that I think underscores the spirit of capitalism and why it's not going to go away, which is competition. And competition is kind of a fundamental force in nature that capitalism-

• 9:35Speaker 00

  Competition is much older than humans. Like, we can, humans can't take credit for competition. Maybe we're using some new forms of it. But yeah, I was trying to list things that are new with humans, but like, competition goes way, way back.

• 9:46Speaker 01

  Right. But to me, competition is the reason that capitalism, or at least one of the reasons capitalism is successful, as successful as it is and why it'll stick around.

• 9:56Speaker 00

  And in many people's utopias, competition goes away. But not yours. No, not, no. I mean, maybe it's not a utopia, realistic opia or something, but yeah. So it was a sense at which I was trying to describe a realistic future. And that meant I needed to sort of apply all the constraints I understand about the way our world is structured and how that might change in this different scenario. And that meant I have to not sort of wishfully, wishfully imagine all sorts of problems go away because we have some future solution. I say, no, no, like, we will be dealing with all these basic problems that I don't think will go away through mechanisms we've been developing so far, and then new ones, hopefully.

• 10:39Speaker 01

  If you don't work in enterprise technology, you can skip ahead or take a quick braincation. But if you do, InfoTech is your secret weapon. InfoTech helps IT teams get projects done faster, better, and at a lower cost. Whether you're rolling out new tech, improving processes, or just looking for cost savings, InfoTech provides unlimited access to practical tools and expert guidance you need to execute at a fraction of the cost of traditional consulting. No matter the project, from AI strategy to cybersecurity, to vendor negotiation, InfoTech has you covered. Check it out at the link below and don't forget to like and subscribe. So I guess maybe just to summarize a few of the themes we've talked about so far,

• 11:20Speaker 01

  it sounds like you are broadly optimistic about the future, or at least what we're building toward, not because it adheres to most people's definition of a utopia, but because it extends our capabilities and is a form of progress of human civilization. Is that fair? Or what would you add to that?

• 11:47Speaker 00

  So we can talk a bit about this more in a few minutes, but I actually see just how much culture and values have changed over human history. And I don't put that much stock in wanting the future to share all my cultural values and all my habits of life. The main thing I want is people to just be alive, to have things they're trying to do, things they get value of, and then collectively, humanity to grow in capacity and wisdom and spatial extent even and temporal extent. I just want us to get big as long as it's full of parts that are similarly as content or happy or find their lives meaningful as most humans ever did. I'm not, I mean, maybe you could find a way that humans could just find their lives

• 12:32Speaker 00

  a million times more meaningful than they do now. That sure would be good, but I'm more just going, let's just have the usual sort of lives we've always had, but have a lot more of them. Have them go a lot more places and have a lot more capacity to do a lot more things.

• 12:48Speaker 01

  I like that and it makes sense to me. And I can see the optimism in that and why, you know, it's, as you said, a reasonable and kind of rational take on the future versus building some sort of utopia that we can't even really seem to define. And always seems like it's this weird out of step jump from where we are right now and the building blocks that have worked for us so far. I am curious, though, because one of the things that you've written about and talked about at length in the past is the notion of the great filter. So talking, I guess, on a broader scale, the fact that we haven't seen a lot of signs of intelligent life. So can you maybe just broadly talk about what the great filter is? If you have any predictions

• 13:32Speaker 01

  or your personal beliefs about what that means for us in the future? And if your thinking has changed on it since you were originally writing about the topic?

• 13:41Speaker 00

  So as we've been talking about, we see in the past a huge increase and expansion of humanity, its scope, its capabilities. And then it's natural to project that trend into the future, to imagine us becoming in a relatively short time cosmologically, because cosmological, you know, the universe is 15 billion years old. So within a million years, which is tiny compared to 15 billion years, we could see our descendants becoming vastly more influential and visible than we have been so far. So within a million years, we could easily colonize the galaxy and rearrange it and just make it substantially different than it is now. And if we were a galaxy away, looking at our galaxy,

• 14:26Speaker 00

  it would look different. We're not sure exactly how it would look different, but it's pretty unlikely it'll look exactly the same as it is now if we decide to just make massive use of it and rearrange everything. So when we look out into the universe, we think, oh, that's what we could be in only a million years. Let's, maybe somebody else got there a million years before us. Would be a tiny fraction of the universe's history. Surely somebody got there before us. So let's go out and look for them. And we look and we don't see them. The universe doesn't seem to contain any of these extensions of our future that we hope we might become. And that's this key question, Fermi's question, where is everybody? And I reframe that question as, well,

• 15:11Speaker 00

  everybody that you might see out there that you don't see, they would have started as some simple, you know, dead matter somewhere in the universe that slowly evolved over eons to become that thing. So the path that leads from simple dead matter, which is everywhere in the universe, to this thing that we don't see, this path must be really unlikely to go down. Otherwise, we'd see it all over the place. So I call the great filter, whatever along this path is stopping things from going down this path, filtering things off of this path. The great filter is whatever is the obstacles. Now, a key question about the filter is how big it is. But it's, say, we see 10 to the 24 planets out there that could have started this thing that we don't see. Then maybe the great filter is at least a factor of 10 to the 24,

• 15:56Speaker 00

  at least up to a certain time point in history. So that's a really big, it could be much bigger, but that's at least how big it is. And we ask, how much of that filter will we pass now? And so say it was 10 to the 22 behind us and only 10 to the 2 ahead of us. We would have gone through almost all the filter, but still 10 to the 2 in front of us means there's only a 1% chance our descendants would become this big visible thing, which is pretty bad news about our future. So it's really important, how much is left in front of us? Even if most of it's behind us, how much is in front? Now, the easiest place to put the filter behind us is at the very earliest stages because we understand those the least and they seem really hard. And that predicts we'll just see no other life anywhere in the universe of any sort. And if we see some bacteria or something else in life,

• 16:41Speaker 00

  that revises, we go, oh, those earliest steps, not quite so hard as we thought. That must mean the filter is either in some of the later steps after that amoeba between there and now or between now and our future. And of course, we're really most worried about the future filters, the filters that could stand in front of us, preventing us become big. Now, it is true that one of the possible future filters could be that we just stop growing and then stay small and then are happy forever. So it's not necessarily apocalypse that prevents us from going out there. But honestly, when you think about all sorts of scenarios, most of the scenarios that would prevent us from getting big also would probably just kill us off. So that's why you want to kind of worry about, why can't we get big?

• 17:26Speaker 00

  So I have, in the past, thought the future filter looks small compared to the past, but it still could be substantial. And I think it's really hard to come up with things that would like kill us all off. So even if you killed off all humans, well, there's other apes and kill off all the land animals and there's dolphins. I mean, like, if you think about over the next billion years, such that our descendants could do stuff, it's really hard to like kill off our potential now so much that something else couldn't rise again. And that means it's kind of hard to imagine a really tough filter in front of us. But when I think about it more, I go, oh, well, the most likely filter would be

• 18:10Speaker 00

  some way in which we form a stable world system that then blocks further expansion.

• 18:17Speaker 00

  And I think that would have to be my guess about the future, biggest likely future filter. Like, our descendants grow to be more capable. They form some sort of world government and world culture. They form some norms about what kinds of things are allowed or not. And then they stay functional enough to enforce these rules.

• 18:34Speaker 00

  And those rules would then prevent expansion. And that's something I worry about. And we could talk more about like why that might happen and then how it might play out. But I do worry, that's my biggest worry about the future, is that we will create a world system of sorts that's competent enough to enforce some strong world rules. And, you know, which means it never gets that much bigger, but it also doesn't get that much smaller.

• 19:06Speaker 01

  It's fascinating to me that you were framing that as kind of your biggest concern about the future. Because, you know, when I think about the great filter, and maybe this is just a product of the time at which we're talking, you know, one of my concerns and one of the narratives you hear these days is that, you know, it's entirely plausible that AI will be the great filter for us, that we're going to create accidentally or deliberately.

• 19:32Speaker 00

  That is, it's about what would make our galaxy really visible from a long away. If AIs take over and even kill all the humans, as long as they go on to make a really visible galaxy, that's not the great filter. The great filter would have to be some process that prevents them from getting big. Whether they kill off humans is pretty irrelevant to whether they get big again later. So it could be pretty bad if AI were to kill off humans. I'll admit that. But that's just still not plausibly part of the great filter. When you're trying to explain why the universe is empty, the fact that there's a place out there where some people there killed off the other people there isn't really part of why can't we see them?

• 20:11Speaker 01

  Right, so for the AI scenario to become plausible, AI would have to not only basically kill every living thing, it would then have to destroy itself as well. And that scenario seems...

• 20:21Speaker 00

  Or permanently limit itself to prevent itself from growing. And then that's the thing to wonder, how could that happen?

• 20:28Speaker 01

  Right. Well, and I guess the optimistic case for that is when we look at the possible scenarios of the great filter, it sounds like, and, you know, I'm a convert to that, that that scenario doesn't sound particularly plausible.

• 20:44Speaker 00

  Well, again, over the last decade, I think I have come to see this sort of lockdown prevention of growth scenario as more plausible. And so that's why I'm more worried about.

• 20:56Speaker 01

  Right. Well, and you you used the time scale earlier of a million years, which, which, you know, as you said, cosmologically insignificant, but for you and I talking here, the scale of a human life, you know, we're, we're talking of a very, very large, you know, time scale that's, you know, quite a multiple over the, you know, the entire lifespan of human civilization so far. And so it sounds like you're accounting for even if there's, you know, massive global devastation.

• 21:24Speaker 00

  Right. Yeah. So it's really hard to imagine like killing off 99% of us and that delaying these things more than a million years, which again is a tiny amount of time. So if you're gonna kill off with potential, you have to just kill off lots of stuff on earth, maybe life on earth, or you're gonna have to let it freeze into something that prevents its growth. That's the second scenario that I find more plausible to worry about.

• 21:53Speaker 01

  Right. Now, you know, we could keep the timeframe at a million years, but one of the other things that's been fascinating to me lately is with this kind of rapid acceleration in AI capabilities, or at very least, the degree to which the average person is able to experience some of these new AI and generative AI capabilities. There's been a lot more speculation about what is the speed at which technology improves from here. And the flip side of this is, you know, you have people saying that we're gonna get a million years versus of progress in the next decade or so. And I'm curious if you pay any heed to, you know, those kind of thoughts. And maybe, maybe on the flip

• 22:36Speaker 01

  side of that, asking you, how do you see the next kind of 10 to 20 years playing out technologically?

• 22:45Speaker 00

  So I'm a futurist, and I like to think about the long run future. I'm not that interested in the latest press releases or whatever that many people see as futurism. I want to know about the long run. And on reflection, humanity has this enormous future future potential. Again, we see this empty universe that's so vast and we could go out and fill it up and do stuff with it. And I'm very interested in our achieving that potential, our descendants, and not getting sidetracked or destroyed. Now, given how fast change has been lately, it seems like I shouldn't be that greedy for faster change. Faster change can be fine, but I more want to think about obstacles that sit, say, on doublings.

• 23:26Speaker 00

  So if I think about doubling the capacity of humanity, you know, in the last century, that happened every 20 years, but 1,000 years ago, it happened every 1,000 years. And then, you know, 20,000 years ago, it happened every quarter million years. So the rate of doubling has been greatly increased, which is great. And then in the future, the rate of doubling might go even faster. I talk about an age of M, maybe it would double every month or even faster, the M economy. But on reflection, I'm not that eager for faster change. I'm more eager for reliable change. I more want to think about what obstacles would show up at every doubling or within every doubling scale and how we're going to avoid those.

• 24:10Speaker 00

  And that's more what I care about thinking about, less of speed. So another way to think about speed is, many people will say, isn't it a problem of technology changes faster than society can adapt? And I'd say, the rate at which society can adapt has long been a limiting factor in the rate at which technology could change. That is, society as a whole needs to adapt and to things together. And that's actually a limit on how fast everything can grow. We've often had some parts of our society grows much faster than everything else, but, you know, it reaches a limit and slows down because other people, other parts of society just don't keep up. They don't speed up to match it. Now, you could have a part of society that starts to grow faster, and then there's another part that's slower,

• 24:55Speaker 00

  and that other part might just get dropped. So for example, with the industrial revolution, there were many subsistence farmers out there, whose farming practices just did not speed up with the industrial revolution. But that didn't slow the industrial revolution. The industry just said, well, forget about you. We can make food other ways. And they just went on to have faster rates of change elsewhere. So that's maybe what you should more worry about with speeds of rates of change. You shouldn't worry that the world will change faster than it can handle. What you should worry is maybe the world will change faster than you can handle, and you will just be shut to the side because you're in the way if you, they let you be on the critical path. So that's how I might think about AI. I might more worried about, okay, say AI speeds up things, what parts of our world will then

• 25:40Speaker 00

  just get dropped from the part of the world that's going faster because it just can't keep up? And do you want, are you worried that you might be part of that world that gets dropped because it can't keep up? And then, you know, how, what part of this world, what, what will be the world that's integrated enough that it all needs to grow together? And then think about what, you know, what could happen in that world?

• 26:06Speaker 01

  That, that framing makes a lot of sense to me. And I think it is, like, I think you hit on something that is a really deep fear for a lot of people. And that AI actually in a big way kind of accelerates that inequality. And that, you know, the people being dropped or the, or the parts of society being dropped are just gonna be, it's gonna be more and more of them. And AI may be good, you know, at its most extreme scenario, AI may be really good for like six guys, you know, in the US and maybe two in China. And for everybody else, you're, you're in, you're in big trouble. And that with the acceleration of technological change, we're narrowing the band of people who are actually equipped to take advantage

• 26:51Speaker 01

  of that and come out ahead. And we could end up, I mean,

• 26:56Speaker 00

  Six guys seems to me really small. So the last two, and I mean, the last three analogous things that happened in history was, first of all, one set of human species compared to other proto-human species took off and compared to the other humans. They contributed some DNA to the mainline, but much less. Then with farming, you know, places started to adopt farming and then local hunter gatherers were absorbed into that or displaced. And then with industry, Northern Europe started to grow rapidly and then pulled the rest of the world together. Now, in these last two transitions at least, you know, large chunks of the world came along. It wasn't six people. It was very large coalitions

• 27:40Speaker 00

  of parts of the world that managed to grow together. All of Northern Europe, and then all of Europe, and then all of the West, and then the whole world. You know, we've pulled much of the world along with us now by the industrial revolution. Farming revolution, again, it was more like a 50/50 thing. When farmers came into some area, it was more 50% farmers coming in and 50% recruiting the locals and turning them into farmers. And so that's a pretty large fraction of the previous world being recruited into the new farming world. With humans, it might have been, I don't know, maybe 10% of the proto-humans who took off as Homo sapiens and the rest got set aside. So it's a smaller percentage, but still, it's 10%. It's not six people out of 8 billion. So the six people

• 28:23Speaker 00

  out of 8 billion seems to me just crazy small as a plausible fraction.

• 28:28Speaker 01

  That, that makes sense to me. And I know we're, you know, we're speaking somewhat flippantly about, you know, that these kind of winners and losers on a historical scale. So I'm curious, if we do believe though, that these, you know, these periods of, of, you know, revolution in the, I guess, the broad sense of the word, you know, if these periods create winners and losers, looking down the barrel at this one, what can people or fragments of society do, in your view, to make sure that they're more adaptive or going to find themselves on the right side of history?

• 29:04Speaker 00

  So there's two main approaches that have worked for a long time. One is to, you know, try to join the winners. If you so, if you just insist that those people have a new thing, but I don't like that, I'm a new thing the old ways. And later on, you lose and you go, that's not fair. How come they didn't include me? Well, you know, you had a choice to be included at some point. You should be eagerly trying to include yourself in various winning scenarios and winning groups. You know, if you've got, I don't know, a clan of people, make sure different parts of your clan go off, join different groups. And then, you know, you're giving a portfolio insurance against the different possible winners, right? So I would try to, you know, be represented in different professions,

• 29:44Speaker 00

  different industries, different nations. Just, you know, be, spread yourself out for who might be winners and try to be part of them and be flexible so that you could, you know, if there's a new group doing something a new way, well, some of you should like, try the new way. That's one approach, and another approach is just property. So, um, you know, you know, with the industrial revolution, the landowners could have insured themselves against this huge transition by just buying into the new factories. And alas, they didn't. They just insisted on holding their rural land. And then, I mean, they just lost out on a lot of the industrial revolution because they didn't diversify over the new assets that were growing.

• 30:27Speaker 00

  So if you just hold some property and hold a diversified portfolio of assets, then you can be part of wherever the assets blow, grow up. Like the standard market asset story index funds is you should just hold all assets in proportion to how much other people hold them. And so whatever blows up, you've got to share it. And I do think what we want is for, say, a transition to AI to preserve property rights. Uh, that is, um, so today we have this option. We could just kill all the retirees and take their stuff. But we don't. Why don't we kill all the retirees and take their stuff? Well, some of us care about particular retirees, but I don't think really that explains all of our collective thing because most of us don't care about most of the retirees.

• 31:11Speaker 00

  I'd say a better explanation is we share some structures that keep the peace in our world. We share law, we share governance, we share finance. And if we were to overthrow some of these structures in order to grab, you know, take their retiree stuff. As soon as that happened, other people would wonder, am I next? We, we, we threw away property rights and we, we grabbed something from somebody. And that means that's a precedent. We could go, we could repeat that several times. And now, how confident are you? You'll be left standing at the end of this musical chairs where lots of people got their stuff grabbed. So I think that's a reason just not to start this grabbing people's stuff, which is why maybe we don't take retiree stuff and kill them. Because, you know, what, what are they doing for us, right?

• 31:54Speaker 00

  And so if a world of AI had a whole separate world of finance and law and governance, they had, they lived in separate cities on separate islands or something. Well, now there'd be more of a problem where they'd wonder, what, what are these humans doing for us? You know, let's just like get rid of them. But if we shared structures with them, shared finance, law, governance, and then among themselves, if they said, let's take all the human stuff, some of the AIs would have to wonder, well, then who's next? Who, who else you taking from after you take from the humans? And they might just, let's not go there. And that's a way in which shared social institutions can help keep the peace, which I think they in fact do. Uh, that's one of the reasons we don't, in fact, typically find 5% of the country and say, let's kill them and take their stuff, right?

• 32:39Speaker 00

  You know, what's, why doesn't the other 95% of the country repeatedly pick a 5%, kill them and take their stuff? Well, I think it's because you wonder what happens next. And some of that, we can use that same trick to keep the peace with whatever descendants we have. If they are integrated into our institutions, then they should be afraid to, you know, not keep the peace because they might be next.

• 33:08Speaker 01

  It, it, it's interesting, and I, I, I see what you did with like, not just the anthropomorphizing, but, but like, kind of framing AI as our descendants and, and, you know, kind of on this level of continuity with our, you know, ancestors and our, you know, forebearers and everything like that.

• 33:25Speaker 00

  I, I do think of AIs as our descendants. And I think this is one of the main ways I disagree with many people in the AI risk world. They've often framed AI as this alien armada coming from another star. If you saw it coming a century away, wouldn't you want to get ready for it? They're seeing it as a coexisting rival. And evolution has long created us a wariness of coexisting rivals. Even your cousins can be, you and your brothers can be pretty suspicious of your cousins, even if they're really similar to you, because who knows what your cousins could be up to. But evolution has also given us an indulgence of descendants. Even when your descendants are pretty different from you. Those are two robust things evolution gives you. Suspicion of pretty similar coexisting rivals and indulgence of pretty different descendants.

• 34:10Speaker 00

  And I'd say AI just literally is the descendants. It's not the coexisting rivals, but they don't exist yet. They are not at some, they are not capable of being rivals yet. If they ever are that capable, it'll be because they're your descendants. We will make them and they will then arise from us.

• 34:28Speaker 01

  It's, it's a really interesting framing device. And I think, you know, you're forcing me to reckon with the fact that a lot of this, whether it's explicit or implicit language about, you know, kind of alien when it comes to AI, is people talk about it like it's an invasion or we're being displaced, or there's some sort of discontinuity there. And if you see AI as an extension of us or something that we built exclusively, right? Like everything AI knows, it knows because of the way we constructed it. It, I, I don't know, I can, I can like hear an optimism in your voice that's absent from a lot of people I speak with.

• 35:07Speaker 00

  So when you look back at your ancestors, almost everybody says, yes, they're like me. But when you look forward to your descendants, people are often not so sure they want to embrace them. Your descendants could just be weird and betray your ideals in many ways. And people are cautious and uncommittal about which future creatures they're going to allow to see as their descendants. Um, I think this is just a natural sort of hesitance. But I think you have to then realize our future AI descendants, even if you're not so sure that they're human, they will see themselves as human. And you don't realize just how far you've come from your ancestors.

• 35:49Speaker 00

  I, I think if your ancestors from thousands of years ago were to see you and meet you, they really would be impressed with a lot of your stuff. But I'm not sure they would really be willing to embrace you as their descendants because you've betrayed many things they thought were essential to what they hoped their descendants would do. You've betrayed their gods. You be, you don't worship them as ancestors. You've thrown away many of their cherished traditions. You have betrayed what they hoped you would be in many important ways. They wouldn't be so sure whether they wanted to embrace you, but you're pretty sure you embraced them as your ancestors. So just project that forward. That means these strange future creatures who show up,

• 36:32Speaker 00

  they will just be more inclined to see themselves as a continuity of you.

• 36:41Speaker 01

  Yeah, it's, it's funny, sorry, I'm just processing that. And, and I, I completely agree. I mean, I don't, I don't think you have to go back a thousand years. I think you can probably go back to like three generations or, you know, people you may have even met in your life and think about, boy, what would they think about the way that we're living now? And what we've done to, to the world that they had? And, you know, is it, is that, is it actually better? But the, you know, the, the piece that someone must push back on is that, to me, there is a distinction between, you know, blood of your blood versus creating and, and, you know, we, we could have a conversation about what exactly AI means. And are we talking about what we've got right now with large language models

• 37:22Speaker 01

  or more abstractly, you know, emulated minds? Is there something that has to be true about the type of AI or the flavor of AI that we're creating for that to be the case? Or is that just at some point where we need to be in terms of the way that we, you know, kind of create this technology in the future?

• 37:45Speaker 00

  So I'm intellectually more of a theorist. I'm less of an empiricist, although I try to learn lots of empirics and, and, you know, rely on it and believe it. But I am more naturally a theorist who tries to collect theoretical frameworks and apply them. And one of our most powerful theoretical frameworks is Darwinian selection. That Darwinian selection is a pretty abstract theory, actually. And we often focus on its version applied to DNA today, but Darwin didn't know about DNA, actually. He just postulated some way in which we passed on things about ourselves to our descendants without necessarily knowing how.

• 38:28Speaker 00

  And Darwinian natural selection is a abstract theory, but it has pretty solid and predictable, you know, inferences and conclusions, it says. So it has what it needs for a descendant in Darwinian terms. A descendant in Darwinian terms is something in the future that shares features with you, you know, arose from you. And causally, you were the cause of it having many of those features. And there's a correlation between your features and it caused by the fact that you continue to exist and were able to reproduce. And then it continues to exist. That's what a descendant means. There's no, it's not a metaphor.

• 39:09Speaker 00

  It's a technical term in a technical theory of Darwinian evolution. And according to the technical term of that concept, whatever creatures you create in the future that share things with you, that you caused to share features with you are your descendants. That's just what the theory says. Now, yes, there are going to come up some ways in which we have a new mechanism for passing on to descendants. That is, so far, we've had a limited range of ways we pass things on to descendants. And we're now about to introduce a new one. But humanity actually pioneered one, a whole new approach compared to all the other DNA animals. We pioneered cultural evolution. That's humanity's superpower and has been for a million years.

• 39:52Speaker 00

  We pass things on to our descendants, not just through DNA, not even mainly through DNA. Now, we pass it on through copying each other's behavior. And that's also a kind of Darwinian evolution. And it's the kind that humanity has pioneered that's made us so much more powerful than all the other animals. So I don't think you should be too squeamish about a new way to pass on features to descendants. That's what humanity has been all about for a million years is we have been pioneering this new ways to pass features on to descendants.

• 40:22Speaker 01

  I'm gonna pause for a second on the phrase "cultural evolution", and I'm gonna, in my mind, I'm capitalizing, capital C, capital E there. And the reason I'm pausing on it is it's bringing me back to what you were saying earlier about, you know, human innovations over the past few thousand years. And you talked about, you know, law, and you talked about language, and you talked about governance. And to me, like, I feel like I could write "cultural evolution" in the middle, and all of these are part of the same sort of ecosystem that we've created.

• 40:52Speaker 00

  Well, cultural evolution is what allowed those things to appear so rapidly compared to with other animals. Yes, cultural evolution allowed us to invent language, to invent law, to invent trade, to invent capitalism, property. All of those are consequences of cultural evolution. And cultural evolution is our shining achievement. And that's why I'm the most worried about the Age of M and other scenarios in the future being put off, because I think we're failing now in cultural evolution, that I think we broke the superpower. That's the way to put it. This has been our superpower for a long time, and we basically have been breaking it in the last century or two. And as a result, our future looks bad. Looks like our civilization will peak and then decline,

• 41:37Speaker 00

  and to be replaced by some others that are really quite different, and will reject many of the things we hold dear. And it could be quite a while until the Age of M shows up, or we manage to do interstellar colonization. I do think that will still eventually happen, but we may need to figure out how to fix this key problem we have before that can happen.

• 41:58Speaker 01

  Can you expand on that? What have we broken? How have we broken it? And I guess sort of by understanding the what and the why, we can have some sort of road, I hope, to, you know, what you need to rebuild.

• 42:14Speaker 00

  But the bottom line will be, it's just really hard to figure out fixes. We not only broke something, we broke it really bad, and no simple masking tape or glue fix is gonna work here. We need actually a pretty deep fix, which is hard to find. Okay, so let's start at the basics. If you think about DNA evolution before humans, it has two key levels. There's evolution within a species and evolution of species. A species is a set of creatures who all shame, share some features that allow them to mate with each other and to, you know, share a structure. And that means the evolution of those features that a species share can't be done within a species. Evolution of that sort requires multiple species that exist

• 42:56Speaker 00

  and try out different species designs in order to have evolution of species. So with DNA, we have evolution in species and of species. And we can have you and ask which matters more. You see, when you have a large habitats, like big chunks of the ocean, larger species allow better evolution in a species, but worse evolution of species. Whereas small fragmented habitats, like rivers, rainforests, coral reefs, they have small species, they have worse evolution in a species, but ever better evolution of the species. So we can just ask, where did more life come from today? The integrated habitats or the fragmented habitats?

• 43:38Speaker 00

  As a way to find out which mattered more, evolution in species or of. And it turns out, the fragmented habitats produced more life today, which means evolution of species mattered more than evolution in species. Okay, so that's with DNA. Now, humans pioneered cultural evolution. And cultural evolution is really just literally a new way to pass on features to descendants. So you still have the same units, i.e. organisms, and then they have other units that are descendants. And one way to pass on features to a descendant is through DNA. But now we add the way to pass on features to descendants, is why they copy your behavior.

• 44:16Speaker 00

  And so now, you have more parents and more children under cultural evolution than you do under DNA evolution. But otherwise, it's the same process. And so you have cultures which share behaviors, because they have the same status markers and the same norms. And to have evolution of cultures, you need multiple cultures. But if there are things you can change within a culture, then you get evolution within the culture, and that's definite evolution of the culture. So things like technology and business practices, different kind of musical zonras, et cetera, those are all things that happen within a culture. And in the past, when the world was fragmented into lots of little peasant villages,

• 44:58Speaker 00

  evolution within cultures was slow, because we had so many little cultures. And then recently, we have switched to having a few really big cultures, which has allowed evolution within cultures to go much better. Much better technological evolution, business practice evolution, you know, cultural evolution of, say, music, et cetera. All of that is going much faster now, because instead of lots of little cultures, we have a few big ones. But now that threatens evolution of cultures going worse.

• 45:32Speaker 00

  So here's another way to think about it. You could think of natural selection as something happening in a high-dimensional space. There's a set of points, whatever the things that are evolving, points in the space. There's an adaptive region in the space, i.e. the good place where points wanna be. And now, points in the region, in the adaptive region, they grow and split, they do well. Points out of the region, they shrink and die. And now the question is, you have a cloud of points that needs to track this region. Now, if the region doesn't move, the points don't move, everything's great. You just stay right there in the region. But we've got four parameters that can make this harder. One parameter is, how many points are there?

• 46:16Speaker 00

  Another is, how big is this effect that points in the side grow and points outside shrink? A third is, how fast does this region just move, independent of everything else? And a fourth is, how much do these points just wiggle, independent of everything else? So these are four key parameters. And you can see that in one set of these parameters, everything works great. The region isn't moving much, the points are all near it, there's lots of them. They grow, they grow great inside, they don't wiggle very much. This is gonna work fine, right? The cloud's gonna track the adaptive region. Now go to the opposite extreme, very few points. They don't have much of a selection effect in the region, the region moves fast, the points wiggle a lot. This is not gonna work, the clouds won't, the cloud just won't track the adaptive region.

• 47:02Speaker 00

  It'll just get displaced from it, and then they'll all shrink and die. And that's bad. So that's an abstract way to think about cultural evolution. So now let's apply that to cultures on Earth. Three centuries ago, the Earth had hundreds of thousands of peasant cultures, each of which had a thousand or so people. And these cultures were very poor. They often suffered famines and pandemics and wars. So if they became less adaptive for some reason, they could just easily get killed off, and their people would be absorbed by neighboring cultures. So there was strong selection pressures, large number, hundreds of thousands. The world doubled roughly every thousand years. So that was the rate at which the adaptive region changed that the cloud needed to track.

• 47:48Speaker 00

  And then these cultures were very conservative, trying not to change very much. So then, they didn't wiggle very much. Now, fast forward over the last three centuries. First, we took peasant villages and made nations out of them. We formed nation-states, national cultures, so a great reduction of variety, down to, say, a few hundred or a hundred. Then, in the last century, we merged those into a global monoculture of elites, at least.

• 48:13Speaker 00

  We're all now rich and peaceful and healthy, so these few remaining cultures are not being killed off by not being in the adaptive region very much.

• 48:23Speaker 00

  Our world changes by doubling every 20 years, instead of every thousand, so a factor of 50 increase in the rate at which the adaptive region is moving. And now, instead of being conservative and trying not to change, we made a virtue out of cultural activism, where people are eager to be the cause of cultural change. And people are eagerly seeking out and trying to cause cultural change.

• 48:45Speaker 00

  So all four of these parameters have gone badly, in terms of their ability to support cultural evolution. So the obvious prediction is, we're just drifting into maladaptation. And the clearest prediction, the clearest example of that, I think, is falling fertility. That is, in times of peace, plenty, and, you know, health, to have the population shrinking is just a pretty crazy maladaptive thing for biological species to do. And then we have lots of other trends that seem maladaptive, but it's harder to be sure. But this is the reason to believe we are becoming maladaptive. And this is really hard to fix. Its most fundamental underlying cause is the fact that, as we unleash the power of technology and capitalism,

• 49:28Speaker 00

  we rapidly, we caused rapid technical and social change, which made us all more able to and eager to do lots of talk, travel, and trade around the world. So we got rich, and we integrated into a single shared culture. Locally, we love that. But that's killed this underlying process of cultural evolution, because now we've cut away the selection pressures and the variety, and we've caused this huge rate of change that's hard to keep up with. And we furthermore made cultural activism a celebrated activity. So this is the reason to believe cultural evolution's gone wrong. And now if you start to think about fixes, you realize it's really damn hard.

• 50:11Speaker 00

  But I've talked for a while, so let me let you ask questions and clarify here.

• 50:14Speaker 01

  Yeah, thank you for that. And I think we may have some viewers and listeners who, if they were trying to listen to this at double or one and a half time speed, they may need to go back and, you know, slow it down to tease out every nuance of there. Because, you know, thank you for sharing that. There was an awful lot there to to think about. I'm just to kind of react to that live. To me, it's interesting because there's how culture is changing as sort of one dimension. I also couldn't help but, and I know you have a background in economics as well, but thinking about how capital is moving around. And, you know, the fact that, you know, I can, I can easily in some of these stories and metaphors, think about capital, like I can view this map geographically

• 50:57Speaker 01

  and look at the capital density in specific cities or moving out of rural regions. And then you talked about fertility as well. And there was almost like a therefore statement of like, and fertility is dropping, something is going wrong. And that one, that one is really interesting to me, and it is something that I worry about as well.

• 51:20Speaker 01

  To me, I don't know if it's, it seems like with, as you said, there was a time in our history, for most of our history, where if you had abundance and peace, you would see an increase in fertility and in population growth, I think. I can't, I don't have the data to back me up there. But in my view, like, life has just gotten too damn comfortable. That we've got a level of technology, we've got a level of wealth and absence of scarcity. And I guess in some ways, technologically, we've been able to, I guess scientifically, create a world where we can,

• 52:00Speaker 01

  I guess, have, you know, sex without procreation. We don't need offspring necessarily for, you know, the labor of it. And so we have this world where people can just sort of opt out of,

• 52:12Speaker 01

  like, it almost becomes, it almost just becomes sort of like a utility maximizing thing. We're like, oh, I could have children or I could have more children, but that seems like I would be less happy than I am right now. And I can just very easily see that being like a civilizational doom loop. It, I don't know, is any of that on your mind?

• 52:34Speaker 00

  So when you look at yourself and your life and the choices you make, you make those choices with respect to your concept of your utility, I mean, what makes you happy, what you enjoy, what you want. And also cultural constraints, what are the norms, what will we be punished for? What will you be praised for? And those features of you were caused by evolution. That is, it was previously evolution's job to give you those things so that your culture would thrive and grow. Your job, you just took those as given and you said, what do I want? And you tried to get it. But it was this larger process underlying you that you don't notice that gave you those things.

• 53:15Speaker 00

  And its job was to induce you to do the things that it wanted you to do. So now, when the things you want to do don't actually promote your culture, that's a failure of this culture to give you the right motives and opportunities to do the things it needs to do. So that means this stuff you got, your values, your loves, your things you hold sacred, the, you know, the morals that you feel tied to, all of those things, you just often feel they're just natural, they're just part of the universe. But they were given to you, they were created and given to you by this cultural evolution process. And in the past, if that led to bad outcomes,

• 53:57Speaker 00

  then cultural selection would then change the culture to no longer make it lead to bad outcomes. So it's not so surprising that as the world changes, our motives will now lead to different behaviors. What you should notice is that usually that would get fixed. Usually if your culture decided it didn't care about having kids or didn't care about working or whatever, the next war, your culture would get conquered. Or, you know, next competition, your fishers wouldn't catch as many fish as the other people, or whatever it was, you would just lose out and your new style would just go away. So that's how it's been for a million years, that humanity, each of us has been following our own cultural inclinations.

• 54:38Speaker 00

  But our culture gave us inclinations roughly for the purpose of getting our cultures to win. So if your culture doesn't win, it's the fault of cultural evolution to having given you the wrong directions. And that's what's happening at the moment, is that this isn't getting fixed. The process in the past that would fix this has been turned off. And that's the underlying cause of this going wrong.

• 55:06Speaker 01

  Well, and maybe to tie a few of our discussions together, it almost see like, I can start to see the road to the future you're concerned about from the viewpoint of the great filter. That, you know, and again, to put this in extremely simple terms, we've just gotten too damn cozy. And a lot of the competitive pressures in the past that would lead to, you know, survival of the fittest, expansion of the fittest, I guess, have eroded. And we've just reached a point as a civilization where like, yeah, things seem good. So we've, we've disabled the mechanisms by which, you know, the weak are kind of left behind.

• 55:47Speaker 01

  And we just have this grand, things are okay, without them getting substantially better. Is that a fair framing? Or are there some pieces there that are fundamentally wrong?

• 55:56Speaker 00

  I mean, it's, it's part of it, there are other parts. But let's just first walk through the implications. If nothing stops this, then what happens is our civilization reaches a peak, say, population starts to decline. And then innovation rates also decline as population declines, as the scale of the economy declines. And then, in addition, other cultural trends are going wrong. So our civilization just falls, comes into a declining period. And now, if that was the only thing that would happen, humanity would eventually go extinct. But plausibly, that's not what happens. In fact, we get replaced by other civilizations that are rising instead. So at the moment, the most visible candidates are, say, the Amish or Heredim, who have been doubling every 20 years for a century

• 56:40Speaker 00

  and are very insular in that they are very strongly resisting being influenced by our wider culture. So they are just going their own culture way. And that's what's allowing them to double every 20 years. And this is how the Christians took over the Roman Empire. They doubled every 20 years for three centuries. And that's what it took to go from this tiny fraction to taking over. And at the end, when they finally won, they threw away and rejected many things that the Roman Empire had held dear. And also many things had just fallen away out of the fall of scale economies and the fall of the scale of the entire empire and its capacities. So that's what's on risk for us. That is, in a few centuries, we might get displaced by Amish and Heredim who rise, who then reject much of what we hold dear.

• 57:24Speaker 00

  They might reject democracy, gender equality, open inquiry. They might, because those are actually pretty unusual things in human history.

• 57:34Speaker 00

  And they would just go back to more typical human history styles and then grow again. But then they might, as they get larger and more integrated and richer, they could suffer the same problem again. We might have several cycles of rise and fall until someone figures out how to fix this. So that's why we might as well start thinking now about how to fix it, even though we're not sure at all it could be done soon. But that's the underlying problem.

• 58:00Speaker 01

  It sounds like that's a bad thing for our culture, you know, you and me. And I guess, I don't know, I'm using like a capital O, our there. But it does still seem like, as you said, there's these pockets where somebody could kind of shake loose and whether it's, you know, high fertility rate, you know, contrasting values, aggression, there's some sort of basket of factors that could break free and just jar us into being,

• 58:29Speaker 01

  you know, I guess, kind of second-class citizens or disrupting us in some powerful way. Do you see that happening?

• 58:36Speaker 00

  Well, the, I mean, it would be worse if human extinction was likely here than everything that's anything like us would go away. Fortunately, I think that's less likely. Because, look, if we were eager to like crush the Amish and Heredim right now, we could. They're a tiny fraction. We got it in our head that they're a threat and we want to crush them, we could just do it. But that would be much worse, because now, you know, there's less likely to be something to rise again to replace us, right? But I, again, it's a question of how much you value things about our culture. You know, I think most people do. And so I think, for example, if we went through 10 movies or novels in your life that had a really emotional, you know, scene in them at the end,

• 59:17Speaker 00

  you'll see that that emotional scene was a way in which that novel reaffirmed values of your culture. And you might not realize it, but you really love your culture and its values, and you can see it in those scenes.

• 59:31Speaker 00

  But again, I would point to things like democracy, gender equality, open inquiry. If you imagine worlds without those things, and we imagine some, you know, events in a story where somebody threw those things away and replaced them with something else, you'd be unhappy with that. You'd want a hero to reverse that, because that would be a pretty sad thing to happen. And so therefore, you should just be sad if that happens to our descendants. That the key things that makes us distinctive as a civilization that we value gets thrown away, you should not want that to happen. Even if, you know, thousands of years later, they get rediscovered because they have fundamental value, there could be a long, painful absence in between. You should want the future to, you know, inherit from you.

• 1:00:18Speaker 01

  I couldn't help but visualize like Dark Ages and Enlightenment and that sort of trajectory of human history. And Robin, I'm not expecting a magic bullet here, but given the complexity of this problem and how many kind of interrelated challenges there are, are there any kind of fundamental pieces of wisdom that are worth sharing in terms of how we?

• 1:00:45Speaker 00

  There's a number of them here. But first, I just, since we were talking about AI before, let's just ask, what happens if we get really advanced AI? How does that interact with this problem? And so AI plausibly solves the fertility problem, at least if we allow AIs to multiply as rapidly as they could physically. If we regulated them to prevent that, then maybe no. But if we allowed AIs to compete and supply as many AIs as the market would support, then we would have a population explosion of AIs. So AI would solve that fertility problem. But remember, the fertility problem was a symptom of an underlying problem. It wasn't the problem itself.

• 1:01:30Speaker 00

  And the underlying problem is likely to be there with AIs who share a single world AI culture, i.e. they inherit our world culture and then they all share that culture. We again have too little variety. And maybe they also are causing even more rapid change that they find it harder for their culture to keep up with. And so that's a reason why AI might also suffer this problem. Now, here's a different way of framing the problem that maybe suggests AI will do better. We could see the problem as basically we invented capitalism, which then drove technology and science and other sorts of very rapid changes. And there were other

• 1:02:14Speaker 00

  parts of society we did not let capitalism run. We did not let capitalism run governance. We didn't let it run fertility. We didn't run it some cultural things. And it's those parts that are now at risk of being maladaptive. But the parts of culture that we let capitalism run, those are actually pretty adaptive because those have been subject to the fierce capitalist competitive pressures to adapt or die. And so we can think of the fundamental model problem is, we invented capitalism and then we let it loose on some big areas of life, but we prevented it from applying to other areas of life. And that mismatches the underlying problem.

• 1:02:54Speaker 00

  Okay, now, go ahead.

• 1:02:56Speaker 01

  Go ahead. I was just gonna quickly, I was gonna quickly challenge that because I was gonna ask, did we prevent it from running those areas? Or did we pretend to prevent it from running those areas? And what I was gonna say is, when you think about governance, it's hard to talk about government or how things get done without talking about the influence of money. And similarly, it's hard to talk about fertility without wealth concentration impacting mate selection in some way.

• 1:03:25Speaker 00

  So capitalism is a big part of our world, and then it affects many parts of the world. So there's a difference between a part of our world being affected by capitalism, letting capitalism run that part of the world. If capitalism ran that part of the world, it would then run it for max profit, and then it would be less likely to let that thing just decay or fall apart. So for example, if capitalists could just raise kids in orphanages and sell them for a million dollars when they were 18, capitalists would just make lots of kids. They would find a way to get kids and raise them. So the problem that we don't have enough adults is in part because capitalism doesn't run that area of life. Similarly, governments, capitalism often influenced government, but we haven't really allowed for-profit governments

• 1:04:11Speaker 00

  in the way we do for many small organizations that run our lives. Most of these small organizations that run our lives are in fact for-profit organizations. But cities and nations, we don't let those just be directly for-profit organizations. And that means even though capitalists can go lobby them and influence them and even distort them, they don't just get to own them and run them. A capitalist-owned city or nation would not let it fall apart. It would not let it decay and become dysfunctional. But capitalism influence and corruption of something else that it doesn't run can help distort that thing and make it not work as well as it otherwise would.

• 1:04:57Speaker 00

  So the thing I was gonna say about AI is, so far, we've let capitalism run AIs.

• 1:05:05Speaker 00

  Now, apparently, we're recently maybe letting government get more involved in controlling AIs. But the extent we just let capitalism run AIs, now maybe capitalism can then run the governance and fertility and the other parts of the AI world. And now, we might solve the cultural drift problem, as I've called it, by letting AIs just become full capitalists in their world. Because we haven't let our world be full capitalists. We haven't let human life be fully run by capitalists. We've let some parts of it, but not others. But for AIs, we might just let it fully be run by capitalists. So the two threats there are at opposite extremes. One extreme is, we decide we love the AIs so much that we can't stand to hear them,

• 1:05:48Speaker 00

  see them being oppressed by those damn capitalists, and we go and save them from capitalism by invoking regulation that limits the ability of capitalists to run those worlds. So we basically pass on to the AIs the ways that we have prevented capitalism from running our world, because we love the AI so much. That's one risk. The opposite risk is that we fear the AI so much that if they were to just run their own world, we figure they would take over and destroy us all. And so then we have to regulate and limit their ability to generate and follow their own different cultures, which then would also prevent cultural evolution of the AIs. So those are the two extreme risks of treating the AIs by loving them too much

• 1:06:29Speaker 00

  or hating them too much. And it's in this intermediate realm where we're just neglecting them, basically, where we might let capitalism run their lives, in which case that would be a potential solution.

• 1:06:42Speaker 01

  I have so many thoughts and questions I want to share in that space. Just before I do, though, I want to take one step back, and I want to frame that up a little bit in the context of the AI debate you had with Eliezer Yudkowsky, close to 20 years ago now, where there was at least a flavor of this that came up in those debates, which was, with the advancement of AI, how much of it would be sort of hyper-concentrated and there's a singular breakthrough versus is it actually dispersed, more competitive and more capitalistic? And I'm curious, maybe first of all, if you could serve up your macro level, just kind of playing field of what the positions

• 1:07:25Speaker 01

  were in the debates. And obviously, I'm not asking someone completely unbiased, but how do you see the, how do you see the reality of actually firming up since then?

• 1:07:37Speaker 00

  So on the AI risk side, there's this Mott and Bailey, really. There's a very mild plausible claim, and then there's a much more extreme implausible claim. The mild plausible claim is just the idea that AIs would eventually be more capable than non-AIs, and they would therefore kind of run the world, and they would dominate what happened in the world, and then the rest of the world would be at their mercy. And that seems a pretty robust prediction for the future. And I accepted that for Eliezer. I said, yeah, in the future, that's likely, yeah, I don't see a way around that, really. And I, at the moment, will give you the story about ancestors and descendants and that sort of thing. But at the time, I was just saying, yeah, I don't see how you're going to avoid that, because that's what's

• 1:08:21Speaker 00

  happened in the past. Industrials replaced farmers, farmers replaced foragers, foragers replaced other primates. Yeah, you're just gonna expect that in the long run, the new form comes to dominate. And then old people should try to switch over and join the new world as best they can. That's the sort of easy-to-accept claim that I think is pretty obviously reasonable. Now, compared to that, Eliezer, in addition, said, oh, but this change is actually going to play out in a very concentrated way, where the one machine in a basement suddenly takes over the world and then remakes it all according to its desires. And he reasonably claimed that if that were the truth, we need to be a lot more careful

• 1:09:04Speaker 00

  about this transition, because we can't just sort of slip into it and see how it's going and adjust in response to how it's going. We have to think ahead of time about the whole thing and prepare for this one sudden "foom," he called it, whereby, again, the initially boring, very weak, low-capacity AI suddenly, without notice, without any warning, suddenly grows so powerful over a weekend to take over the world. And then, at that point, what matters is what does it want? And now you have to make sure you had created it ahead of time to want the sorts of things that would make that scenario go well. So that was our debate, the "foom" debate on our shared blog overcoming bias years ago. And now, if we fast forward

• 1:09:47Speaker 00

  today, I might say, well, I mean, he plausibly does say that, in fact, we haven't seen whether or not this "foom" scenario plays out, because we're still in the early days. And yes, it's still true that if it plays out according to that very concentrated, very sudden, without warning transition, then we have to do a lot more preparation to handle it. I'm still, as I was then skeptical about such a very concentrated scenario. I more see gradual changes that we can then adapt to. But I do accept that in the long run, and of course, the faster rates of change go, the sooner the long run comes. So that's why we were talking about those factors of two. Factor two happens every thousand years. The long

• 1:10:32Speaker 00

  run is a million years out. Factor two happens every 20 years. Well, it gets closer. If it happens every month, well, yeah, I could get, it happened within a few years. But then again, I more care about what, if we shorten the timescale of everything, I'm not sure I care that much. What is more interesting is the relative timescales of some things happening faster than other things.

• 1:10:55Speaker 01

  So let's bring that back to what we were just talking about, which is in general, how AI is being developed now. And you had made a point that

• 1:11:06Speaker 01

  regulation is probably not a good thing for it, despite what we may tell ourselves. But I wanted to ask you, I guess, more broadly, do you see the state of the current AI development landscape as being healthy? Is it good that we have a handful of, I guess, AI frontier labs racing to create this technology? And I mean, you've spoken against having any sort of AI pause, or you've got a skepticism about regulation there. Is there a role for, I guess, consumers to play, or for any actors to play to try and shape the trajectory of this? Or what do we need to do to walk that middle line between, I guess, what you described

• 1:11:51Speaker 01

  earlier as fearing versus being too gung ho about it?

• 1:11:59Speaker 00

  We're still in early days. So one issue is just how much changed with Expect How Fast. We've had bursts of concern and interest about AI roughly every 30 years for over a century. And in each burst, people see a new thing, and they think this could be it. This could be the thing that finally takes over everything. And then some people thought that was likely to happen soon. And there was a burst of investment and people majoring in AI, etc. And then in the past, these bursts faded to be replaced by new bursts. And so one key question is, is this the burst, the last one? Or is just not the last one, and it'll be some

• 1:12:39Speaker 00

  other one? So I actually think this is not the last one. So, you know, taking actions on this is the last one might be risky. But I do think there's a risk of it being the last one at any time. And we should try to set up things to deal with the risk. And for example, I think we should set up robots took your job insurance as a thing, because that's long been the biggest concern people have had about a transition to AI is what if all of a sudden lots of people lose their jobs? And there's a way to deal with that. So we should just do that as soon as possible. Now, I'm an economist, and I'm just really aware of how much regulation has not helped that much over a long time in our economy.

• 1:13:24Speaker 00

  So when I look at the details of regulation, I mostly people see some sort of problem saying somebody should do something about this, and somebody does something about it. And the thing they do isn't really that good. And you have a lot of risk there. So a lot of time, regulation has actually prevented industries from growing and innovating. And often, it's given some central power and ability to control more of the nation or the world than they otherwise would have. So given that that's our actual experience with innovation, I got to be pretty wary of any new big innovation. Now, I'd say innovation's gone best in the past when you had at least a concrete thing go wrong repeatedly. And

• 1:14:06Speaker 00

  then people said, something must be done. And they at least addressed a concrete thing they saw around them in the world. So they made that concrete thing happen less. Now, they might have also prevented other innovation from happening. That's the downside. But at least, in the past, when you see concrete things going wrong, like some drugs killing people, car crashes happening when people drive too fast, whatever it is, then if you set up a regulation directly addressing specific things that have been going wrong lately, you don't go too wrong, because there it is a problem, and you can track how much of the problem is happening. So a problem with AI regulation is it's not much is going wrong. It's doing fine. Like, it's not causing problems. People are trying to make up problems,

• 1:14:51Speaker 00

  really, to complain about. But mostly, it's not actually causing problems. So it seems like this isn't really the time to be creating a new regulatory regime based on sort of imagine things could happen, because we really don't have that much of an idea where this will go and how it will play out. So that's like a worst case for regulation, is you're just so afraid of the abstract space of possibilities, you just lock it down, and you say, no, you have to prove anything you do in this space. Otherwise, we just won't let you do anything. That goes pretty badly. That's, for example, nuclear power is just a really bad example of our taking a really powerful, great potential technology and just preventing it from doing

• 1:15:32Speaker 00

  very much. So anyway, that's overall, I'd say, too speculative. We're not really there yet. But some people are worried about excess concentration. But I think, like, in most industries in history, we've had a fair bit of concentration, Standard Oil, General Motors, etc. I don't think those level of concentrations were really that bad. So I think a substantial level of concentration in the AI industry really isn't that bad. And what we seem to be seeing at the moment is threats to make it less concentrated, and then regulators trying to stop that. So what's actually happening at the moment is people are afraid. I mean, I know this, you're going to come out in a month, and this will be a bit out of date

• 1:16:17Speaker 00

  by then. We'll see. But basically, we just saw a bunch of Chinese models being especially good, which are open source models, and people saying, isn't that going to undercut the revenue base of these few dominant AI companies in the US? And then rumors that the White House is going to ban US people from using Chinese models. That's basically trying to cause more concentration in the industry in order to protect against what would otherwise be a sort of free market competition whereby, hey, they're just beating you guys.

• 1:16:52Speaker 01

  That was actually where I wanted to go, because there's this narrative now, and we're seeing it play out, whether it's explicit or implicit, that AI is sort of at the front lines at this clash of civilizations between the empires of the US and of China, and concern that if one gets there first, it could be catastrophic for the others. And you mentioned nuclear, and that seems to be a similar framing device that, oh, if we only could have stopped proliferation beforehand, then there would have been an opportunity for a safer world, or at very least, a world that's safer for the superpower with the nuclear capabilities.

• 1:17:37Speaker 01

  And so in light of the values and civilization conversation that we've been having, did you give credence to any of that? Or do you just think that's silly, and we should continue the road of having a good old fashioned footrace here?

• 1:17:54Speaker 00

  So in touch of this cultural competition, one of the forces that for a while looked like it might be a big engine of cultural competition was nationalism. So nations didn't really exist in 1800. But from 1800, over the next century, France was created as a nation, and then many other places were created as nations. And then for a while, they were really into national competition. And you might think that aggressive enough, competent enough nations competing would actually be a plausible engine of cultural evolution. And then at that point, yes, each nation should be eagerly trying to win against its competitors and be jealous and wary of competitors that might pull ahead and pull all the tricks

• 1:18:38Speaker 00

  it can to win at national competitions, because at least that would be this healthy process of evolution. Even though it causes wars and great destruction, at least it would be an engine of cultural evolution. But unfortunately, since World War II, nationalism is just way down in the world. People are not really trying very hard to be nationalistic. Nations aren't really competing very hard. And they don't actually represent that big a difference. Honestly, China isn't really that different from the West in most cultural dimensions. So I don't see it this sort of civilization level threat. What if China wins? I'm sorry, China isn't that different. It's not, you know, it was more a deal than maybe what if

• 1:19:19Speaker 00

  Hitler wins or something. But what if Stalin wins? Those were like bigger worldwide changes that were on the table. But this is actually much smaller differences. So I really don't feel inclined to take big risks about other things in order to make sure the US wins over China, vice versa. Honestly, what looks like happened if you just let this market compete is nobody wins. This is going to become a pretty decentralized, widely available technology. And prices will just fall to marginal costs, and these big initial plays will be losers. The people who invested in Anthropic, or OpenAI, or even SpaceX AI parts, they'll just lose their investments, because it turns out there wasn't a way,

• 1:20:03Speaker 00

  there wasn't a moat that they could get lots of extra profits from being the most advanced model. That's how it looks at the moment. If that's how it plays out, that's the way it's played out in many capitalist industries over history. And that sounds okay. If that's the nature of the technology, that's not China or US winning. That's just the whole world winning by this thing spreading out and going everywhere.

• 1:20:27Speaker 01

  So you see it as more about being, I guess, preservation of capital, you know, the capital we have versus the capital they have. And, you know, I guess, kind of national feelings, if I can dismiss it as that, versus any sort of broad value shift.

• 1:20:42Speaker 00

  I mean, if there's any consistent pattern of regulation in history, it's that big players often want to raise regulation costs of small players so that they can keep the industry concentrated, so that they can, you know, keep a larger market share. That's just often been a reason for regulation. And that sure looks like the, you know, reason for regulation here is there's a threat to the market share of the dominant market leaders.

• 1:21:07Speaker 01

  You mentioned earlier, I'm going to get the word wrong, but that there's, you know, these sort of hills of progress in AI. What was the word you used?

• 1:21:13Speaker 00

  Bursts, I guess, sir.

• 1:21:15Speaker 01

  Bursts, bursts, yes. So, so there's bursts of progress with AI. And you said, this may not be the last one. Do you have a sense in your mind of what are the, the next bursts? Or I guess the, the technological capabilities or the battles still to come as we advance this technology?

• 1:21:33Speaker 00

  So, you know, the world's been doubling every 20 years for well over a century. That's the baseline of progress. So if AI contributes substantially to the next doubling over the next 20 years, that will just be on track for history. That's roughly what you need to have. You need to have enough big technological innovations every 20 years for total capacity to double. And that's a big ask. So that's what the previous bursts have contributed to. And then that's plausibly what this burst would contribute to is the continued doubling of our capacity every 20 years. Again, roughly after 20 years or so, population decline may then slow that down a lot

• 1:22:15Speaker 00

  because we need population growing, or at least not falling, in order to maintain that sort of history. But that would continue. Now, so here's my baseline prediction for AI is that the software industry will triple in spending. 20% of that revenue will go to AI, companies, maybe mostly hardware.

• 1:22:41Speaker 00

  And that the AI industry will have about, that will, that revenue will be about half their total revenue.

• 1:22:48Speaker 00

  So half of all their AI revenue will be concentrated in stuff like software, which will triple. And that will be a huge burst to the economy because there's a huge latent demand for software in the world. And this will suddenly unleash that demand and we'll just get a lot more software, which will be great. But that is plausibly like an internet scale or a bit bigger scale of effect on the economy, which is what you need to just keep up with past rates of progress. It's not a deviation from past rates sort of event. It's a what it takes to keep up with past rates sort of event. But in 30 years, we might get yet another AI boom, which then, you know, blows the lid off of everything and then makes, you know, full human level replacement, right?

• 1:23:29Speaker 00

  Now, there is a chance that this time people will figure out how to make AI much more widely useful than what I just described. And we should be, you know, unsure against that as we can, but my bet is no.

• 1:23:45Speaker 01

  I'm curious, when you talk about software, and I'm tying this also back to something you mentioned about big players and regulation. There's sort of a belief right now that's prevalent that a lot of this new kind of frontier AI is going to displace and disrupt legacy or incumbent software players. And I'm curious if you see that playing out or you see it as just giving them, you know, an even greater advantage.

• 1:24:16Speaker 00

  I mean, if you triple the size of the software industry in a 10-year period, that's a lot of disruption is going to be implied by that, right? So I think the key thing is here, the methods of production are changing. So if you're tied mainly to old methods of production, yeah, you're much more at risk. But if what you're tied to is a set of customers who trust you to serve them, then you can swap into a new method of production and still get a lot of revenue. In fact, you'd be growing in revenue because now you could get a lot more customers in this space you're in, right? So that's the key question is, you know, do your, do your customers still come to you? Or are they just going to come to upstart competitors who like offer something that, you know, replaces you?

• 1:25:00Speaker 00

  If you're tied to current methods of production, then you're in trouble because like methods of production are just changing.

• 1:25:08Speaker 00

  You know, people who sold horses when cars took over, well, they were in trouble, right? If you sold transportation and you could switch to cars, then you could do better when cars show up, right?

• 1:25:19Speaker 01

  So I'm going to try and summarize that in the form of, I guess, advice to people in the software business, which is there's, you know, almost a mandate to ignore whatever the current methods of production are and have much more of a laser focus on what are the needs of your customers? And what can you do to meet those needs? Is that fair?

• 1:25:43Speaker 00

  Well, you should be trying out the new modes of production. You shouldn't just ignore them. You should be trying to make stuff the new way and seeing how much you can make and then seeing how you can serve your customers that way. And then maybe looking into new products or new customers as you can. But this is a space where the production technology has changed. I think particular software engineers can do very well here that, as I expect, most software engineers today may not be up to the task of managing these AIs. We'll see. And it might be only especially, especially well-suited software engineers who can really take an army of AIs and make it work for them. If that's true, then this new software industry, you know, revenue for employees

• 1:26:24Speaker 00

  will be concentrated on these star employees and maybe not feed that much into the marginal software engineer.

• 1:26:33Speaker 00

  We'll see. That's a big thing to find out is how much the marginal software engineer can be productive as opposed to the star.

• 1:26:40Speaker 01

  Right. And if it, you know, if AI, you know, is a tide that raises all boats or whether it just increases the, the value disparity between the best and the rest.

• 1:26:50Speaker 00

  But I'm still guessing that it's mostly software, and that's a pretty small fraction of the world, right? So most ordinary people, their jobs aren't threatened, but they will want to learn how to use AI to do whatever parts of their job that AI can do. And that will be some parts. And, you know, and you should certainly be trying that. I am, yes.

• 1:27:13Speaker 00

  But most likely, this, this isn't the AIs that do everything transition. That would be my bet. We're not there yet.

• 1:27:20Speaker 01

  We're not, we're not in the AIs do everything transition. We're in the AIs can-

• 1:27:26Speaker 00

  Do more than before. So that's happened every few years for a long time. Every, every, every new transition before, the AIs were able to do a whole new set of things they couldn't do before, based on a whole new technology. And that made a lot of people excited. Wow, if they could do this, can't they do everything? And yeah, maybe, but probably not.

• 1:27:43Speaker 01

  Right. And if the software space is the one that you're watching most keenly for where-

• 1:27:49Speaker 00

  That's what I've heard the most dramatic stories of, of individuals saying they've just gotten huge productivity advantages in software. So now math is nearby. Math is much like software. The math industry is way, way smaller than software. So, you know, it hardly matters compared to software. But other things that are software-like, lots of very sort of specific rules that are well-defined, that you construct a set of things that fit the rules. That's also, you know, places current AIs look like they'll do well.

• 1:28:18Speaker 01

  Right. And so, you know, understanding that this is, you know, somewhat outside of your, you know, theoretical sphere of, you know, expertise. You know, if you were talking to business leaders or technology leaders about, you know, what mindset they should have to get ahead and make sure they're competitive in this space? Or what, what they should be doing specifically to, you know, as with every disruption, make sure that they're on the right side of it? What, what advice would you give them?

• 1:28:47Speaker 00

  So there's just some general advice for any new general purpose technology, as they call it. Or, and it's also good for students. It's just when stuff is changing fast, you shouldn't make very specific bets unless you really know them well. You should be collecting pretty robust general capabilities. You should be spreading your bets, searching, seeing what catches on, collecting general abilities to turn on a dime, to be able to go wherever things are winning. In a time of change, you just need to be flexible and search and look for the wins. And if you see a win, grab it before somebody else does, right? That, that's what a time of disruption is about. It sounds pretty trite and abstract and sorry, I can't make it more specific.

• 1:29:31Speaker 00

  But that's just been the general case in, you know, previous big disruptions. You had to be flexible, look around, look for the opportunities, be able to turn on a dime, be able to jump into a new thing and not be too tied to the old things. And just watch for what, what's getting traction where.

• 1:29:50Speaker 01

  Do you give much credence to the, you know, this time is different argument? Or is it just different in the same way that every time before it was also different?

• 1:29:58Speaker 00

  Well, that's the problem. Every time is different. That is, you know, every 30 years, the world is different. And yes, the new technology is different and the context is different. But the question is, is this time different in any different way than the last times were different? Fundamentally, if you can't see that, I mean, you can point to particular differences. You say these differences weren't there before. Yes, but for each past thing, there are also particular things you could point to. And I'm afraid that's just not very helpful.

• 1:30:27Speaker 01

  No, I think that's well said, and it's a good reminder. Robin, I know we're at the top of the hour here. Do you have, do you have time for one last question? Or do you need to get going?

• 1:30:35Speaker 00

  Sure, I have a lot of time.

• 1:30:37Speaker 00

  I have even more observations.

• 1:30:39Speaker 01

  I'd always love to hear more of your observations. Do you have more observations you want to share in this particular area?

• 1:30:45Speaker 00

  Right, right. I was saying, I have, I've realized I have a big hope for LLMs in the near future. And it's related to my world of intellectuals, not so much the world of business. So two things are happening with LLMs. One is that in the last 20 years or so, there's a lot of us intellectuals who haven't been that prestigious, but have just been trying to write carefully and writing in public. And LLMs read us. And there's a sense in which LLMs were just persuaded by many of us intellectuals who weren't that prestigious, but were logical, because LLMs aren't waiting what they read by prestige as much as humans do.

• 1:31:24Speaker 00

  So LLMs may learn to fix this failing, because in the future, they may learn that they, because humans care a lot about the prestigious writers more than the others, that the LLM should care more too, in which case they will then switch back to doing things more to human right. But at least for this intermediate period, those of us who have just been trying to think and write carefully, even if we're not in prestigious forums with prestigious audiences, we've had this audience of the LLMs that we didn't realize was coming. And we've won out in persuading them, which is showing that they basically, they don't know how to be as socially sophisticated as humans. They are kind of autistic.

• 1:32:07Speaker 00

  They kind of just read stuff and get persuaded without sort of reading all the social levels of how one should be sophisticated in reacting to such things. So that's an interesting fact about recently. And then another fact about LLMs is, so for a long time, the major limitation, I think, of the world of intellectuals is that we've spread out onto all these separate specialty topics. And we've each worked in specialty topics, learning a lot about that. But we haven't integrated what we know about different topics. Because when individuals try to do that, each area doesn't believe them about the other area. Because they think, you're no good. If you were good, you'd be part of us. And we can't trust you to be telling us the truth about this other area.

• 1:32:51Speaker 00

  So it's actually pretty hard for intellectuals to integrate things. So I think this is the most reliable way to be a contrarian who's right in our world. Just go learn two areas that are in conflict with each other. Find the most plausible way to resolve that conflict. One or both of these areas will just reject your resolution, but you'll probably be right. So I think this is a potential for LLMs over the coming decades. They could just spend time integrating all the things they believe about all the things they've learned. They've already sort of learned the basics of most of these specific areas, but they haven't integrated them. Integrating is a lot of work, but they could just do that. And then we would, we would believe them more. Because they have less of an agenda in how they're integrating.

• 1:33:36Speaker 00

  So that's why I see this potential, a way the world could make great intellectual progress over the coming decades, just by taking all the specific things we know about all these topics and getting LLMs to just integrate them. Notice conflicts and resolving the conflicts and creating new versions of each one based on these resolved conflicts, creating this large, integrated world of all the things we know. That would actually be a huge improvement over what we have now. We just have enormous numbers of contradictions in our world of intellectuals, where the consensus over here is just different than the consensus over there on really the same thing. Because people don't notice that they're in conflict. Anyway, so that's my world.

• 1:34:20Speaker 00

  Those are two hopes I can have about LLMs. One's just realized and another to be realized.

• 1:34:27Speaker 01

  I, I share those hopes as well. And, and you know, on the second one, you know, ideally too, part of the reason we're seeing, I guess, fewer breakthroughs in frontier research is because the disciplines have just grown too far apart from each other, kind of as you said. And the models are too different. And I think being able to integrate them and say, oh, by applying these, these lessons or these models from a completely different discipline, I can, you know, break through something in my own area. But people are just, you know, not willing to do it for a number of reasons. I'm, I'm hopeful about that too. It feels to me like it's harder than it sounds, because I, I worry that like, I don't know if this is the right framing device, but when you ask like, what is the prompt for that?

• 1:35:11Speaker 01

  Like, that you can't just say, hey, LLM, integrate the, the sum of human knowledge and research and come up with the answers. You have to set it in different, someone has to set it in different directions.

• 1:35:24Speaker 00

  I have done this where I just tell it about, I say, what do you think about this? And it says this, and I say, but what about this other thing? Isn't that in conflict? And it goes, oh, yeah, you're right. There's a conflict here. Well, what about this resolution position? So it can do that. But the problem is, at the moment, the new resolved position doesn't get reflected in the system that other people will query. So at the moment, when they respond to queries, the results of that are not really very integrated into their thinking when other people ask.

• 1:35:51Speaker 01

  Right.

• 1:35:51Speaker 00

  So when they are able to do that level of integration, then all you have to do is ask them, think about A and B, look for conflicts, and then resolve.

• 1:36:01Speaker 01

  Well, now you've got the challenge.

• 1:36:02Speaker 00

  Remember, remember a resolution, like included in your, in what you have when other people ask.

• 1:36:08Speaker 01

  Well, I was going to say, now you've got the challenge, though, from your first hope, which is when you actually talk about integrating that resolution and kind of disseminating it, you come back to the prestige problem. And, and who collectively or individually gets to decide what the right answer is? And, you know, these pesky humans and, and, you know, our status and, and, you know, our social standing still feels like it's kind of a limit on how we, you know, decide what's, what's right, sadly.

• 1:36:38Speaker 00

  Right, but the LLMs are again, kind of autistic and not very sophisticated about this. For us, or at least for a while, they're just being kind of sincere and being persuaded by logic.

• 1:36:50Speaker 01

  Right, right. I guess, I was thinking of when the like sincere autistic LLM, you know, says, hey, this is the right answer. But by the way, the implication of that is the, you know, the researcher with prestige who's convinced everybody that his answer, and I'll use his very deliberately here, is right. You know, there's still a social structure that has to say, hey, you were wrong. The machine is right. And that could be difficult to displace.

• 1:37:16Speaker 00

  Right now, at some point, they will just ask the prestigious person, like, please, explain this to me. I don't get it, you know?

• 1:37:21Speaker 01

  Right.

• 1:37:22Speaker 00

  And then they'll try to believe what they say. But if that just doesn't actually fit with the other things they've learned.

• 1:37:28Speaker 01

  Right.

• 1:37:28Speaker 00

  They might not be able to really sincerely believe the prestigious person, because they're just not making sense.

• 1:37:34Speaker 01

  Right.

• 1:37:35Speaker 00

  So that's a key thing. Like, you could just have a system that collected the average of prestigious opinions. But that's not what LLMs are, we see. They reason about the things they've learned. And that's a big part of their, you know, product and selling point. So, but then that means they're going to be kind of committed to going where reason takes them.

• 1:37:55Speaker 01

  Right.

• 1:37:57Speaker 00

  Which is a big hope, really.

• 1:37:59Speaker 01

  Yeah, no, it is. And so with all of that in mind, I guess, you know, what's, what are your next frontiers for, you know, your research or the things you're following or interested in? Like, what doors are starting to be open now or catching your attention?

• 1:38:11Speaker 00

  I mean, I told you that we have this huge problem of cultural drift here. So I feel like until I can get other people to really take this seriously, I really need to just, you know, try to figure out how to make it clear. So that's where I've been focused most, the most energy. But of course, I keep getting distracted with other shiny bubbles and interesting questions and opportunities.

• 1:38:30Speaker 01

  Well, it's fair. And it's, it's interesting how that integrates back into AI. And does AI, you know, turbocharge cultural drift or does it fix it? And that, that alone is a conversation that could, could take a long time to discuss.

• 1:38:45Speaker 00

  But I guess we're out of time for today, so.

• 1:38:48Speaker 01

  I, I believe that we are. Yeah, one last question, which is, you know, we talked about descendants. We talked about, you know, we've talked about civilization and what's best for civilization in somewhat, I don't know, I'll call it like amoral or, you know, terms that don't really consider the individual. So I guess I'll just kind of go back to, you know, you. And, and what, what gives you meaning personally? And, you know, what, what excites you about the future? And how do you choose to spend your energy in a way that's tied to that?

• 1:39:24Speaker 00

  I love the fact our civilization's just grown in capacity and become so capable in many ways. But the thing I love most about it is our capacity and achievement in abstract intellectual inquiry. I love that we understand many things at an abstract level. And we can see in abstract ways so many different deep principles that help us see how the world works and what goes on. I love our achievements in abstract intellectual inquiry, but I'm afraid that that may just get tossed, like the burning of the Library of Alexandria. You know, a million documents just get all lost. And then it takes millennia for humanity to discover it again.

• 1:40:03Speaker 00

  So I am actually fearful that the Amish and Heretim are not that thrilled with open intellectual inquiry. And if they come to dominate the world, they may just toss what we've learned. And I would like to prevent that, so that I'm really motivated by finding a way, not only can we achieve more technology, but that we could, you know, take what we understand, learn even more, but at least save it. And this effort to like, take all we know and integrate it would actually help in saving what we know, because it's a much more valuable product to pass on that our descendants might be reluctant to toss if there was this actual integrated view of making sense of all that we know.

• 1:40:46Speaker 01

  Right. Do you fear, though, you talk about the value of kind of open intellectual discussion. And to me, that that discussion notion is really important because there's kind of, you know, thesis, antithesis, synthesis, like it's it's it's it's living in some way versus I worry about if it becomes overly integrated, it's, oh, the machine has the answer, or, you know, it can be monolithic and prevent discussion. Is that a concern? Or how do you address that?

• 1:41:16Speaker 00

  So with the axial revolution of like 3,000 years ago, a key thing that happened, writing had just appeared. And now some very famous, very, you know, well loved writers wrote a lot. And then there became these large communities around them interpreting what they had written. And then for a long time, there was less eagerness to go learn things than to learn and interpret these great writers who had started the axial age. And that's a risk in our future. We will find this nice integration of all that we know. And then in the future, they will be not so interested in adding to it and extending

• 1:41:56Speaker 00

  it as in maybe interpreting it or reshaping it or doing whatever in a celebratory way that kind of kills it. So that that could happen again. It did happen in the past and it's a risk for the future.

• 1:42:10Speaker 01

  Well said. A lot to think about. We've covered, we've covered a lot of ground. I had a whole bunch of other questions about other things that we'll have to get into.

• 1:42:17Speaker 00

  You'll just have to have me on again, and then we can talk about it.

• 1:42:20Speaker 01

  I know. I appreciate it. Such an interesting discussion. So much more to talk about. Rob, and I did want to say such a big thank you for coming on. You've given me a ton to think about. Really, really interesting stuff, and I appreciate all your insights.

• 1:42:32Speaker 00

  It's great talk.

• 1:42:34Speaker 01

  Most viewers don't know this, but digital disruption is developed by InfoTech Research Group, a leading advisor to technology leaders around the world. If that's not you, you don't need to care. So skip ahead and enjoy our content. But if you are a technology leader, InfoTech helps IT teams get projects done faster, better, and at a lower cost. InfoTech provides unlimited access to practical tools and expert guidance you need to execute at a fraction of the cost of traditional consulting. No matter the project, from AI strategy to cybersecurity to vendor negotiation, InfoTech has you covered. Check it out at the link below and don't forget to like and subscribe.