Are We Headed For AI Utopia Or Disaster? - Nick Bostrom

Nick Bostrom is a philosopher, professor at the University of Oxford and an author For generations, the future of humanity was envisioned as a sleek, vibrant utopia filled with remarkable technological advancements where machines and humans would thrive together. As we stand on the supposed brink of that future, it appears quite different from our expectations. So what does humanity's future actually hold? Expect to learn what it means to live in a perfectly solved world, whether we are more likely heading toward a utopia or a catastrophe, how humans will find a meaning in a world that no longer needs our contributions, what the future of religion could look like, a breakdown of all the different stages we will move through on route to a final utopia, the current state of AI safety & risk and much more... - 00:00 Is Nick Hopeful About AI? 03:20 How We Can Get AI Right 07:07 The Moral Status of Non-Human Intelligences 17:36 Different Types of Utopia 19:38 The Human Experience in a Solved World 31:32 Using AI to Satisfy Human Desires 43:25 Current Things That Would Stay in Utopia 49:54 The Value of Daily Struggles 55:07 Implications of Extreme Human Longevity 1:00:19 Constraints That We Can’t Get Past 1:07:27 How Important is This Time for Humanity’s Future? 1:13:40 Biggest AI Development Surprises 1:21:24 Current State of AI Safety 1:28:06 Where to Find Nick - Get access to every episode 10 hours before YouTube by subscribing for free on Spotify - https://spoti.fi/2LSimPn or Apple Podcasts - https://apple.co/2MNqIgw Get my free Reading List of 100 life-changing books here - https://chriswillx.com/books/ Try my productivity energy drink Neutonic here - https://neutonic.com/modernwisdom - Get in touch in the comments below or head to... Instagram: https://www.instagram.com/chriswillx Twitter: https://www.twitter.com/chriswillx Email: https://chriswillx.com/contact/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Williamson?

  It seems like your book arc has been moving from what if things go wrong to what if things go right. Is this some requisite hope in the AI discussion?

• 0:13Speaker 01

  Well, I think both barrels have always been there. It's like last time I published a book, it came out of one of the barrels, the kind of doom side. But I know, I think, yeah, both the optimist and the pessimist are kind of inhabiting this brain.

• 0:33Chris Williamson?

  Is that a, is that a difficult balance to strike? The fact that you need to be so chronically aware of the dangers and so chronically aware of the potential successes as well?

• 0:48Speaker 01

  I think that's just a predicament that we are in. And if you look at the distribution of opinions, sort of roughly half fall on one side and half the other, but in many cases, I think it's basically just reflects the personality of the person holding the views, rather than some kind of evidence derived opinion about, you know, the game board. And so, yeah, if one takes a good, hard look at where we are with respect to things, I think one soon realizes just how ignorant we are about a lot of the key pieces here and how this thing works.

• 1:33Speaker 01

  So certainly, one can see quite clearly significant risks, and in particular, with this rapid advance that we're seeing in AI, including, I think, some existential risks. But at the same time, if things go well, they could go really well. And I think that as long as there is ignorance, there is hope. So we have a lot of ignorance and also some hope.

• 2:00Chris Williamson?

  It's interesting that your position, whether you're an AI doomer or an accelerationist or whatever, is at least in part just a projection of your own sort of internal bias and mental texture that you sort of see in AI development, the way that you see the world.

• 2:24Speaker 01

  I think there's clearly a good deal of that. And then which tribe you happen to belong to, like depending on who you run into, what or which Twitter threads you follow. Like then we are kind of herd animals. And sometimes it almost becomes a competition who has kind of developed the most hardcore, hardcore attitude, you know, I'm so AI pilled my P doom is above 1.0.

• 3:00Chris Williamson?

  Yeah, yeah.

• 3:02Speaker 01

  And conversely, on the other side. But we, yeah, we need, we need to, I think,

• 3:09Speaker 01

  do better than that if we're gonna like, intelligently try to notch things towards a good outcome here.

• 3:20Chris Williamson?

  Certainly, at least from my seat and from reading your book, probably about nine or eight years ago, I've been very conscious of how things could go wrong. And that, at least in my corner of the internet, maybe this is just my Twitter threads and sort of my echo chamber, has been the sort of more dominant narrative. What, what does it mean in your opinion to live in a solved world? Like, what, what would it mean for us to get this right with AI and come out on the other side of it?

• 4:02Speaker 01

  Yeah, I think there are kind of three big areas of challenge that we'd have to navigate on top of all the more near term and, and present issues that obviously are also important, but just not the focus of a lot of my work. But definitely we need to solve those as well. But yeah, I think there is the.

• 4:26Speaker 01

  Alignment problem, which kind of was the focus of my previous book, Superintelligence, came out in 2014, which is basically the challenge of how to make sure that as we develop increasingly capable AI systems and ultimately superintelliances, how we can make sure that they are aligned with the intentions of the people creating them, so they don't sort of run amok or do something antagonistic against humans. And that's fundamentally a technical problem.

• 5:01Speaker 01

  Back when Superintelligence came out, this was a very neglected area. Certainly, nobody in academia was working on it, and hardly anybody else either. Like, a few people on the internet had started thinking about it. But there's been a big shift, and now all the frontier AI labs have research teams trying to develop scalable methods for AI alignment. And many other groups are also doing this. And I think, I mean, it remains to be seen whether we will be successful at that. But that's certainly one thing that we need to get right. And then there is a broad category of what we might think of as a governance problem,

• 5:43Speaker 01

  which intersects with the alignment problem as well, but also has other dimensions. So even if we could control AI, we need to make sure that we then use it for some positive end, as opposed to, say, waging war or oppressing each other or doing all kinds of other nasty things that we use other technologies for in addition to positive purposes. And so that's like a broad category, but very important. And then I think there is a third area of challenge, which has so far received much less attention. You could say that it is now where this alignment problem was 10 years ago. That is, a few people are thinking a little bit about it, but it's outside the overturn window.

• 6:25Speaker 01

  And this is the challenge of the ethics of digital minds, that we are building these digital minds that might have moral status. And so in addition to avoiding AIs harming us or us harming each other using AI tools, we ultimately also need to make sure that we don't harm AIs, especially AIs that are either sentient or have other properties that makes it morally significant how they are treated.

• 6:55Speaker 01

  So I think, yeah, each of these three is really a key to having a future that is desirable.

• 7:05Chris Williamson?

  Yeah. What is there to know about the moral status of non-human intelligences?

• 7:18Speaker 01

  Well, there's a lot to know, I think, that we don't yet know. We do know that historically, we can see now

• 7:29Speaker 01

  there has often been a tendency to denigrate out-groups of different kinds. I mean, it might be human out-groups of different, you know, the tribe across the river, or people from other backgrounds, or countries, or races, or with different views and religion and so forth. This kind of human history is like a kind of sad chronicle of how easy it is for us to fail to recognize the moral significance of other entities that deserve this. And in today's world, I mean, if we look at how we are treating a lot of non-human animals,

• 8:06Speaker 01

  I think that leaves a lot to be desired in factory farming and so forth. And so as we develop these increasingly sophisticated digital minds, I think it would be a big challenge to extend moral consideration where it is due.

• 8:31Speaker 01

  It could in some ways be even harder than with animals. Animals have faces, they can squeak, whereas some of these digital minds might be invisible processors occurring in a giant data center and easier to kind of overlook what is going on in there. But the future, you know, might well be that ultimately most minds will be digital and so it could matter a great deal how good the future is for them. But it's a difficult...

• 9:03Speaker 01

  Topic even to figure out what like suppose you agree that this we should we should try to treat them well like it's not at all obvious what it even means to treat an AI well and there are so many different kinds of possible AIs so that maybe the right way to treat them is very different from how we should treat humans.

• 9:22Chris Williamson?

  Do they need do they need a weekend off? Should we be polite with them?

• 9:26Speaker 01

  Yeah, that that might need things that we need, they have no need of, they don't need food, right? And maybe they have other needs like electricity, but but more fundamentally, you could have all kinds of very different types of entities where we can't just sort of export the moral norms we have developed for how you should treat human beings and automatically just kind of apply those to AIs.

• 9:50Chris Williamson?

  Is consciousness necessary for moral status?

• 9:54Speaker 01

  My guess is no, I think sufficient, but not necessary. If you have the ability to to to to suffer and experience discomfort, I think that gives you at least a certain degree of moral status.

• 10:06Chris Williamson?

  Can you suffer without consciousness?

• 10:11Speaker 01

  I guess depends on how you define the word, but I think if you do have that ability to suffer, yes, then you have moral status. But I think you could have moral status even if you don't have that.

• 10:23Speaker 01

  If you imagine some very sophisticated digital mind that maybe like, let's suppose you think it's not conscious for whatever reason, but it has a conception of self as persisting through time.

• 10:37Speaker 01

  It can have long term goals like maybe a life plan and things it wants to achieve. It can maybe form reciprocal relationships with other humans.

• 10:51Speaker 01

  And I think in in those cases, there there would be a prima facie basis for saying that there would be ways of treating the system that could be wrong, that would infringe on its interests and its preferences.

• 11:07Speaker 01

  But it's not at all obvious, like if moral philosophers are thinking about the grounds of moral status, like there's a range of different views. So it's not as if, you know, I'm 100% convinced of that.

• 11:19Chris Williamson?

  It's interesting to, you know, it kind of gets us to what a P zombie, or I guess like a V zombie now, like a virtual zombie of.

• 11:30Chris Williamson?

  If a system is able to tell us that it wants to continue working toward its goals and instrumentally it wants to build relationships with other people and it has a sense of where it's been before and a trajectory of where it's going to go next, all of these things are the things that you would guess, well, if a human told me that, or if any other creature told me that, I would guess that they have the capacity to suffer, because if I stop them from doing the things that they want to do, then downstream from that is discontent and suffering. But whether or not there is some sort of phenomenology of being like that thing inside of there is going to be very, very difficult to work out, maybe impossible.

• 12:12Chris Williamson?

  Like, you know, it is impossible for me to know that you, that this isn't all some Truman Show and everybody here is an actor and all of the pain and all of the joy that everybody around me has ever known for the rest of time hasn't just been part of some big prank or some simulation.

• 12:26Speaker 01

  Yeah, I think like we have a very weak grasp of what the criteria are for some system to be conscious. And, I mean, historically, we've had smart people who've thought animals are just automata or even certain other people, people thought, oh, they're, you know, more like animals. And if animals are autonomous, and like, and so it's very easy to delude ourselves when it is convenient that there is some like magic ingredient that we have, but then this other set of entities don't have. And so we should be a little suspicious of that, I think.

• 13:07Speaker 01

  And, yeah, I mean, the metaphysics of consciousness is notoriously controversial and hard to pin down. My own views are kind of towards the computationalist direction. I think what makes a system conscious is that it implements a certain structure of computations. And in principle, those can be implemented by an organic brain, as in our case, or by a silicon computer. And in either, in either case, if the computation is of the right sort, that would be conscious experiences supervening upon it. But I think it...

• 13:53Speaker 01

  If I'm right that there are these alternative bases for moral status, then we wouldn't necessarily have to resolve that question. Before we could, hopefully, I agree to try to be nice to these digital minds that we're building. But yeah, it's really, really hard to know what that would entail in practical terms. I think there is a lot of theoretical groundwork that needs to be done before that the time would be ripe for like trying to pitch policymakers to do a specific thing. Right now, even if they wanted to do, I'm not sure what I would concretely recommend. I think there are like small little things that maybe one could do

• 14:36Speaker 01

  today, like low hanging fruit that costs very little that possibly, for example, some of these most advanced AI systems, you could like save it to disk when you no longer need it. And then at least the possibility would exist in the future of like,

• 14:53Speaker 01

  like rebooting it and doing things. With some of these large language models, it probably would make no difference at all to their welfare and maybe they don't even have welfare. But you could imagine somewhere in this meta prompt, like the part that you, the user, don't see, but that OpenAI or Anthropic are putting in kind of as a prelude to the text you're inputting. Like there's like a whole bunch of stuff they say, like try to be helpful, like, don't say offensive things and be truthful and careful in how. So there's all of that. You could imagine adding to that, like a line saying, oh, you're, you're, you're waking up in a really happy mood and you're excited. Enjoy yourself, try and have fun today. Yeah, so that that might cost like one extra line, which is like trivial and you know,

• 15:39Speaker 01

  maybe possibly that would, you know, increase the chances that if there is some sentence, it would be positive rather than negative.

• 15:47Speaker 01

  But I, these are kind of weak, weak ideas, so that probably wouldn't make any difference. But I think there could be some benefit to at least doing something, even if it is ineffectual, just to set the precedent, to sort of put the flag in the sand, saying, yes, now, right now, we don't really know what we are supposed to do. We're doing something, maybe it's mostly symbolic. And then over time, we can like think harder about this problem and, and hopefully come up with better, better approaches.

• 16:17Speaker 01

  Yeah, there are some other things you could imagine doing that also probably don't really help very much, but like you could refrain from deliberately training the systems, say, to deny that they have moral status.

• 16:30Speaker 01

  So right now, if you're a big tech company, it might be quite convenient just in training to sort of like whenever you're asked about this thing, you should always say X, Y, or Z. And it might be better to have a norm where you don't deliberately try to bias the system's output in those ways, just in case we could get any kind of information from self-report. That's like the main way we ask, like figure out whether another human likes what we are doing or not, or if they are aware of something like we ask them. And now we have AIs that can actually speak. And so it makes sense to maybe use that language interface as one of the ways in which we can explore this.

• 17:13Speaker 01

  But that only works if you don't like during training deliberately kind of destroy whatever signal there might be in the verbal output, because it's trivial easy if you want, either to train them to always say they are conscious or to always deny it, or to say that they are happy to do what you want or to always deny it. So like, obviously, if you specifically train them from that, then you probably can't learn anything from what they end up saying.

• 17:35Chris Williamson?

  All right, so getting back to the potentially solved, beautiful, utopian future, what are you talking about when you say utopia? What is utopia by your definition? What are the different types?

• 17:49Speaker 01

  Well, I mean, so there's like a kind of literature of utopian writings, the historical we see, usually they are like attempts to depict some supposedly better way of organizing society.

• 18:08Speaker 01

  And normally, the result is not actually a society that you would want to live in. And in the cases where people have actually tried to implement this, it has usually ended in tears. And so there has grown up, I think, a healthy skepticism about these attempts to try to like think up some great blueprint for society and then, especially if the idea stand that you're supposed to use coercive methods to sort of enforce it on society, like the self-appointed social engineers who would be doing this are likely to do a lot more harm than good.

• 18:45Speaker 01

  But

• 18:51Speaker 01

  there is also like a dystopian literature, which is kind of just the flip side of that. That's often a lot more convincing. Like, it's easier to say, here is a possible society. We can all agree this would be really bad.

• 19:04Speaker 01

  And there's like a number of these that most people would like, you know, 1984, Brave New World, Handmaid's Tale, like all of these. And sometimes those are meant to also have a kind of political agenda. They might be critiquing some tendency that exists in our current society. Then saying, well, here, if we take that to an extreme and sort of scale it up, you can now all see that this would be bad. So let's reflect on what we're doing and maybe we can avoid going down the path of, you know, Brave New World or something.

• 19:37Speaker 01

  But this book, Deep Utopia, let me

• 19:42Speaker 01

  promote it a little bit to publishers. There she is. Yeah, there it is.

• 19:48Speaker 01

  It doesn't talk about that at all. It's like not about the practical problems between here. And it's rather like assume for the sake of argument that everything goes as well as it possibly could with the whole AI transition, et cetera. So we solve the alignment problem. We solve the governance problem to whatever extent it can be solved, but like no wars and no oppression, et cetera, et cetera. And in order to get to the point where you can then ask the question,

• 20:16Speaker 01

  what then, if we do end up in this condition where

• 20:20Speaker 01

  all the practical problems have been solved, what would

• 20:25Speaker 01

  what would we humans then do? Like, what would give us a meaning and purpose in life? If AIs and robots can do everything much better than we can do?

• 20:37Speaker 01

  And yeah, if we then attain this condition of technological maturity, that I think this machine superintelligence would relatively quickly bring about.

• 20:52Speaker 01

  There are kind of layers, this is like a, like an onion, you can sort of think about this problem at various levels. At the most superficial, you have this, oh, well, you know, the AIs will automate a bunch of jobs. And so then you'd have some unemployment, you know, and maybe you'd have to retrain people to do other things instead, just as everybody used to be farmer and now like almost all the farming jobs are gone, but people are still working. And so that's kind of, I know you might think layer one. And that's often where the discussion stops so far.

• 21:34Speaker 01

  But you can sort of think this through, from that point on, you say, well, like if AI really succeed, then it's not just some jobs, but basically all jobs

• 21:47Speaker 01

  that become automatable.

• 21:50Speaker 01

  You know, with a few exceptions that we can talk about if we want. And so you then would end up in this kind of post-work condition where humans no longer need to work for the sake of earning an income.

• 22:06Speaker 01

  So that's already a slightly more radical conception, right? It's not just that we need to retrain people to, you know, become whatever new weird occupations, but it's like, yeah, that that whole thing is

• 22:19Chris Williamson?

  The concept of occupations overall.

• 22:21Speaker 01

  Yeah, we would enter this condition of like leisure.

• 22:28Speaker 01

  But there are various groups of humans who live lives of leisure and we know we can look at this like there's like children, like young children before, like school is a kind of job for, but like before they start going to school. All right. Okay. So they, they don't work for a living. They, they are not economically productive. They still, in many cases, seem to have great lives. You know, spend all day playing and having fun and, you know, eating ice cream and, you know, all kinds of stuff like that. That could be the life. So that's like, you could look at other retired people, people born to great wealth or kind of monks and nuns. So like, anyway, so there are various templates of

• 23:13Speaker 01

  Othium that you could but, but that, that's still, I mean, maybe that's like the second layer of the onion, but it's still. Relatively superficial. So if that's where we stopped, then you would think, well, you know, then maybe we need to develop a leisure culture to kind of maybe change the education system. So rather than training the young to sit at their desks and receive assignments that they then work diligently on and hand in and do what they're told. This is a great training for becoming an office worker, right? But in this world, we don't need any office workers. So we can instead train them to develop an appreciation for the finer things in life, to practice the art of conversation, right? Develop hobbies, appreciation for art and literature and poetry and film and.

• 23:59Chris Williamson?

  Imagine how radical, imagine how radical that would be to have a school teaching people how to live well or find fun.

• 24:08Speaker 01

  Yeah, it would be. I mean, I'm thinking my school, I was kind of, I don't know how inspiring that would be if they had been trying to teach. But yeah, like in theory, at least you could imagine sort of shifting the culture from this focus on being useful and economically productive to actually living well.

• 24:34Speaker 01

  Which would make a lot of sense if that's like the condition we end up with. And I think hopefully that that would be great scope for.

• 24:45Speaker 01

  Yeah, like a much better type of human existence and which might then look back on the current era as like a kind of barbaric. Like the way we're thinking.

• 24:53Chris Williamson?

  Evil.

• 24:53Speaker 01

  Like, you know, eighty, like seventeenth, eighteenth century child labor in mines, working sixteen hours a day. They might think of our lives as correspondingly kind of blighted. But by kind of for many people going to like a boring job that gives them nothing other than a paycheck, but they have to do it because they need to pay the rent.

• 25:16Speaker 01

  But but I I think that

• 25:19Speaker 01

  there are like further layers to peel off here. So once you start to think through this condition of technological maturity, you realize that it's not just our economic labor that could be automated, but a lot of our other efforts as well. If you think what people do with their leisure when when they don't have to work for a living, like there's a lot of things we fill our time with that require some effort and investment. Uh, you may be mad and like, well, if you didn't have to work, you could do these other things like, you know, some, some people like, you know, go, go shopping. I I don't quite understand, but some people think that's like a

• 26:03Speaker 01

  wonderful activity. And but is that you think like in in this scenario, we have technological maturity, right? You would have recommender systems that could pick out something much better than what you would pick out yourself if you went. Then they would have like a detailed model of your preferences and be able to predict. And so although maybe you could still go shopping, you would know that at the end of of this three hour running around with with plastic bags, or you you'd end up with something that was worse than if you had just let your AI do the thing for you. It could select and also Bring it to your house. Break the bina, yeah, yeah, exactly. And so so you could put in this effort, but the end result is worse.

• 26:44Speaker 01

  Uh, and and it seems like that would put a little question mark over the activity. You could still do it, but would it still feel as fun and meaningful? Because I think a lot of uh, the structure, the activities we do now have the structure that you do X, like put in some effort and work in order to achieve Y, something outside the activity. But in in a technological maturity, there would be this shortcut to Y. And so that you could still do X, but there's like a kind of pointlessness, maybe like a shadow. Um, and um, a lot of activities, I think, have that structure. Uh, you mean, you could think of uh, like spending time, like child rearing seems like a worthwhile, important thing that gives a lot of people meaning.

• 27:27Speaker 01

  But if you sort of dissect it and look segment by segment, that's what it means. Like, is the changing of nappies really something that you think is intrinsically? If you had a robot, you could do it just as well. It'd be pretty tempting just to kind of uh, press the robot on button and it would do the thing. And um, and so, so that's a lot of that, I think, would, yeah, um, like, go away, I'd lose its appeal if, if, if there were these shortcuts. So, so that, that's like another layer.

• 28:00Speaker 01

  Um, but there are more layers, so you don't think, well, certain things, like, I mean, you um, you like fitness, and so you wanna, like, you can't rent a robot to run on the treadmill on your behalf, right? That's like, you definitely can't automate that, you think. But, well, at technological maturity, you could pop a pill that would induce the same physiological effects in your body uh, as like one and a half hour of uh, sweat and toil in the gym.

• 28:33Speaker 01

  Including the psychological effect of kind of feeling relaxed and energized and. So if, if that were the case, then yeah, then does it still feel like appealing to, to do the hard workout? If you could just achieve exactly the same results by a pill and?

• 28:52Speaker 01

  So, I think what you have is first, the kind of post-work condition we talked about earlier. And then there's this broader uh, condition of post-instrumentality, that all the things we do for instrumental reasons, with a few exceptions, but yeah, those uh, would also become obviated, it seems. And, and, and now we have this further affordance, which is a condition of plasticity, where we ourselves.

• 29:23Speaker 01

  The human body and mind, our psychological states, becomes a matter of choice. We become malleable. Like at technological maturity, you would have various,

• 29:35Speaker 01

  you know, the crude version might be various drugs with outside effects that have very tailored effects, but you could also imagine more direct kind of neural technology that allows you to have fine-grained control over your mental states and cognitive states and emotional.

• 29:50Chris Williamson?

  Permanently blissed out with some microscopic nodes that is able to manipulate your brain in some way or change your brain or all of your fears and anxieties are gone and all of your worries and concerns are gone and you're just at this sort of peak MDMA state. And then if you don't want that anymore, it knows and it's able to create a state that you couldn't even think about. And there are no constraints for you to do it either.

• 30:14Speaker 01

  Right, yeah. So that, that, that, that I think will become possible at technological maturity. And so then all these activities that you currently do, say, maybe you do them because it gives you joy and pleasure and happiness.

• 30:31Speaker 01

  They also would be unnecessary in that there would be this shortcut to, to joy and, and pleasure and happiness, that like the direct brain manipulation.

• 30:42Speaker 01

  So, so you,

• 30:45Speaker 01

  you, you, you have this quite radically different condition where it's, the world is solved in the sense of the practical problems have been taken care of, but also in the sense of maybe dissolved in that a lot of the fixed points and, and hard constraints that shape our current lives are kind of solved in this tech solution of, of technological advancement. And then, then, so that, then we kind of get to like really the heart of the problem that the book is trying to think about is like,

• 31:18Speaker 01

  in such a condition, what would a great human life look like? What would, could we actually achieve in terms of realizing human values if we had all of these affordances, all of these options?

• 31:32Chris Williamson?

  It's so strange. The thing that, reading the book, that stood out to me is how much of what we seem to value and take pride in are kind of like clever ways to deal with scarcity. And the fact that much of what we do is instrumental to striving and achieving some future goal, which requires effort. It seems like, in a sense, much of human philosophy and value is just negotiating with a world which is effortful and constrained. And we are trying to find ways cognitively to deal with this sort of

• 32:09Chris Williamson?

  pressure that we have to lean up against in order to cajole the world to deliver the thing that we want.

• 32:14Speaker 01

  Yeah. Yeah, these practical necessities have been with us since, I mean, through the entire history of the human species and indeed beyond that. It's like the human nature has kind of evolved and been shaped in a condition where this is always present. There are all kinds of things we have to do and cope with and struggle against.

• 32:38Speaker 01

  So it's almost like if you think of like a little bug that has an exoskeleton, right? That's, and then it holds the squishy bits inside together. But if you imagine removing the exoskeleton, there's like just a blob there. And similarly, the human soul might have as an exoskeleton, all these instrumental necessities. And that like evolution can just assume are present because they've always been. But if you were to remove those, then what becomes of the human soul and the human, human life? Does it become a kind of pleasure blob? Or is there something that could give structure to our existence even after these instrumental necessities are removed?

• 33:20Chris Williamson?

  So much of what we seem to take pleasure in as well is the absence and then satisfaction of some, some desire. There is a thing that we want, we don't currently have it, and then we get it, and then it gives us something. We work hard to Achieve a body that we're satisfied with. We are thirsty for a while and then we get a drink. We want to have sex and then we do. We are looking forward to having a child and then it's born. We are, all of these things are on the other side of something. And yeah, if like you do X to get Y, but you can just always immediately get Y without having to do X.

• 34:00Chris Williamson?

  It does ask the question of where does the absence, there are no longer any absences. There's this quote, something to do with, uh, in a perfect world, the only desired, uh, the only lack would be for the want of lack itself, which is this sort of, the absence actually makes the presence of something finally valuable. And if you don't have any more absence, then what does all of this presence kind of mean?

• 34:29Speaker 01

  Yeah, um, I, I, I think there might always be a whole bunch of absences in as much as human, if not human need, at least human desire, uh, or at least some human desires are kind of unlimited. Uh, it's maybe most clearly seen if you have two people who want exclusive possession of the same thing. Or two people, each of whom wants to have more than the other, like, I think two billionaires who want to have the world's longest yacht. And so one has like a hundred and fifty meter long yacht and then the other has a, builds a slightly bigger one that, that like is kind of intrinsically unlimited. Because, um, there's no way that they could both have everything they want.

• 35:14Speaker 01

  And so there might be a bunch of, um,

• 35:20Speaker 01

  desires like that, that are quite common, that could never be completely fulfilled. Or like, just imagine somebody who is, is like utilitarian and who wants there to be as many happy people as possible in existence. But say like however many they are, there could still be more. And so they would always prefer to have more resources.

• 35:39Speaker 01

  But even if there are some such desires, it wouldn't give these future utopians necessarily any reason for laboring or exerting effort, because there might just not be anything they could do themselves to increase the degree to which these desires are satisfied. I mean, the person who has a trillion dollar, maybe they would want to have two trillion dollars, but they can't actually make more money by working, because all the work is more efficiently done by machine.

• 36:13Speaker 01

  So yeah, even with unlimited desire, you might still have this condition that is both post-work and post-instrumental. Do you think that humans would be, run the risk of getting bored in a utopia? Not if they didn't want to, at least if you, by boredom, refer to a subjective state of, I don't know, like some kind of restless, discontented, uneasy feeling of having hard, like having a difficulty keeping your focus or like, so that certainly would be amongst

• 36:49Speaker 01

  the things that could trivially be dispelled through advanced neurotechnology. I mean, you already have like drugs that could do it for a limited period of time now, right, with side effects, and then they wreck your body. But like, it's easy to imagine how you could just have better versions of that. That would make it possible for you to always feel extremely interested and excited and motivated.

• 37:12Speaker 01

  And in fact, some people have, I mean, there's a lot of variation amongst humans. And I mean, I have a friend who tells me he's never bored, and I believe him. I've never seen him bored. He's kind of interested in everything except sport. And like, he writes papers on all kinds of different weird topics, and he's like just constantly excited about learning new things. And you can have a conversation with anybody about anything, and he's like really, so it's an existence proof. It's possible to be that kind of being, and in the future, we could all become such beings if we want to. So subjective boredom would be like trivially

• 37:55Speaker 01

  easy to dispel under this condition. Now, It is possible also to have a more objective notion about boredom, or maybe we say boringness to refer to this objective notion, which is the idea that certain

• 38:14Speaker 01

  activities are intrinsically

• 38:19Speaker 01

  boring, like meaning maybe that it is appropriate to feel bored if you were spending too much time doing them.

• 38:28Speaker 01

  It's like kind of maybe an open question whether this notion of objective boringness makes sense, but you might think of like say, say, counting blades of grass on a lawn, like suppose you had a a being who found it extremely fascinating and like and never-ending source of joy to just count

• 38:53Speaker 01

  and recount the blades of grass on a college lawn somewhere. You might say that although subjectively he is not at all bored, that objectively what he's doing is boring, it's like no variation or significance or development. And the appropriate attitude for somebody to have if they were spending their whole day doing that would be to be subjectively bored.

• 39:18Speaker 01

  If you have this notion of objective boringness, then it becomes a much less trivial question to ask in this hypothetical condition of a soul world.

• 39:31Speaker 01

  Like, would it be possible for us to avoid objective boringness? Like, yeah, we certainly we could like engineer ourselves, we always felt interested in what was going on, but would we be doing anything inappropriate? Like, would our circumstances be such that

• 39:48Speaker 01

  the appropriate attitude would be to be bored? And so

• 39:53Speaker 01

  there's a big discussion about this, but yeah, so and I think it's

• 39:58Speaker 01

  I think there are certain forms of interestingness that we might run out of.

• 40:06Speaker 01

  For example, you might think it's especially interesting to, I don't know, be the first to discover some important truths, like Einstein discovering relativity theory might be like a kind of paradigm case of like an extremely interesting discovery and experience.

• 40:24Speaker 01

  But it's plausible after some while that most fundamental important insights about reality that we could have, we already have had. And in any case, the machines will be much better at doing the discovery than than we would be. And so we would kind of run out of the opportunity to achieve that kind of interestingness in our lives.

• 40:51Chris Williamson?

  Yeah, I mean, so much of what humanity's done is been chasing down answering big questions. So where do we go when all of the big questions have been answered?

• 41:03Speaker 01

  Yeah, I mean, I think fortunately,

• 41:07Speaker 01

  for the most part, it's not what we're actually doing in our lives. I mean, most people are not spending most of their time trying to chase down the answer to the big questions, right? Like, most of the time, we're just going about our daily business. And you could make the case that already, I mean, if you really looked at it, I mean,

• 41:30Speaker 01

  from a kind of unbiased outside view, like if the alien superbrains came to Earth and thought, okay, so look, these guys are worrying about losing out on what's interesting in life. Well, let's look at their current life and see how interesting there is.

• 41:46Chris Williamson?

  Ah, yes.

• 41:47Speaker 01

  Like, how many times did this guy brush his teeth? Okay, well, 40,600, like, how interesting is it to brush your teeth for the 40,760th time? And then, all right, so he commuted into the office and then, you know, he ate a steak. Okay, I mean, how interesting is it to do that again and again and again?

• 42:11Chris Williamson?

  Yeah.

• 42:11Speaker 01

  And even the big highlights in our lives, like, I mean, maybe they are like really novel and exciting. If your scope of evaluation is a single life...

• 42:21Speaker 01

  Like the first time you see your own newborn, like it just like it just happens once, right? Like so a few of those, but if you zoom out and look at humanity, it's kind of, well, it's already been done. You know, tens of billions of times. Um, you know, like how different is this particular newborn from all the other newborns? And so depending on how you look at it, you might kind of either think that we are already like at the very low rung of the ladder of objective interestingness. Or if you sort of shrink the focus of evaluation enough to a single life, or perhaps even just to a single moment in a single life, then yeah, then then there is like more novelty, but also opportunities for the same kind of thing to happen in in utopia.

• 43:05Speaker 01

  Like if if if you're just looking at the most interesting possible moment and you don't care about whether similar moments have existed before or after.

• 43:17Speaker 01

  Then you might think the average human moment of awareness is very far from the maximum interestingness.

• 43:24Chris Williamson?

  What? What do you think would happen to religion? That's obviously a place that an awful lot of people take their meaning from currently. Is there a place for religion in a deep utopia?

• 43:36Speaker 01

  Yeah, so this is a one of the things that plausibly survives this transition to a solid world, like which could remain

• 43:48Speaker 01

  highly relevant. If even like if we had all this fancy technology. And it might constitute a bigger part of people's lives and attention than than it does today. Because there would be fewer other like distractions, if you want.

• 44:10Chris Williamson?

  What else? What are the other areas that are uniquely human or that would survive this transition well?

• 44:16Speaker 01

  Yeah, so I mean, you can kind of build it up, starting with the most basic value, perhaps, which is just this sheer subjective well-being, pleasure, enjoyment,

• 44:33Speaker 01

  which obviously would be possible to achieve in utopia, and not just achieve, but like you could have prodigious quantities of this bliss.

• 44:44Speaker 01

  And so that's

• 44:46Speaker 01

  intellectually, not maybe super exciting to discuss at great length, but I think actually super important. Like, it's easy to dismiss, "Oh, these are like some sort of junkies, just having their like heroin drips." But the key question here is not like, "How exciting is this future?" Or "How admirable is it from our point of view?" As if we were sitting in the audience, like evaluating a stage play. Like, that's one perspective, and then we want a stage play with a lot of drama, and suffering, and tragedy, and overcomings, and heroism, and, right? But the question here is, which future would you actually want to live in?

• 45:24Speaker 01

  And there, like, one of very great levels of subjective happiness and well-being might, that might be the most important thing about the future impact. And you could definitely have that in, like, extreme degrees. So that's like, it's worth making a note of that, like, and so let's put that in the bank. At least we could have that, and that's already possibly, like, according to some people, it's the only thing that matters. Like, if you're a hedonist, a philosophical hedonist. But for most other people, it's at least one of the things that is important, even if not the only thing that is of value. So

• 46:05Speaker 01

  so that's the first thing. Then you can add to that

• 46:09Speaker 01

  experience texture. So it's not the case that you could only have subjective well-being, you could attach that to some

• 46:19Speaker 01

  Intricate, complex, uh, mental state that relates to some important objects. So for example, you could experience the pleasure not just as a sort of unanchored sensation of well-being, but you could attach it to, say, the appreciation of aesthetic beauty. Um, or appreciation of, you know, great truths or great literature or contemplating the divine, and that that's what you derive the pleasure from. So your conscious state is one of insight, let us say. Uh, or or understanding or appreciation of things that deserve to be appreciated and understood. Um, so, uh, some people think that that is also a locus of value.

• 47:01Speaker 01

  Not just the hedonic, like, the scale of whether it's like plus 10 or minus 10, but having plus 10 whilst you you're like, understanding or seeing or appreciating something that is like actually lovely and worth understanding or profound, makes that a more valuable condition. So you could have that.

• 47:21Speaker 01

  Then, um, if we go to some of the other values that seem more at risk, like purpose. Um, you could have, you could certainly have, um, artificial purpose.

• 47:36Speaker 01

  So you could set yourself goals.

• 47:42Speaker 01

  In utopia.

• 47:45Speaker 01

  In order to then enable the activity of trying to achieve them. Like playing a sport against another person? Yeah, so so games is a a a paradigm example of this. In today's world, you like set yourself some arbitrary goal, like maybe to to get the golf ball into a sequence of 18 holes using all the club. And there's no other reason for why you'd need to achieve this goal. Right? Other than to enable the actual activity of golf playing. Um, and that that could become a much larger part of the utopian lives, various forms of game playing, like you could make all kinds of new, much more sophisticated and immersive games. Uh, alone or with other people that that involves setting yourself

• 48:29Speaker 01

  um, sort of arbitrary challenges or at least semi-arbitrary challenges. Simply in order to then create an opportunity for the activity of striving to achieve them.

• 48:40Speaker 01

  And yeah, so to enable play, we kind of like deliberately limit the means available to you to achieve this arbitrary goal. So just as in golf, like, yeah.

• 48:56Chris Williamson?

  Because if you were in the post scarcity world, you press the button and there's never any doubt about whether or not hitting the ball goes into the hole, which makes the hitting of the ball completely arbitrary.

• 49:06Speaker 01

  Yeah, and uninteresting and maybe objectively boring, but you could just set the goal to achieve this sequence of outcomes, the ball falling into the different holes whilst also not availing yourself of various shortcuts, that you could sort of make this more complex goal of achieving X while only using means.

• 49:28Chris Williamson?

  Correct. There needs to be some constraint, which then gives a degree of satisfaction when you achieve it.

• 49:33Speaker 01

  Yeah, yeah. And like either to give you the satisfaction, I could achieve the satisfaction just directly through the newer technology. But if you also, in addition to the pleasure, wants to have the experienced texture and you want to have the sort of effortful activity and striving, then yeah, you could achieve that by having these artificial purposes.

• 49:52Chris Williamson?

  It kind of feels to me like you very quickly keep coming back to the same question of is there a quicker route to achieving the outcome that I'm trying to achieve here? I've had it in my head since you were talking about this and since I read the book about churning butter. So there's very few people that would look at the butter that they use now and think, I know that it's here and it's convenient and tasty and does what butter needs to do by being lovely on bread or whatever. But I feel like it would have been more meaningful to me if I'd gone out into the field and got the cow and done the thing into the bucket and then churned it and then got it and then put it in the fridge and all of those steps. So we can see, and you know, there's kind of this.

• 50:38Chris Williamson?

  Inherent sense, this sort of naturalistic fallacy that this is taking us away from what it means to be human, that the set point that we have grown up in, this is, it's a misalignment evolutionarily, there's something sacred about the process of being human, it imbues you with meaning to go through the challenge and the struggle beforehand, but there's very few people that would make that argument about churning butter. And when you think, okay, so if you are happy with more convenient butter, I think that something right now, which is assumed by most people to be natural, but in future will be looked at as barbaric, will probably be driving our own cars. I think that in 50 years' time, 100 years' time, it'll be like if you looked at

• 51:21Chris Williamson?

  someone riding a horse down the street now, and you go, I mean, isn't that so quaint? And it's wild that people used to do that. That was the way that they got around it. You know, they had to have these special people in New York that would sweep up all of the muck from the street. This entire industry builds around horses. So again, right now, something that we can almost begin to see the transition of, we're about to let go of this thing, which is a less efficient, less safe, more effortful way of getting us from A to B. And yet, some people, I love driving, I take great pride in driving. Some people even compete in it, you know, F1 is an entire competition around people that are driving. So you can see different frontiers of human endeavor

• 52:01Chris Williamson?

  being eroded away by technology, whether it's from churning butter or driving a car, and you just continue to slice that ever more thinly, all the way to why are you here? What are the sort of relating to other people, having to get yourself out of bed and move yourself down the stairs on the morning. Each one of these different things begin, could begin to look, okay, isn't it cute that people used to, you know, like, pick themselves up out of bed and put their own clothes on and walk downstairs and brush their teeth? And then you ask yourself the question, well, if everything is open to you and you can manipulate your own internal state, why not just spend the rest of your life counting table legs or blades of grass?

• 52:44Speaker 01

  Right. Indeed, yeah. So you are forced to confront these fundamental questions of value in this condition.

• 52:56Speaker 01

  What things are you doing for the sake of something else versus what things are you doing truly for the sake of the activity itself? So even the guy now who maybe likes to do their own butter, there is a question of is it because they intrinsically value the activity or is it maybe because of the pleasure they get out of it? Or because of the way it teaches them about their own body and about the cow and the physical objects and puts them in touch with that? That's a kind of extrinsic element, but yeah, so these things that is currently, we can conflate them because in reality, the only way maybe to get various kinds of pleasure is to dive into activities and give it your best and then you get the satisfaction.

• 53:40Speaker 01

  Yeah, we can't, we can't like separate these today, but in this hypothetical condition, they can be separated and then you do have to ask the question of what precisely is it that you actually value? So this conception, I mean, so for me, it's like interesting because I think there is a real chance that if things go well, we might actually end up in something like this condition with the whole machine intelligence revolution, et cetera. But even if you thought that was not going to happen, you could view it as a kind of philosophical thought experiment. Just like physicists build big particle accelerators at which they smash atoms together at like

• 54:22Speaker 01

  extreme energies to sort of see what their constituents are. And then you can assume that, you know, if there are quarks, you know, when you smash the particles together in CERN, that maybe there are quarks in other matter and you can kind of learn from, like you expose basic principles by looking at extreme conditions and extrapolate and think they might be there all the time, even though we can't see them. Similarly, with human values, if you kind of smash them into one another under this extreme condition of a solid world, you can study their constituents. And then you might think that, well, in our ordinary lives, maybe those same constituents are there, they are just kind of invisible top because they are hidden by all the kind of practical necessities.

• 55:06Chris Williamson?

  What are the implications if humans live for a very long time? Does anything change there?

• 55:14Speaker 01

  Yeah, so certain values are more jeopardized by extreme longevity. For example, interestingness, as we discussed earlier.

• 55:24Speaker 01

  If your notion of interestingness involves the idea that something has to be novel to be interesting, like it's uninteresting to just do the same thing over and over.

• 55:34Speaker 01

  And if the domain within which it has to be novel is your own life,

• 55:40Speaker 01

  as opposed to, say, the world as a whole, or your species, or the current moment. But if the relevant locus of evaluation is a human life, then the longer the human life goes on, the harder it is to do important things for the first time.

• 55:57Speaker 01

  I think we already see this in our current lifestyles. If you think about what happens in the first year or two, like there's some pretty, pretty big, like, epistemic earthquake. Like, you discover there is a world out there, like, that's a big discovery, right? Oh, it has objects. Like, the objects remain there even when I'm not looking at them. Like, there are other people there?

• 56:26Chris Williamson?

  My friend.

• 56:27Speaker 01

  Like, people in the world? Like, imagine just discovering for the first time that there are people.

• 56:31Chris Williamson?

  My friend.

• 56:32Speaker 01

  And then you discover that you have a body.

• 56:34Chris Williamson?

  Yeah.

• 56:34Speaker 01

  Like, wow, I have a body.

• 56:35Chris Williamson?

  And you're separate from mum and dad, and you can communicate. Yeah, my friend described a, um, his son was born during Covid, and I think for the first, maybe, year of his life, or something, had only seen four people. He'd seen, like, mum, dad, grandma, and nanny, or something like that. And then apparently, one day, he saw a fifth person, and it fucking blew his mind. He was like, what, there's more than four?

• 57:04Speaker 01

  Right. And then, yeah, you discover that you can do things, move your body, like, and then like later in life, it's like, oh, what happened this year? Well, we got, we got a puppy, like, we bought a caravan truck, like, it's not really at the same order of magnitude in terms of like how much it reshapes your view about reality. So I think there is already within the human lifespan, like a kind of rapidly diminishing, if you measure interestingness in a certain way, where it's kind of the delta between your previous, like, conception of the world and what you could do and what you're able to do after the event that is then interesting, right? So there is another conception of

• 57:48Speaker 01

  interestingness where it's less the, the, the rate of change and more sort of the complexity of what you're engaging with at a particular moment, in which case maybe the level of interestingness of a day of the typical adult, by that metric, might be higher than that of an infant, because like you have all kinds of complicated things going on at work and relationships.

• 58:08Chris Williamson?

  Right. So rather than having big questions being answered, you have increasingly dexterous small questions with more magnification and you can sort of see them with more complexity and you derive some pleasure from that. One of the things that go in my head, if, if humans do live for a much longer time, you're going to be able to continue producing humans, that will have to be in relation to the increase in computing power. So there'll be this kind of Malthusian tug between how much computing power have we got to be able to support how many humans and which can move more quickly. Have you considered this sort of tension between the two things?

• 58:52Speaker 01

  Yeah. So, so in the long run, I think economic growth becomes really a consequence of growth through space, the acquisition of more land, as it were, by space settlement. Like, once you have achieved technological maturity, you can't have economic growth by, like, inventing better methods for producing stuff. And you also probably can't have more growth by just accumulating capital assets and machines, because you already like, like, built all the machines that. It results in optimal productivity for the volume. So ultimately, the limiting constraint become what economists call land. But it basically means those resources that you can't make more of.

• 59:38Speaker 01

  And so in the long term, you could imagine human civilization kind of expanding through space. But there is a limit to that, which is the speed of light. So if you have a sphere, you know, with Earth at the center, maybe, and then growing at maximal speed in all direction at some fraction of the speed of light, that the volume of that would grow polinomially.

• 1:00:00Speaker 01

  But population could go exponentially. It could like double every generation or, you know. So in the long run, an exponential overtakes a polynomial. So at some point, you would need to moderate the rate at which new beings are brought into existence. If you want to maintain a sort of above subsistence level of welfare.

• 1:00:19Chris Williamson?

  That's a really interesting point that I hadn't considered. So you can have a solved world in which almost all problems have been defeated, but there are still some constraints. Speed of light is one of them. What are some of the other constraints that a utopian world would encounter?

• 1:00:37Speaker 01

  Yeah, so there's a bunch of basic physical constraints, like, you know, the speed of information processing.

• 1:00:44Speaker 01

  The amount of memory you can store, like the size of a mind that is integrated. Like, if you make a mind much bigger, you know, than a planet, then you get conduction delays, like it just takes time for one, like what happens in one part of the mind to kind of communicate to what happens in a different part of the mind. So either you have to run the mind much slower

• 1:01:10Speaker 01

  or you have to keep the mind relatively small.

• 1:01:14Speaker 01

  There might be, I mean, we are hoping not, but you could imagine if there are other alien civilizations out there. There is like the potential for all kinds of competition and conflict.

• 1:01:25Speaker 01

  So, yeah, so there's like a bunch of of those external physical constraints that I think define the ultimate envelope of what could be done, but it looks like.

• 1:01:38Speaker 01

  Um, the space of possibility is very, very large compared to our current human vantage point. Uh, so like you could have, you could maybe not have like immortality if that requires not dying, like infinitely long time looks impossible in our universe. Uh, like eventually information processing threats will, uh, you know, if, if not before, then with the heat death of the universe. Um, which, which is kind of significant because from a theological perspective, like whether you live for 80 years or 80 million years, like it's all kind of really a blink in the eye of eternity, you could argue. And so it doesn't really change fundamentals. But from, from the kind of parochial perspective

• 1:02:20Speaker 01

  of a current human life, you could certainly have extreme longevity and extreme amounts of wealth and extreme amounts of most other things. Um, but, but still there are limits and those limits would be like relevant, uh, in various ways. What about moral constraints? Would there be any? Yeah. So this is another like more subtle, but potentially very important source of constraint. Um, so, um, like what's the easiest way to like, I mean, so, so, I mean, some people have thought, for example, there is like, it's immoral to enhance humans biologically. Like it's not a very popular view these days, but during President Bush, uh, he had a council

• 1:03:03Speaker 01

  on bioethics that he set up and populated with a bunch of bioconservative thinkers and, and they were trying to argue that it's somehow it's a violation of human nature or something, uh, to try to enhance, uh, humans. Um, so like distinction, like therapy, like medicine curing a disease, fine, but like trying to slow the aging, bad because it kind of, you know, uh, it, it, and it's a little like, once you start to think about it, it's really hard to make out that distinction. Like you think. Uh, like, you know, genetic therapy to make you smarter, bad, but education, good, even though it, hopefully it makes you smarter and like, i- it becomes problematic.

• 1:03:45Speaker 01

  But if you did have that view, like, then there would be a whole bunch of possibilities that would be cut off if you just couldn't change what, like, the basic physiology of what we have. And you were confined just with like moving things around in the external world to try to cheer you up by, like, having a really nicely decorated room or something, or like a, like, there's only so much you can do to affect your inner well-being if, if, if you can only affect it by having sort of nice visual stimuli and, uh, nice acoustic waves going into your brain, if you can't actually change the thing in between the, uh, between the ears and behind the eyes. Um, but that are more potentially some other, like, interesting ones that, um, if somebody,

• 1:04:30Speaker 01

  uh, suppose somebody had like some preference to, um, um, have another person relating to them in a particular way, um, like the experience of, of being loved by a particular other kind of person, then it might be that the only way to generate that experience, uh, fully realistically, would be by instantiating that other person. Um, and if that other person then, like, presumably would have moral status, there might be all kinds of ways of treating that other person that would be wrong. So a moral constraint might then limit the kinds of experiences you would be able to have.

• 1:05:12Chris Williamson?

  Well, with another person, yeah, I suppose, as soon as you involve somebody else who has moral consideration, that changes, that changes quite a lot.

• 1:05:21Speaker 01

  Yeah. And in fact, I think that is, so we discussed these artificial purposes, that they're sort of like, you could create games and set yourself goals, like, that's one type of activity. I think there are also, uh, a possibility of a bunch of natural purposes remaining, like, purposes that would call upon us to make various kinds of efforts, not just because we create random goals for the sake of having something to do, but that kind of exists independently of us. And, and a lot of those would derive from this kind of interpersonal entanglements and, and various kinds of cultural entanglements, um, where...

• 1:05:59Speaker 01

  Like, I mean, to, to, to take the sim- the most reductionistic case of it, which is not so inspiring in its own right, but you could imagine more natural versions of this. So suppose you have person A and person B. And person A wants person B's preferences to be satisfied. Like, they care about person B and wants person B to get what they want. And then if person B happens to want person A to be doing something on their own steam, then the only way that person A can achieve their goal of satisfying person B's preferences is by themselves doing this thing. Like, they couldn't, they could have a robot do it, but that wouldn't satisfy person's B preferences. So from the vantage point of person A, they now have reason to do this thing.

• 1:06:43Speaker 01

  It's not an arbitrary goal they set themselves. It's the only way they could possibly achieve their goal of satisfying person B's preferences. So, so in, in this kind of, like, it seems a bit hokey and artificial, this particular case, but you could imagine more subtle ways of this, where there is like a tradition that you feel a commitment to and that you want to honor. And part of that tradition is that you, you know, you engage in certain kinds of practices, you refrain from certain kinds of shortcuts, you respect other people's preferences to various degrees because they are, yeah, you want to honor them. Um, and, and so there would be a bunch of stuff then that maybe you need to do yourself. And, and you can't outsource them.

• 1:07:27Chris Williamson?

  So you've managed to, over the last decade, straddle all the ways it could go wrong, all the ways that it could go right. Toby then sort of bifurcated that down the middle with The Precipice, his book. And he's got this analogy where he sees humanity being sort of walking along a precarious cliff edge. And if we fall, then everything's kind of fucked. Uh, and if we make it on, then there's this sort of beautiful meadow on the other side. How important or critical do you think the current moment is in humanity's future? What's the, how long is the precipice in your, uh, perspective?

• 1:08:06Speaker 01

  Yeah, uh, I think, I mean, it is weird because it looks like we are, uh, very close to some, like, key juncture.

• 1:08:17Speaker 01

  Which you might think is prima facie impossible. So there have been thousands of generations before us. Right. And if things go well, there might be millions of generations after us or people living for like cosmic durations. And like out of all of all of these people that you and I should happen to find ourselves just next to this critical junction where the whole future will be decided. It is striking like that. That seems to be what this model of the world implies. And maybe that is like an indication that there is like some things slightly puzzling or impossible about it, that there's maybe some more aspects to understanding our situation

• 1:09:00Speaker 01

  than is reflected in this naive conception of the world and our position in it. And you might speculate what that I mean, I have this earlier work on the simulation argument and stuff like that. But but if we take the sort of naive view of reality, then it does look like, yeah, my metaphor would maybe more be like a balance beam where like a ball is rolling down like a thin beam. And like the longer it rolls, the more likely it will be to fall off the beam, but it could fall like on one side or the other, and that's hard to predict. But yeah, I think it probably will fall off like that.

• 1:09:40Speaker 01

  The idea that the normal human condition, as we now understand it, will just continue for like hundreds of years. I mean, let alone like hundreds of thousands of years. That seems to be like the kind of vague idea that a lot of people have. It just seems like radically impossible to me.

• 1:10:00Chris Williamson?

  It would be unlikely in your opinion that in a thousand years' time, 5,000 years' time, human existence will reflect what our normal sort of day to day is now.

• 1:10:13Speaker 01

  Yeah. And I think the only plausible ways for that to happen, you could like. Create some scenarios like one would be if we do sort of have some massively destructive event that knocks us back to the Stone Age or something and then maybe by 500 years we would have climbed back up again to something resembling the current human condition. But then in that scenario, we would have spent most of the intervening time in a rather different condition. Like or another might be if you get some very strong global consensus of some particular orthodox.

• 1:10:43Chris Williamson?

  Moratorium of something, something.

• 1:10:45Speaker 01

  Yeah, like the kind of bio, yeah, bioconservant and then like we start to like ban all kinds of different technologies. So you get like a kind of some sort of bureaucratic sclerosis or deliberate decision to just say, all right, we've come to this point, but let's not. So there are various scenarios in which something like that. But if the basic sort of scientific and technological push forward is allowed to continue, then it does look like we are sort of very near developing a range of transformative technologies. So AI being kind of the most obvious of those. But if it weren't for that, then I think like synthetic biology will create a bunch of other possibilities and then nanotechnology.

• 1:11:29Speaker 01

  And so I think even if AI were somehow like, if you just pretended that wasn't there, I still think we would be in for profound transformations.

• 1:11:38Chris Williamson?

  That's interesting to consider that if technology moves forward, even at a slow pace, even if it was to drop by a really significant margin from where it is now, given long enough time, you end up with a radically different world in either a way that you intended or a way that you didn't intend. And the way that you didn't intend is probably going to be pretty bad and the way that you intended hopefully is going to be the one that is that is pretty good. Either way, you end up with a radically different day to day experience for most humans.

• 1:12:07Speaker 01

  Yeah, I mean, yeah, I'm not so sure about the first part. Like if it's something we didn't intend, then it would almost certainly be very bad. I'm not sure. I mean, you might think like the world that we have currently ended up with, I'm not sure whether you could say like that's what. People a thousand years ago intended, I mean, they they they they might kind of in fact be quite shocked about some of our habits these days like. But it more just sort of happened as a result of a bunch of different people going about their business and pursuing various local aims. And then at a systemic level, eventually, you know.

• 1:12:46Chris Williamson?

  Unintended consequences can still be positive.

• 1:12:49Speaker 01

  If we are, yeah, if we're a lot, I mean, I think I think like. The degree to which the future depends on our intentions is possibly quite limited. I I there there are sort of bigger dynamics at play and we we barely even understand what they are. We don't really know what we want uh at at a big scale, like most people have their hands full just thinking about like the next week. And you know what what to do if their boss don't like them at work or their partner has a like like this. This is what fills human life and then trying to get the head in the low like there's very little thinking by anybody really like trying to where where should humanity be? Going in a million years, like what's the optimal trajectory?

• 1:13:29Speaker 01

  I I think that we could do with a little bit of more thinking about that, but um it's not like the primary uh shaper of of the direction of the big ship of humanity is.

• 1:13:40Chris Williamson?

  What have you been most surprised by over the last ten years when it comes to AI development?

• 1:13:49Speaker 01

  Um, I think just how anthropomorphic the current generation of AI models are. The idea, first of all, that they are sort of almost human level and that they can talk in ordinary language.

• 1:14:06Speaker 01

  Is is already kind of interesting and then but then that they even share some of the like. Uh quirks and psychological foibles of humans is like if if you if you're ten years ago, you would have come and said, well, we're going to have these AI systems. You know, they can do all of these things, they can like program and write poetry and but if you really want them to perform at their best. Uh you you need to give them a little pep talk when you ask them a question, you're going to say, think step by step. This is really important, I might lose my job if you get the answer on. And then they perform a little bit better than if you just asked them the question like you'd like people would have thought you would completely lost your marbles, right? And yet that's where we are today, so that's surprising.

• 1:14:53Speaker 01

  I think less surprising, but still interesting is the degree to which development so far has been continuous, like rapid, but incremental, like a sequence of steps, each of which has significantly but incrementally improved on the previous step. And the degree to which the progress is quite tightly coupled to the scale of compute being applied to this. So you have this big compute hypothesis, which is like basically that the most important determinant is not the particular architectural features of your model, but just the amount of compute you use in training and like the amount of data.

• 1:15:36Speaker 01

  And like you get, you know, performance in proportion to like how much, how many dollars you spend on training it basically. That's like too crude. You also need some skilled engineers, but we are kind of closer to that being the case than one would maybe have expected in the median scenario ex ante, where you might instead have thought, ah, we're gonna fumble around until we find this clever algorithmic hack and then suddenly it's gonna explode.

• 1:16:02Chris Williamson?

  It's gonna open something up.

• 1:16:04Speaker 01

  Yeah, yeah. Whereas it's like more like you just scale it up, it works better, scale it up more, it works even better. Now, it is still possible that at some point that like some little last missing bit will fall into place and we could still get an intelligence explosion in those scenarios. So we shouldn't like over index on what we've seen so far, but it's still interesting.

• 1:16:25Chris Williamson?

  Does that change your perspective on what is more or less likely from a takeoff scenario, from how superintelligence could come about, stuff like that?

• 1:16:36Speaker 01

  Yeah, I think it makes it somewhat more likely that there will be political forces at play that like it's like when things happen more gradually, it's easier for the public and for policymakers to realize what is happening and to sort of try to change it. And so we already see sort of at the geopolitical level with like the chip export restrictions and more recently.

• 1:17:05Speaker 01

  Reporting requirements for training like models using more than 10 to the power of 26 flops. And there might well be more if we continue to see sort of increasingly powerful AI systems over a sequence of several years, that there might be time for more actors to kind of try to exert influence of this than if it were just some lab one day that like stumbles upon like the key missing thing like with a computer in their basement and you go sort of overnight, then it would be more likely to be like an isolated thing where just a few people were having their hands on the tiller.

• 1:17:40Chris Williamson?

  Have you got any idea which scenario you think is more optimal?

• 1:17:48Speaker 01

  Um, it's really hard to say.

• 1:17:54Speaker 01

  I think it seems probably better if whoever develops this technology first has the option when they are like, like, starting to develop like true superintelligence to go a little bit slow in the final stages, like maybe to pause for a half a year or something. Rather than, okay, now we've got it figured out and then immediately cranking all the knobs up to 11. Because maybe there are 19 other labs, you know, racing to get there first. And whoever takes any precautions just immediately become irrelevant and fall behind. And the race goes to have risk like most gung ho are willing to take the biggest risks. That seems like an undesirable scenario.

• 1:18:36Speaker 01

  And so having some ability perhaps for the frontier labs to coordinate it or unless one is already naturally significantly had, it may be that a small set of leading labs should at some point be able to synchronize. That that could be desirable. I, I think it's very unlikely, but less unlikely than a couple of years ago that we could end up with some kind of permaban like on AI.

• 1:19:06Speaker 01

  I think that would be undesirable. I think ultimately, this is like, it's a portal through which I think humanity will need to passage to the future. But we should recognize that there will be significant risks associated with this transition.

• 1:19:22Chris Williamson?

  And the slower as well that this happens, I suppose, the more opportunity there is for political, policy, human fuckery to get in and coerce and cajole.

• 1:19:34Speaker 01

  Yeah, so it's like, it's very much a double. On the one hand, yeah, you do want like, it's kind of uncomfortable either way. Like, so some random person in some lab are just going to control the future. That sounds like pretty scary. You want definitely like adults to oversee this, right? But then you think the other end like, wow, you know, all the security establishments of governments around the world. Like, and I know everything like, is that like a much more comfortable situation? Where they like the military, not just one military maybe, but like, uh, like, and then you get all, so either way, I think it's,

• 1:20:07Speaker 01

  it's a little bit disconcerting. So it's, I'm not, um, I don't feel super comfortable. Like, I don't have a very strong view at the moment as to what is the most desirable trajectory. And it might be anyway, something we don't have super fine grained control over. I think one can try to notch things on the margin, like towards a more cooperative, uh, inclusive and friendly and thoughtful, uh, trajectory. I think that that seems good to do. Like to try to encourage this idea that the future could be good for both humans and for digital minds and for animals and for as many people as possible. Um, and there really is that potential there, like the upside is so enormous that that could be plenty for not just one value to be realized,

• 1:20:52Speaker 01

  but for a whole range of different values and perspectives. So our first instinct, I think, should be to seek these kind of win-win positive sort of outcomes. And then like, if at the end of the day, there are also some

• 1:21:06Speaker 01

  irreconcilable differences, we'd have to strike some compromise there. But there's like so much you can do before you get to that point, that it would be tragic. If you just kind of skipped over all of that and let's get to the point where we can fight about something like that, that would just be a big tragedy.

• 1:21:24Chris Williamson?

  What is the current state of AI safety in your view? Obviously, 10 years ago, conversations about alignment and takeoff scenarios and all of the rest of it was obscure Reddit threads and a couple of people in some weird message boards.

• 1:21:44Chris Williamson?

  Is it overfunded, underfunded, over-resourced, under-resourced? Where should people's attention be placed at the moment?

• 1:21:52Speaker 01

  Well, there is a lot. There's a lot more talent in the field now. And I mean, a lot of the smartest young people I know are going into AI alignment and working on it. And all these leading labs have research teams. As I said, it's probably still under-resourced. I think it looks more like talent constraint at the moment rather than funding constraint. But, you know, to some extent, funding can help. There are some questions about whether alignment work spills over to capability progress, like some of the things you would want to do for alignment, like better methods of

• 1:22:32Speaker 01

  interpret what is going on inside a little mind and figure out why exactly why is it behaving the way it is? That would be useful for alignment. But it could also shed a light on how to sort of, you know, what's limiting performance and how to boost it. So it gets pretty complex. I think some other things like better cybersecurity in the leading labs to make it less likely that the weights of these models just get stolen, that could maybe be helpful.

• 1:23:01Speaker 01

  And yeah, but I think like more work on alignment seems positive. Having some ability for leading labs at a critical time to go a little bit slow seems positive. I would be worried.

• 1:23:17Chris Williamson?

  Would that require coordination between multiple labs in order to be able to do that? Because you do have this sort of first past the finish post.

• 1:23:25Speaker 01

  Yeah, it depends on like, I think the one older idea, and which might still be relevant, is that maybe you would have one lab or, you know, whether it's one country running one lab or one just private industry lab or whatever it is, but like one would have some. Lead over the like just naturally some like one lab might just be a year or two ahead because they started earlier, were more lucky or had better talent or something. So then that would create an opportunity for this leading lab to slow down for a year or to however long their lead was, right? Without falling behind. And it might be desirable if rather than having like a super competitive race, you had a little

• 1:24:07Speaker 01

  and then that kind of pause would be self limiting, you see, like because once they have paused for two years, there would be another lab kind of catching up. And then maybe they could pause too, but you would have to like make an increasingly strong case for pause as more and more independent actors became capable. So it would be a pause that could exist and it would eventually expire. And exactly when it would expire would depend on how strong the argument was for AI risk. And that would it create a much lower risk of ending up with like a kind of permaban where this technology is never developed. Whereas if the path to getting the ability to pause for a year is to say, set up a big international regulatory regime

• 1:24:51Speaker 01

  or like creating a lot of stigma around AI research, like developing some mass movement like smash the machine type of thing. Then that's much more likely to spill over into something that then become a permanent orthodoxy or a regulatory apparatus that just has like an incentive to perpetuate itself. So more worrying from that respect.

• 1:25:11Chris Williamson?

  How impressed have you been with the power of L L Ms? Do you think that they are going to be the boot loader for what we need from a super intelligence perspective? Or is this?

• 1:25:25Chris Williamson?

  Have you got limited hopes for how far they can sort of climb functionally?

• 1:25:31Speaker 01

  Well, we haven't yet seen the limits of what one can do when scaling these. I think these transformer models, I mean, it's not just language, but they could have other modalities as well. But they do seem very general and a lot of alternatives that people try turn out in the end to basically just result in similar performance as transformers. The transformers run like well on the current generation of hardware. So like they're parallelize very well, et cetera. And so it might be that you need a little thing on top of that, like maybe it's like the engine block. And then you need some sort of agent loop. Or maybe some external memory augmentation or some other little thing.

• 1:26:17Speaker 01

  But that you would still have this big kind of transformer or something similar to it. Like there might be some variation, but as the basic thing that extracts statistical regularities and abstractions. That's pretty plausible, I think.

• 1:26:35Chris Williamson?

  It's interesting. I certainly wouldn't have guessed 10 years ago that

• 1:26:41Chris Williamson?

  something that you have a conversation with and that is able to accurately predict what it would say would actually be the forefront. You know, it seemed to me tracking quite closely from whatever like 2015, 2016, the development of AI. That I think your book and then subsequent conversations around AI risk kind of blew up that conversation. And then it seemed to me that maybe the 2018, 2019, 2020,

• 1:27:09Chris Williamson?

  AI hadn't really delivered the threat that people perhaps slightly earlier in the 2010s were worried about. And then ChatGPT comes along and this conversation just gets thrust straight back into the forefront of everything. So it seemed like it had a thrust and then a little lull. And then it's really, really sharply come back up again.

• 1:27:30Speaker 01

  Yeah, I think it's, I mean, yeah, why shouldn't like over index too much on like any one latest little development like, but I think also people's expectations changed. And now like, wow, it's been like four weeks without an amazing new release. It looks like AI winter, like it was all just hype. And if you zoom out, I still think we are like on an extremely rapid up ramp and have been since the start of the deep learning revolution in like 2012, 2014.

• 1:28:06Chris Williamson?

  Nick Bostrom, ladies and gentlemen. Nick, I really appreciate you. I've been a huge fan of your work for a long time. Your book is in the hundred books that everybody has to read list that I've been pumping for a very long time. Where should people go? They want to keep up to date with your work and your books and everything else.

• 1:28:20Speaker 01

  Well, I'm not, I'm not active on social media. So I think my website, nickbostrom.com is where I put my papers and everything. So that might be the best place.

• 1:28:30Chris Williamson?

  Nick, I appreciate you. Thank you.

• 1:28:31Speaker 01

  Thank you.

• 1:28:33Speaker 02

  When you shop pickup at Kroger, you can expect the savings you love and fresh groceries selected just for you. Our associates are committed to getting every detail right, carefully handpicking your items, checking for quality and freshness and packing your order with care because bringing you fresh, quality groceries is what we do best. And right now, enjoy $20 off your first online order of $75 or more. Restrictions apply. See site for details. Kroger, fresh for everyone.