AI's Doomsday Philosopher Says Maybe It'll All Be Totally Fine — With Nick Bostrum

Nick Bostrom is a renowned philosopher and bestselling author of "Superintelligence" and "Deep Utopia." He joins Big Technology to discuss the potential outcomes of advanced artificial intelligence, from existential risks to utopian possibilities. Tune in to hear Bostrom's thoughts on how humanity might navigate the transition to a world of superintelligent AI and what life could look like in a technologically "solved" world. We also cover the evolution of AI safety concerns, the concept of effective accelerationism, and the philosophical implications of living in a post-scarcity society. Hit play for a mind-expanding conversation about the future of humanity and the profound challenges and opportunities that lie ahead. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Want a discount for Big Technology on Substack? Here’s 40% off for the first year: https://tinyurl.com/bigtechnology Questions? Feedback? Write to: [email protected] Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  The Oxford philosopher who warned the world about the dangers of AI superintelligence comes on to talk about how we might end up in utopia instead, and whether we should want that. That's coming up right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial, to go inside a large scale agentic CX transformation, including the business decisions, trade offs, and lessons along the way.

• 0:43Alex Kantrowitz?

  You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast, a show for cool headed nuanced conversation of the tech world and beyond. We're here today with Nick Bostrom. He's a philosopher and the bestselling author of Superintelligence, and also the

• 1:26Alex Kantrowitz?

  author of a new book, Deep Utopia: Life and Meaning in a Self-World. I have it here with me today. Nick, welcome to the show. Great to see you again. Good to see you. You know, I've really struggled to figure out like where exactly to start this interview because I was going to ask you about like your past, talking about superintelligence and dangers of that, or the beginning of utopia. And then you know what I said? I'm just going to go back to our last conversation that we had. I'm not sure if you recall it, but I was writing my book Always Day One, and I had a Black Mirror chapter talking about what could go wrong with artificial intelligence technology. And I was like, all right, I'm going to call Nick Bostrom for the AI Black Mirror chapter because you became famous predicting or talking about the probability that we might end up with dangerous superintelligence, and we should be prepared for that.

• 2:11Alex Kantrowitz?

  And this is what you told me. You said, "I don't necessarily think of myself as being in the dark." Mirror people come to me for a quote on the negative side of AI. And then other people will read me saying something negative. And then more people will come to me to get the negative side. It kind of gets to be self amplifying. And people then assume I only have negative things to say about AI. We spoke in 2019. And now you have this book coming out a little bit about how we might end up in utopia with AI. And of course, there are some concerns there. But did I kind of catch you in part of the journey between the dangers of superintelligence? And maybe the the utopic living that we might end up in?

• 2:49Speaker 01

  Not really. It's always been there, both these sides in my thinking and my expectations. The previous book, Superintelligence, did focus most of the pages on what could go wrong. That is true. It seemed at the time more pressing to try to draw attention to the potential risks of developing artificial general intelligence and superintelligence. So that we could hopefully develop ideas for how to avoid those pitfalls. But even back then, it always was clear to me as well that if things go right, they could go very right indeed. And that there was this enormous upside as well. And the more recent book tries to like actually analyze what that would look like

• 3:34Speaker 01

  if things, as it were, go maximally well.

• 3:38Alex Kantrowitz?

  And we're going to get back to some parts of the interview that we did back in the day as we talk through like what this utopia can look like. In the meantime, I do want to talk to you about what it was like to be effectively this focal point for AI fear or negativity. Even if you had this more nuanced view, you certainly have lived sort of the perspective of being this vector for for people with fears of AI, they would come to you. And this is from this New Yorker, a story written about your book in 2015. The title was The Doomsday Invention. And this is describing your arguments. True artificial intelligence, if realized, might pose a danger that exceeds every previous threat from technology, even nuclear weapons. And that if its development is not managed carefully, humanity risks engineering its

• 4:23Alex Kantrowitz?

  own extinction. Central to this concern is the prospect of an intelligence explosion, a speculative event in which AI gains the ability to improve itself and in short order exceeds the intellectual potential of the human brain by many orders of magnitude. So first of all, how do you feel looking back to that portrayal?

• 4:43Speaker 01

  Well, not wrong, I think, still today, that there are very substantial risks in developing greater-than-human machine intelligence, including existential risks. They have become a lot more widely recognized since the book Superintelligence came out. Back then, it came out in 2014, but it had been in the works for six years prior. At that time, this whole idea of, well, A, superintelligent AI in the first place, and B, that it could pose existential risks, was very much neglected. Certainly, academia completely ignored that until the book came out.

• 5:25Speaker 01

  And more broadly in the world, there were, you know, basically nobody, maybe a handful of people on the internet who were starting to think about the alignment problem. That has radically changed in intervening years. Now, there are many research groups focusing specifically on trying to develop scalable methods for AI alignment, including all the frontier AI labs. Also, the governance challenges have started to receive a lot more attention, including from top-level policymakers in the last two years. So the landscape has shifted dramatically, and it's, I mean, I guess it's it's in some sense validating to see that these concepts that used to be super fringe and totally outside the overturn window are now kind of very mainstream.

• 6:08Alex Kantrowitz?

  Where do you think society is in terms of their concerns with AI now? Is there a proper amount of concern? Are they overly concerned? Because the truth is that the sort of doom contingent has really elevated dramatically and is very prominent, you know, even in the most serious research houses, OpenAI, Anthropic, etc.

• 6:29Speaker 01

  Yeah, there has been this surge of the doomers in the last, I mean, two years or something like that, where they really started to become vocal, like on Twitter and stuff. I think probably we are still below the optimal level of concern. But the kind of first derivative is that we're moving rapidly towards increasing levels of concern. And I've started worrying a little bit about us not, as it were, increasing the level of concern to the optimal level and then stopping there, but kind of overshooting if this negativity about AI.

• 7:07Speaker 01

  Snowballs like it might in some scenarios get out of control and we might then lose out on the enormous upside that could be unlocked by developing the good kind of superintelligence.

• 7:19Alex Kantrowitz?

  What does that overshooting look like? Is it just that these concerns grow, I mean, bigger and bigger?

• 7:25Speaker 01

  There are different ways. I mean, one scenario class which still looks unlikely to me, although less unlikely perhaps than two years ago, is that you could imagine so much stigma developing that we get some ban, like maybe it starts with a pause or something, but then like after six months, nothing has really changed. So you extend the pause and you maybe set up regulatory agencies to police it to then have an, and all they are doing is trying to prevent dangerous AI like and, or just a kind of negativity might, might develop into a new orthodoxy where like anybody who says anything positive about AI gets, gets cancelled or shadow banned, it just becomes

• 8:06Speaker 01

  like impossible to, and so it, it still seems a bit unlikely, but we can't really accurately predict the kind of social dynamics that will be playing out and now under different circumstances than historically, because we have new technologies available now that might at some point make it possible to lock in an orthodoxy in ways that were never possible historically, because you could have much more fine-grained control over what people are saying. So that's one like class, like another might be that instead of AI being banned, it kind of gets escalated to the national security level and, which may or may not be good, but you could certainly imagine scenarios in which instead of it being a kind of peaceful

• 8:47Speaker 01

  pursuit for civilian purposes by cooperating scientists generally motivated by some sort of, you know, humane cosmopolitan goals, like it becomes more like a kind of how can we get an edge over our military rivals? And it's not clear whether in that context, the outcome looks rosier than, than if it were more like a kind of free-willing civilian enterprise. So that, that, that could be like a concerned level with possible negative effects, even short of stopping AI development altogether.

• 9:17Alex Kantrowitz?

  I mean, if you look at today's large language models, and I know we know this isn't the end of development, but they are very smart in some areas and really stupid in some areas, and they continually fail tests that are put in front of them to try to demonstrate real world understanding. Where does this fear come from? And what could what could eventually happen as we continue this path of development that could lead to some of the negative outcomes that you and others anticipate as one possibility?

• 9:43Speaker 01

  Yeah, I mean, I don't know how much we should read into the fact that current like AIs still have limitations, that that will remain true until it isn't true any longer, right? But but the idea has always been that we can anticipate that AI capabilities most likely will continue to increase, and at some point AI will succeed. And then we can realize that at that point, like some really powerful dynamics will kick in, and we could try to use the intervening time to prepare ourselves in different ways, including ideally solving the alignment problem, you know, maybe making some progress on the governance challenges and on like the ethics of there's a bunch of difficult ethical questions that

• 10:25Speaker 01

  will arise with like the moral status of digital minds and the distribution of benefits, like a lot of quite tricky questions that we could use these intervening years to try to, you know, think about. Unfortunately, we wasted most of the time we had. I mean, I think, I mean, I, to me, at least, even back in the like 90s, when I started thinking about these types of things, it was clear that we had a good chance of eventually developing AI and that it was going to pose these risks. We could have used those decades to do our homework. And it's become increasingly clear, but really only in the last maybe five years or so has to have been like a significant effort to actually work on these issues. So we we still have some time left on the clock. We don't know how much, but hopefully

• 11:10Speaker 01

  we can at least make a good use of the remaining months and years.

• 11:15Alex Kantrowitz?

  Yeah. Why do you why do you believe that it's a certainty that will sort of achieve artificial general intelligence?

• 11:20Speaker 01

  Certainty is certainty is too strong a word, I think.

• 11:23Alex Kantrowitz?

  But you feel it's very likely?

• 11:26Speaker 01

  Yeah, at least conditional on science and technology continuing. And like, we don't, we could have some sort of civilizational collapse, or we could eventually, you know, go extinct or destroy ourselves in some other way, not unrelated to AI. There are like developments in synthetic biology and other new weapon systems and whatnot. But if, if kind of these development efforts in, in, in hardware and in, in algorithms continue, then it looks very likely that we will succeed in this. I mean, from first principles, we have the human brain as an existence proof that general intelligence is possible. There is no reason at all to suppose the human brain is in any sense optimal,

• 12:08Speaker 01

  like neither from a hardware point of view, nor presumably from the algorithmic point of view.

• 12:16Speaker 01

  So just as we have machines that are physically much stronger and faster than human bodies or any animal body is, then likewise, we will have eventually cognitive systems that are much faster and clever. So that's like a very high level argument. But then you can also look just at the kind of advances we are seeing, where more and more things that used to be impossible for AIs to do have become done. And there are just a lot fewer left of these kind of milestones. There is this phenomenon that like, before it is done, it looks really hard. Once AIs have done it, then we kind of quickly forget how, just how like impressive it was and just take it for granted, like, yeah, of course, computers can play chess.

• 13:00Speaker 01

  Like, of course, what's the big deal? But like, at the time, like, it was a big, and then you can see like, oh, we have, they can play Go, they can see, they can imagine, they can write poetry, they can write computer programs, they can talk to us in ordinary language, they like, can pass like, you know, undergraduate level exams in all these different subjects. Like, this, this, this is a lot of stuff that like, to people 30 years ago would have been like, wow, you must be really close to ADI if you have done all of these things.

• 13:33Alex Kantrowitz?

  Okay, so let's talk about utopia a little bit. Just give us your perspective on what could go right in the best case. Well, why don't we do this way? What could go wrong in the worst case scenario of AI? What could go right in the best case scenario of AI? And how do humans have an influence in terms of which direction we go?

• 13:54Speaker 01

  So I think there is like the real X risks, existential risks that will arise as we develop. And possibly not even super intelligence, but you could imagine even something short of that making it very easy to develop new weapons of mass destruction in using synthetic biology.

• 14:09Alex Kantrowitz?

  Are you more concerned about humans using AI to hurt each other or AI hurting us?

• 14:14Speaker 01

  Well, I think they are both worth worrying about. I think with the X risks, I mean, maybe slightly larger on the AI being the kind of agentic part there. But certainly, we really need to do a decent job on both sort of the alignment and the governance for us to have a good outcome. So yeah, so now on the upside, like there is a huge unlock. And a lot of that is just removal of a bunch of negatives. If you look around at the world as it is now, it's not that rosy a picture. In many ways, it's quite a horror show with people dying from, you know, Alzheimer's

• 14:54Speaker 01

  or kids getting cancer and like starvation and people being bombed and like all kinds of. Or just at the more mundane level, people spending most of their adult life, you know, working in a sort of boring occupation that gives them no fulfillment, but they just have to do it to, you know, to pay the rent. And like, you know, headaches and stomach aches and like all kinds of, just the totality of all of this extreme misery and very common, more everyday misery. That's just within the human sphere. And then you add the animal kingdom with like ameliorating all that suffering would already be,

• 15:34Speaker 01

  I think, a very strong argument that something at some point needs to be done here. Can't like gone like this. I mean, do we want like another? You know, 10,000 years, 100,000 years of justice. But I think on top of that, there is like the potential to also unlock like new levels of flourishing beyond those which are possible even under ideal conditions in the current world. That's a lot harder to paint a very concrete picture of because we are sort of limited in our ability to imagine and appreciate, just as, you know, if you imagine like the great ape ancestors of Homo sapiens kind of thinking about what could be so good about

• 16:18Speaker 01

  being, you know, human. And so they might like realize a few things like, oh, we could have banana plantations and have like a lot of bananas and stuff. And that is true, we can have a lot of bananas now, but there's more to being human than just unlimited bananas, right? Like we have sort of, you know, music and poetry and film and humor and romantic love and like all kinds of stuff, science. So similarly, there is probably like if we unlock, as it were, the a greater space of possible modes of being, there are some in there, I'm sure that are extremely valuable, that I think AI would be the most plausible path towards realizing.

• 17:03Speaker 01

  If I then really dives in and tries to think more specifically about like what would the best possible continuation of a life starting from like our current human starting point look like, then there are some quite interesting philosophical questions that arise. And so this book, Deepi Toppe, it's not really an attempt to sort of, well, before we looked at the downside, now let's make the case of how wonderful the upside could be. I think the upside could be extremely wonderful, but that's not sort of the thrust of the book. It's more like, let's just look at this, what would happen if we actually? Did succeed in creating a solved world, as I call it, like where all the practical

• 17:44Speaker 01

  problems are already solved, or to the extent that there are problems that are not solved, they are in a way better dealt with by advanced AIs and robots than by us. And there is like some aspects of that condition that at least prima facie look quite unappealing to our current sensibilities. We often define our sense of self-worth on the idea of being a contributor, like you're a breadwinner, or you make like a positive difference in the lives of your friends or of society at large, you like you bring value to the world.

• 18:25Speaker 01

  So much of our existence is kind of constructed within the constraints of various instrumental necessities that have been with us since, you know, the dawn of the human species. There have always been a lot of things that we need to do just to survive. And if you remove all of those, there is at least initially this sense of kind of disorientation or an undermining of like, we feel like kind of what's the purpose, like we would just be these blobs.

• 18:57Alex Kantrowitz?

  But this is different from what we spoke about the last time you and I were on the phone. This was in 2019, I think. You said that we'd have to find some new source, this is from our conversation that I put in my book, we'd have to find some new sources of self-worth. But in Disneyland, the job of children there is to enjoy the whole thing. And Disneyland would be a rather sad place if it weren't for the kids. So you say we would all be like kids in this giant Disneyland, maybe one that would be maintained and improved by our AI machine tools. So effectively that even if we didn't have to do any sort of sustainment work that gets turned over to AI, we could be actually quite fulfilled in life.

• 19:35Speaker 01

  Yeah.

• 19:35Alex Kantrowitz?

  So how did you get from there to where you are today?

• 19:38Speaker 01

  Well, I think that's basically correct, but we can distinguish two different senses of fulfilled or of having purpose. So there is, first of all, what you might say, the subjective sense, like the feeling of fulfillment or the feeling of having purpose. Like, it's like the emotion of being motivated and you're really excited about what you're doing, right? Like that kind of psychological state, that certainly you could have that in a solid world, in utopia. The utopians could have like extreme levels of motivation and immersion and subjective purpose. That's easy. That's like a checkmark.

• 20:17Speaker 01

  And more broadly, you can go through different plausible human values. And for some of them, you can just right off the bat say, well, yeah, sure, of course. That would be trivially easy to do in utopia. So in this case, through the psychological engineering techniques that they would have. I mean, already you could have, like, imagine a drug without side effects and without addiction potential. That just induced a state of fascination and motivation. We already have simple versions of that. But you could imagine far more sophisticated ways that would give the utopians, like, very fine-grained, direct control over their mental states and their psychology. That would follow from technological maturity. Right.

• 20:57Speaker 01

  So that's easy. Now, some people, however, think that there is also a more objective concept of purpose.

• 21:07Speaker 01

  Where it's not just that you feel motivated, but that what you are doing is actually objectively worth doing. And that's a little bit less obvious to what extent the utopians would have that. Inasmuch as, at least at first sight, it looks like anything they could do, they wouldn't have to do because they could just press a button and a machine could do it instead.

• 21:28Alex Kantrowitz?

  Except for those few things that you bring up that actually we want humans to do, like for instance, like ordain a marriage, right? Like that is something, or read a poem. People might want this to be humans.

• 21:40Speaker 01

  Potentially, yeah. So you could automate everything, but except there might be certain jobs where the, as it were, consumer has a direct preference that the job would be done by human. In which case, by, as it were, almost by definition, it's not automatable.

• 21:54Alex Kantrowitz?

  Exactly. And this whole idea of a solved world is basically where AI can effectively take care of everything, all of our needs, all of the production. And we are in this utopia because the machines have done all the hard stuff that we don't want to do anymore.

• 22:07Speaker 01

  So you could imagine like a very dystopian scenario with advanced technology. If you have like a sort of totalitarian, despotic, right? But imagine also that to whatever extent governance problems can be solved, they have been solved. Maybe you can't solve governance, but to whatever extent they are like better and worse in terms of social, political structures. Imagine like we get something at the good end of that combined with technological maturity. Then that's basically the definition of a solved world.

• 22:38Alex Kantrowitz?

  Sorry, I know I took you on a bit of a tangent there.

• 22:40Speaker 01

  No, so, so, yeah, so like that kind of layer. So you could say first, well, you could have, like, like a simple utopia might be a kind of post scarcity utopia, where we just have abundance of material goods. So we already, if you're fortunate enough to live in like a developed country, you know, with a decent education, et cetera, you're already pretty close to that. You might, might not be able to have like the ideal yacht of your dreams. Like there are some limitations, but if you sort of plot the line that has like the starting point hunter gatherer and the endpoint is like complete post scarcity. I think we are more than halfway there. Like it's a bigger difference to go from not having enough to eat to having enough to eat than,

• 23:25Speaker 01

  you know, to get like a slightly more advanced version of an iPhone or like, like a third house if you already have two houses, like diminishing returns. So, so that's like, you could first consider this concept, post scarcity utopia. Okay, so then what's a level of, as it were, more radical utopia than that? Well, you could have a post-work utopia where.

• 23:49Speaker 01

  Not just do we have plenty, but we don't have to work to produce that plenty. So it's not just that we work all day long and then we have a lot of money and we buy stuff. But imagine you had all this plenty and you didn't have to work. It's slightly more radical conception, but not that radical. I mean, there are already people who, you know, are born with a trust fund or something and they never have to work and they have plenty. You know, again, there's limits to how big their palaces could be, but like at least some approximation. But I think we can then go further and consider even more radical conceptions. So I've already alluded to, um, there is the post instrumental utopia that you could have, where it's not just that we don't have to work, like to make money,

• 24:34Speaker 01

  but we also don't have to do any of the other things that we currently have to do for instrumental reasons. So if, you know, if you are Bill Gates, you still have to brush your teeth. Uh, you still have to, you know, do a whole host of things just in your everyday life to get the outcomes you want. There's like a limit to how much you could ask your assistant to do or that you could, you know. But, but in this scenario, like a lot of those other instrumental reasons we have for doing things would also drop out of the picture with like super advanced automation technology. And I think there's like a step further than that, which is a, I call it a plastic utopia,

• 25:15Speaker 01

  where, oh, we also have complete control over ourselves, over our own bodies, minds and mental states, using like advanced techno, biotechnologies or neurotechnologies.

• 25:28Alex Kantrowitz?

  You mean that's gonna, we're gonna achieve that? That's wild.

• 25:31Speaker 01

  Yeah, but I think if you consider what would be possible at technological maturity, which we can at least play some lower bounds on through a kind of theoretical analyses. So we, we can sort of estimate what kinds of computational systems could be built. We can see what kind of molecular manufacturing systems are, are possible to build in our universe, even though we can't currently put them together with the tools we have now. We can see that there is like a path there. Other things like cures for aging and stuff like that, like we don't have them yet, but there is no, you know, no laws of physics prevent people, you know, from, from living indefinitely if you had like repair technologies, et cetera, et cetera. Perfect virtual realities, perfect like.

• 26:17Speaker 01

  Ways to manipulate the brain like that are better than drugs like this like a whole you could There's this kind of in fact like a table in the book that outlines some of the affordances You would have at technological maturity. And maybe there will be additional things we haven't yet thought of, but at least these. And so they would, I think, enable Us to instantiate this condition of plasticity where where human nature itself becomes malleable.

• 26:42Alex Kantrowitz?

  It's crazy.

• 26:43Speaker 01

  So that means There are further questions about purpose. So right now, like if you didn't have to work for a living, Like maybe some people would say, well, you know, maybe I would start, you know, going to the gym or to get fit. Right? Like that's, you can't hire a robot to, you know, go and run on the treadmill on your behalf. But with plasticity, there would be a shortcut to that. You could pop a pill that would induce exactly The same physiological and psychological.

• 27:09Alex Kantrowitz?

  We're already on the way to there with Ozempic.

• 27:13Speaker 01

  Yeah, yeah, exactly. That's like one more step in that direction. And so the thing is with superintelligence, you get the telescoping of the future. So all these sort of science fiction like technologies that maybe we would develop if we had like, You know, 20,000 years for human scientists to work for it, we probably will have a cure for aging And perfect virtual reality and space colonies and all the rest of it, right? But all of that could happen very quickly if you have superintelligence doing the research and development. So you get this kind of telescoping of the long term.

• 27:48Alex Kantrowitz?

  Yep.

• 27:49Speaker 01

  But yeah, so then there's like a further set of things that currently fill the lives of people That we wouldn't need to do. Including things we do for fun. So maybe some person say, well, you know, if I didn't have to work and I had like maybe I would play golf all day long, like because why, why would you play golf all along? Well, because it's fun. It gives me joy. Let's suppose somebody says that. Well, then, in this condition of plasticity, there would be a different and easier way for them to get joy. They would just pop a pill and they could get exactly the same level of subjective well being As like a beautifully manicured golf course could induce. And so.

• 28:32Alex Kantrowitz?

  I can't even imagine that type of world. That's crazy.

• 28:35Speaker 01

  Yeah, so it does then require a fairly fundamental rethink of what it means to be human In this radically transformed condition. But. It is kind of the implicit tea loss of our current strivings, if you think about it. So we try to, little problems come up, like, we try to solve it. And then, like, there's like another problem that we all like, so our food rot, let's invent a refrigerator. Like, oh, we get fat, let's like invent Ozempic, like, oh, our cars pollute, let's make cleaner engines. Like, and but if you kind of extrapolate and take all of that to its limit, then you would end up in a situation where we can like do everything with no effort, right? That would be kind of the the limit of technology.

• 29:20Alex Kantrowitz?

  Right.

• 29:21Speaker 01

  AI, you know, the goal of AI has all along been not just to automate a few specific things, but to provide the technology that allows us to automate all tasks, right? Like, AI hasn't really succeeded until all intellectual labor can be done by a machine. And so I think it's kind of, we don't, we don't think about it like that. But if you sort of see what all of this effort is, all these investments we have in science and technology, and our efforts to make the economic system more efficient, to allow kids to learn more in school, like, all of this kind of adds up to some sort of arrow of attempted progress in a certain direction. And you might as well at some point stop to think, what happens if we actually get there?

• 30:04Alex Kantrowitz?

  Yes.

• 30:05Speaker 01

  And then we do end up, I think, in this condition of a sold world.

• 30:10Alex Kantrowitz?

  And the question is whether we're ready for that. So Nick, are you able to stick around for another couple of minutes? Or do you have a hard out?

• 30:16Speaker 01

  I could do a few more minutes, maybe.

• 30:17Alex Kantrowitz?

  Okay. All right. Let's take a quick break and come back and ask a little bit about what, whether we can handle that perfect world. We'll be back right after this. This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud. And the agents acting on your behalf.

• 30:58Alex Kantrowitz?

  It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AVPOINT, the unifying trust layer for AI. Head to AVPOINT.COM to see how enterprises deploy AI with confidence. Learn more at AVPT.co/BIGTECHNOLOGYPODCAST. That's AVPT.co/BIGTECHNOLOGYPODCAST. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone.

• 31:40Alex Kantrowitz?

  They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/BIGTECHNOLOGY, fill in the quick form and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com/BIGTECHNOLOGY. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 32:25Alex Kantrowitz?

  And we're back here with Nick Bostrom. He's a philosopher, the bestselling author of Superintelligence, and author of the new book, Deep Utopia: Life and Meaning in a Solved World. It's out now. Great book, definitely recommend you pick it up. So you've basically said, if we get to this perfect world, you think right now we're effectively unfit to inhabit it. And in your book, we sort of look at, or in the early chapters, you sort of look at the fact that we've increased our productivity, but we're using it for consumption rather than leisure. And that's concerning to you. Is that part of the reason why you think we're not quite ready for this? So where do we fall short in our preparation for utopia?

• 33:05Speaker 01

  Well, I think human nature is kind of forged and evolved under various conditions, including conditions of scarcity and conditions where there are like instrumental demands on us. That we need to exert ourselves, make efforts, try, we need to work just to get by in life. This has been true for hundreds of thousands of years. It's still true to some extent today, although with certain relaxations. Like, for example, food is much less of an issue for people living in wealthy countries. And increasingly also for more middle-income countries, where obesity is becoming an issue. So there you can already see a little bit of a mismatch, like where we kind of evolved to live under conditions of food scarcity.

• 33:49Speaker 01

  And when that no longer obtains, unless we make adjustments, like we kind of balloon in size and then we need to try to find fixes for that. But I think a much more profound mismatch between where we currently are psychologically and biologically and our environment could arise if we sort of suddenly moved into a condition of a solid world. So there would need to be some adjustments in that, I think, scenario if we wanted to take advantage of all the things that would be possible.

• 34:18Alex Kantrowitz?

  And what would those adjustments be?

• 34:20Speaker 01

  Well, I think, for a start, if we take, for example, human hedonic well-being, which maybe is like one of the easiest things to look at, just like the subjective state of positive affect. Like, are you actually enjoying the present moment? Does it feel good or is it like unpleasant? And so this is like a fairly fundamental dimension of our psychological state. Some people think it's the only thing that matters. If you are a hedonist, in the philosophical sense, you think that the only bad is suffering and the only good is pleasure. Not necessarily in the sense of physical pleasure,

• 35:03Speaker 01

  but in the broad sense of sort of positive mental hedonic tone.

• 35:08Speaker 01

  So it looks like we are kind of designed in a way where we have a fairly powerful habituation mechanism. So if somebody's conditions in life improve a lot, often, you know, maybe they win the lottery or something, right? So they get really happy they win the lottery, but very quickly the sort of hedonic tone falls back to the baseline. Because we are not designed for permanent bliss. We are designed in such a way that our reward system motivates us to produce more effort. At whatever level, like no matter how good our situation is, we are designed to always try to want to make it better. And so we only get reward when things improve, rather than when things are like at a good level.

• 35:52Speaker 01

  To a first approximation, it's not completely true. I think people under better conditions are slightly happier than people under worse conditions. And maybe a lot happier if they're like under really bad conditions. It's like certainly has a sort of permanent negative effect. But now, if there were no more opportunities for improvement and there is like no need for our like instrumental efforts, that it seems very stingy that we would still not be able actually thoroughly to enjoy lives. So maybe we would want to change that set point so we could all be much happier all the time. To actually relish this this future.

• 36:32Speaker 01

  Like it would be sad like if if everything were as like super nice and we were still miserable there like that. And then just living like that for like millions of years and like not really. That that would seem to be a very unfortunate. So that that could be one obvious adjustment.

• 36:52Speaker 01

  There's like a bunch of other things as well that like you you might imagine upgrades of various human capabilities. Our cognitive abilities, our emotional repartories, our ability to connect to other people, obviously, physical health.

• 37:09Speaker 01

  And then kind of at the philosophical almost or like

• 37:15Speaker 01

  our overall attitude to life, like so the idea that you sort of conceive of your self-worth as being based on the your ability to make a contribution. Maybe it needs to be rethought here, like if we can no longer make contributions. There is an asterisk to that. I think there are certain ways in which maybe the utopians could. But at least at first, like our opportunity to sort of help out other people would be reduced if there is just less misery and need in the world to begin with. Like if you're a doctor and there was no disease, like you need to like find another occupation or you can't base your self-worth of being a really good and caring doctor if there is no, if nobody's sick, like then you have to rethink that.

• 37:57Speaker 01

  And I think at a more general level, we would all have to sort of rethink what makes like a human life have dignity in this condition where we are no longer really where it's at.

• 38:07Alex Kantrowitz?

  Yeah. Okay. Here's the last one. It's kind of wacky, but I thought I'd throw it out there and see what you think. If we reach superintelligence and we hit this utopia, do you think, and I know you've spoken a little bit about the potential of us to be living in a simulation. So if we reach superintelligence, do you think there's a chance that we're going to crack out of the simulation and effectively figure out who's running it if we're in one?

• 38:34Speaker 01

  That would be one possible scenario, right? If you're in a simulation, the simulation could just end with nothing, or it could be rerun, or it could like enter a different sort of environment within the simulation continues, but the sort of virtual environment changes, or indeed you could be sort of uplifted out of the simulation into like the world of the simulator. So all of those are at least kind of metaphysically possible, conditional on the us being in a simulation in the first place. So the simulation hypothesis expands the space of sort of realistic possibilities and the space of realistic futures.

• 39:17Speaker 01

  You might think if you're living just in a simple materialistic universe, you die, that's the end, your brain rots, there is no more experience. And there's really not much room for other things, given like the laws of physics and us being purely material with no soul, et cetera, et cetera. If you are in a simulation, then there's like a much wider range of things that could happen that that would seem perfectly possible, given the assumption.

• 39:40Alex Kantrowitz?

  I mean, if we're there, it would be so crazy to just unzip and poke our heads out and see what's behind the thing.

• 39:47Speaker 01

  Yeah.

• 39:47Alex Kantrowitz?

  Okay. The book is Deep Utopia: Life and Meaning in a Solve World. It's by Nick Bostrom, our guest today, also the author of Superintelligence, bestselling author Nick Bostrom. Thanks so much for spending some time with us today.

• 39:59Speaker 01

  I enjoyed our conversation.

• 40:00Alex Kantrowitz?

  Me too. All right, everybody, thanks so much. We will see you next time on Big Technology Podcast.