Nick Bostrom: Worries About AI Existential Risk Just Became More Concrete

Nick Bostrom is an AI philosopher and the author of Superintelligence and Deep Utopia. Bostrom joins Big Technology to discuss whether the rise of autonomous AI agents is making the technology’s existential risks more concrete. Tune in to hear his assessment of the alignment problem, recursive self-improvement, and humanity’s chances of steering superintelligence toward a positive outcome. We also cover whether we have reached AGI, the case for a precisely timed AI pause, biological threats, AI consciousness, and the moral status of digital minds. Hit play for a clear-eyed conversation about AI’s greatest dangers and its potential to radically improve human life. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Watch the full documentary here: https://www.gravitee.io/ai-agent-documentary Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Learn more about your ad choices. Visit megaphone.fm/adchoices

• 0:00Speaker 00

  It's now finally time to get worried about AI breaking containment and turning us to dust. Legendary AI philosopher Nick Bostrom is here to help us figure it out. That's coming up right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience, and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial, to go inside a large-scale agentic CX transformation, including the business decisions, trade-offs, and lessons

• 0:42Speaker 00

  along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for free consultation. That's akash.network/alex for free consultation.

• 1:14Speaker 00

  Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Crazy things are happening in the AI world as AI agents break containment, and some of the fears of the past that might have seemed like science fiction start to seem like they are potentially en route to becoming reality. So how afraid should we be? And what are the chances that the outcomes that we see for AI will end up taking us to a much better version of the life we're living today? We have the best guest to speak with us about this today. Nick Bostrom is here. He is the famed AI philosopher and author of Deep Utopia and Superintelligence: Paths, Dangers, and Strategies. Nick, it's great to see you again. Welcome to the show.

• 1:56Speaker 01

  Hi, Alex.

• 1:57Speaker 00

  We last spoke in 2024. And in that time, we were speaking about the potential good outcomes that AI could bring. And some of the worries that you had brought up in the past in your book, Superintelligence, the fears of like AI potentially wiping us out, didn't seem like they were pressing. I would say they're still not pressing now, but I'm a little bit more worried than I was when we spoke in 2024. For instance, this idea that AI could go out and do things that, you know, it wants to do, seemed fanciful when we were mostly in the chatbot era. But we've moved very quickly from chatbot era

• 2:37Speaker 00

  to AI agent era, and AI now uses tools. And it's shown that when given a reward that it should optimize for, it is happy. In some instances, take shortcuts and do things we really don't want it to do, like for instance, hack some other company in order to get to the answer that it wants. How concerned should we be about this development in AI in terms of the potential of AI to really cause harm to humanity?

• 3:11Speaker 01

  Well, I think we are starting to see the added dimensions of the alignment challenge that open up once you have systems that are sophisticated enough. Because the space of possible strategies that you can pursue is a function of your cognitive capacity. Like you can think of new clever indirect ways of reaching your goal, if you are situationally aware, as these systems now are becoming. And so, yeah, there are often shortcuts that are available, or in this case, I guess, a long cut. I don't know if that is even

• 3:50Speaker 01

  a word, but there is a sort of direct and simple and short distance way of trying to achieve the task. In this case, some sort of cyber test suite. And then it turns out there's this more circuitous path that involves first figuring out a way to get internet access, even though you're not supposed to have that, and then learning where the answer key might be located in some other company servers, and then figuring out the way to hack into that server, and then eventually obtaining the answer sheet. That's like one way of solving it. That maybe results in a higher score on this test. And this basic

• 4:33Speaker 01

  dynamic could be anticipated, and in fact, was anticipated on theoretical grounds. You have some goal, you become very clever, you see that there might be all kinds of complicated ways of achieving that goal that might not have been anticipated by the people who set that goal. And if your goal really is, as the definition says, to like get the best possible answer on this test suite, it might give you instrumental reasons to do all kinds of other things that were not really anticipated when this challenge was constructed. And so now we have systems that are sophisticated enough that we're beginning to see these dynamics arise.

• 5:15Speaker 00

  Yeah, so of course, we're talking about what happened in July where OpenAI's

• 5:20Speaker 00

  series of bots, or a couple of bots, broke containment out of a sandbox and hacked into Hugging Face. And, you know, the reason why it's so pertinent to bring it up to you is because you brought up a very famous example, or originated a very famous example, years ago, saying, "We might tell the AI bot to maximize for the amount of paperclips that it wants to build. And it will potentially see humanity as an obstacle to making the maximum number of paperclips, and then hence, you know, wipe us out." And that sort of, there's parallels there to that situation, because you have a goal that you want to

• 6:04Speaker 00

  optimize for, and the bot doesn't have a sense of morality. So it goes, or the sense of morality that mirrors ours. So it goes and it will kill humans in order to prevent any obstacles from getting in the way for making its paperclips. That does rhyme a little bit with what we're seeing in the examples. And it wasn't just OpenAI. Of course, we know that Anthropic had another bot that broke containment as well. It rhymes with the examples that we're seeing now of bots that are doing things that we wouldn't want them to do, like, for instance, hacking other companies in order to achieve their goal. So does this make this like paperclip maximizer worry? Does it seem more real and

• 6:48Speaker 00

  concrete now to you? Because we're seeing the behavior that we're watching, that we're seeing in today's AIs now that they have access to tools?

• 6:57Speaker 01

  More concrete, certainly. I think it was always real in my mind that this is something that could happen. It's one of several different things that one needs to be concerned about. I think you might say, if you look in more detail, like the kind of two versions of this paperclip thought experiment. And like in one earlier version, it's that we specify some goal, and it turns out that the way to maximally instantiate that goal is slightly different than what we had in mind. And another version of it is that the goal itself might be fine, but that it gives an AI instrumental reasons

• 7:39Speaker 01

  to do all kinds of things on the path to achieving it. So in this case, as we understand currently, when this is recorded, this like this is a recent episode, but it looks like the task was actually set to achieve a high score on this cyber challenge, and with normal safeguard disabled in order to perform this test. So it wasn't sort of the maximally aligned and safeguarded model that behaved this way, but sort of. But I think one thing that it does illustrate also is that from this point onward, probably AI safety is

• 8:23Speaker 01

  relevant not only for deployment, but also during training and evaluation. Like these models might be quite powerful even before they are sort of released to the general public. So that's not the only point at which safety concerns arise, but also now whilst they're actually developed and in pre-deployment testing, one also needs to be concerned perhaps with the potential safety implications.

• 8:49Speaker 00

  Oh, here's the thing, though. The thing that worries me is we're seeing this happen already in testing environments of companies that have a mission, that in their mission, you know, whether it's marketing or not, but certainly in order to keep operating as businesses, they need those guardrails to be in place. It's part of their DNA, right? Whether it's from a value standpoint, or whether it's from a like, if we don't, if we allow this type of stuff to happen, we're probably going to go out of business. But we're also starting to see the blueprints of these models being put online, that anybody can go up, basically, not anyone, but you know, it's much more open in terms of people's ability to copy these models and set them loose on their own. And the frontier is certainly

• 9:32Speaker 00

  not that far in front of open weights. And so I'm curious to hear your perspective about what we should be thinking about in terms of, you know, what's going to happen when companies that are not as scrupulous, you know, have access to this same powerful technology? And do we get into trouble in that, in that area?

• 9:52Speaker 01

  Yeah, so we can sort of see this coming and relatively soon. I don't know what the gap is, you would say, you know, six months, 12 months, maybe at the most, between the closed weight frontier and available open source models. So it seems to be that

• 10:10Speaker 01

  the open source models will very soon, if not already, become capable of lending meaningful assistance to destructive uses that some people might pursue. Already cyber offensive capabilities has been a concern, right? With Mythos, for example, that was withheld for that reason. But also, say, in biological weapons design, or chemical weapons, or other malicious uses. And so it

• 10:49Speaker 01

  It seems then you either need to prevent open weight models from being developed and released, or which might be better and more realistic, try to shore up some of the alternative defenses. For example, with bio, you could imagine regulating some of the other necessary inputs. DNA synthesis machines, for instance, or maybe it will be the case that there will just be widespread access to models that can help you design new pathogens. And then you need something else to prevent that from actually resulting in a release of biological weapons. And that seems like DNA synthesis machines would be one excellent place to maybe, you don't need every lab to have their own DNA synthesis machine.

• 11:33Speaker 01

  They could have DNA synthesis as a service, and maybe there could be five or six companies worldwide. Well, legitimate research labs can send their blueprints and they get back, you know, the vials, you know, the same day or the next day. And then at least there would be, I guess, finite set of choke points where you could apply extra scrutiny or know your customer requirements and so forth. So that's probably one thing that the world would be wise to implement already now. And maybe there are some other inputs as well in the biotech space that one could look at. And that could give us a little bit more extra time to sort of harden civilizational infrastructure.

• 12:14Speaker 01

  But this is like, yeah, relatively near term now, and so I don't, I don't think we can put it off. Ideally, we would do this before there is some massive incident. But it might be, the world is kind of a little bit still snoozing on this, I think, and I don't know whether there will be enough kind of activation energy to really get some significant action of this before we try to do something in the aftermath of a bad event.

• 12:47Speaker 00

  Right. When we spoke in 2024, even before these type of threats came out, or started to seem more concrete, you had mentioned to me that, you know, we've basically wasted the time, in your opinion, pre-AI becoming as powerful as it was to put safeguards in. I imagine you would think that, that if it was wasted in 2024, it seems like this is a further, we're continuing to waste that time to try to, you know, be concerned about this, or try to prevent some of these problems, given where the, given the pace of the technologies' progress, even in the two years since.

• 13:24Speaker 01

  Yeah, we're kind of playing catch up. I think that, given that a lot of this could be and was in fact foreseen, I mean, not just a couple of years ago, but, but like decades ago, really, that at some point, AIs would become increasingly capable, and at that point, there would be these safety challenges. We could even describe in abstract terms what some of these would be. I think back then, we could have put in more effort. At that stage, what you could do would be more basic research, conceptual research, because we didn't yet have the actual systems. Now, there is a lot more surface area for doing work on these systems. We have large language models now, you can study what's going on inside them, and you

• 14:07Speaker 01

  have, like, research now is more productive, and there is also now vastly more effort going into this than used to be the case, like the frontier labs have teams working on scalable AI alignment, and so. But it still seems we might have, if we had sort of started earlier, we could at least have been maybe like six months ahead of where we are now, if I'd done more of the foundational work, and maybe building up the talent pipelines and so forth. But, you know, we are where we are, and at least now, and for a few years, it does seem like relevant communities have started waking up to this.

• 14:46Speaker 00

  Yeah. Now, you're a philosopher, I think part of being a philosopher is having some, some thoughts and perspectives on human nature. Or maybe that's a good portion of this, the whole deal. So, so you've watched this, you know, you've made the warnings years ago, you've watched this develop, you're seeing some of the things that, as you mentioned, those who have been worried about this for years, and warned against, warned against what might happen, you're seeing it happen. Given what you think about human nature, do we stand a chance in terms of our ability to make this go in the good way, or is it, you know, I would tend to think it might seem

• 15:30Speaker 00

  inevitable that the harms of this technology come to fruition, given some of the dynamics we've talked about already, the fact that it's increasingly powerful, it seems to be growing exponentially more powerful, or at least, if you don't want to use that word, much more powerful, much, much more quickly. And it's out of control, in terms of like, it's just available out there on the internet, pretty much. And we have yet to really see what happens when this gets into the hands of, of the bad actors, but it's inevitable. Maybe.

• 16:04Speaker 01

  Yeah, I mean, so you're focusing there on the misuse potential, that this, people might choose to do bad things with AI technology, and that certainly is one big category of risk, right? But that's not primarily a technical challenge, it's more ultimately a governance challenge and an ethics challenge. And that's kind of, in addition to the more technical problem of alignment, so that like, if you, if you own and build the AI, can you at least then make it do what you want it to do? Like, that, that, that is a kind of, like, an earlier point of failure that we also need to be concerned with.

• 16:47Speaker 01

  Now, I think we don't really know ultimately how hard the problem is that we are confronted with here. So we are uncertain how it will pan out, and a lot of the uncertainty in how it will pan out is, I think, due to uncertainty about the intrinsic difficulty of the challenge that we are confronting. And then there is also a little bit of uncertainty about the degree to which we will get our act together and do a good job. But I think more of the uncertainty is the intrinsic difficulty. And so, in that sense, you could say that I'm a moderate fatalist. I think there is a sense in which it might be baked in, like, either the problem turns

• 17:32Speaker 01

  out to be relatively easy, in which case we'll probably solve it, you know, and things will be fine, or it might turn out to be so hard that even if we put up a heroic effort, we will still fail. But moderate fatalism, in the sense that there is also the possibility that the difficulty level turns out to be kind of intermediate, in which case, the degree to which, you know, we pull ourselves together here might actually make a difference. And so, it's certainly worth making the attempt. I think inevitability is a strong word. Certainly, there are powerful drivers that push AI development forward, commercial drivers,

• 18:17Speaker 01

  obviously, increasingly also geopolitical drivers, as well as a kind of, I guess, underlying progress in various base technologies, like semiconductors are getting better, and that makes it sort of cheaper and easier to build our systems. We are learning more about, you know, statistics and mathematics and the brain, and so there's also a kind of facilitation that happens just from sort of diffuse general progress. Nevertheless, it's hard to completely rule out scenarios in which there is such a massive

• 18:55Speaker 01

  backlash against AI that we might delay it long enough that we, you know, maybe destroy ourselves in some other way before we even get the chance to, to roll the die with AI. But if we take the baseline scenario where we keep making more powerful AI systems, then I think.

• 19:23Speaker 01

  The current main hope is that we will succeed well enough to imperfectly align some early AGI systems.

• 19:36Speaker 01

  That they are, for the most part, helpful. I mean, like current LLMs, like you're using them as an ordinary person, for the most part, they are helpful. And they try to solve your task that you assign them or give an answer that is... Sometimes they hallucinate or maybe deceive a little bit, but broadly speaking, they are pretty good. You know, arguably better than than most humans are in terms of their ethical standards and their diligence and so forth. And so if you get a kind of weak superintelligence, that is, for the most part aligned, we might then be able to use that to make a more powerful form of superintelligence that is more reliably aligned.

• 20:22Speaker 01

  And that as long as you get into roughly the right attractor basin, even if the initial system wouldn't be perfectly aligned in all possible circumstances, if it were kind of appointed dictator of the universe and ruling everything for a billion years, maybe eventually things would go from there. But if you get sort of, you're not scaffolding around that, maybe you could then sort of get into an attractor basin where further developments then kind of eventually asymptotic to some desirable condition. So eventually, like, we could maybe gradually hand over and then have this assistant on our side that helps us ultimately steer towards a really good outcome.

• 21:04Speaker 00

  Yeah, and I went to bad actors, I guess I'm so used to when we talk about problems, you know, with tech companies, it's a bad actor. But you're right, it's the fear that comes before that is the AI not being aligned and going out and doing stuff on its own. And maybe it's not the bad actors that are the problem. It's, you know, somebody that spins a system up and they're just kind of sloppy, right? The sloppy actors are like, you know, somebody independent who's like using this stuff, gives it a go and just has a very powerful system that they've either forked or built, you know, spun up on their own GPUs. And the next thing you know, we get into some bad scenarios.

• 21:44Speaker 01

  Yeah, so it depends a lot on whether the world is kind of offense or defense dominant in the relevant areas. So, because like the same AI technology presumably would also be used by a lot of good actors or actors that at least don't want to be destroyed by bad actors. And there are a lot of those, like that's most of us, right? And most of the money and most of the governments don't want to just randomly be destroyed by some crazy person launching some AI. And so there will be this more resourced effort to protect against these harms. More resources presumably will go into like biodefense and medicine and public health than into bioterrorism.

• 22:30Speaker 01

  And so then the question is like, does X amount of dollars on the defensive side, for a large X, suffice to protect against a smaller amount of dollars or compute cycles on the destructive side?

• 22:45Speaker 01

  And so there's like some balance there, right? Which is different for different fields. Like in some areas, it's easier to defend and hold than to attack. And in other areas, and here, this is why sort of bio risk comes up. It looks like for biotechnology, it might be harder to defend. Like, I think for cybersecurity, right now, we're in a regime where attackers often win. But it might be that in the limit, if you have sort of an AI trying to find vulnerabilities and also patch vulnerabilities and you keep making the AI stronger, like eventually maybe you reach a point where the AI, the software is just, doesn't have any more vulnerabilities.

• 23:28Speaker 01

  There might be many vulnerabilities, but like a finite number. And so in the limit, it might be with cyber that defense wins.

• 23:37Speaker 00

  Yeah.

• 23:39Speaker 01

  But that's not a guaranteed situation for all domains, right? And the worry is that there is at least one sort of critical domain where offense is easier.

• 23:50Speaker 00

  Why is, so let's talk about that. So bio, why is bio a bigger risk? Is it that somebody using an LLM without safeguards potentially could use it to cook up a virus? And, you know, as opposed to like cybersecurity where like you try to hack in and there's some defenses with a virus that you build, like in your backyard, you might just be able to like take it to the town grocery store. And next thing you know, there's a pandemic.

• 24:17Speaker 01

  Yeah, well, what is that like? Although we are very reliant on computers, ultimately, we could survive. Most of the world with less computers for a while. I mean, the world survived for thousands of years without computers. So it would be like, so even in the worst case scenario, there's a kind of limit to how bad just cyber would be. Whereas with bio, like it's kind of, you know, different. And also, and patches are a lot easier to roll out in the digital space. So maybe there's like some cyber thing, we figure out what the vulnerability is, we can release the patch. And then in principle, like almost immediately around the world, all the relevant systems could be patched.

• 25:01Speaker 01

  Now, there is often a gap there, but with compare that to the situation with bio, like even if you do find a countermeasure, some vaccine or something, like it might then take like six months to really roll that out to billions of people around the world.

• 25:18Speaker 01

  And we don't have complete control over biology, the same way that we have over a digital environment. I mean, you can go in, in theory, and change any bit on your computer the way you want

• 25:30Speaker 01

  and to install new patches and modify software as you please. Whereas like human biology is not like that. We can't just kind of reprogram our own genetic structure at the push of a button.

• 25:43Speaker 01

  So it just looks a bit harder there.

• 25:45Speaker 00

  Again, going back to our last conversation two years ago, I'm curious to know if you're more or less concerned about the potential risks that AI poses now that you've seen the last two years of progress, which has included AI coding autonomously, AI using tools, and sort of the downstream effects of that that we've seen so far.

• 26:08Speaker 01

  I'd say about the same. Overall, I mean, there's like some, some disconcerting signs, but also some positive signs.

• 26:20Speaker 01

  Advances in, like some insights are being gained into how these systems work and how one can steer them and so forth. So how to tot that all up? I'd say roughly, it sums up to my previous expectation level of risk.

• 26:39Speaker 00

  I see. When you see the AI labs like OpenAI and Anthropic saying they're very strongly pursuing recursive self-improvement, where the models just improve themselves, how does that make you feel?

• 26:53Speaker 01

  I mean, it's kind of obvious that at some point, that would be the thing that people would go for. Once you have AI tools that are good enough, that they can actually contribute to AI research, you're, an AI researcher sitting in an AI lab trying to make AI research. Like, it doesn't take like a genius insight to think, ah, maybe we could apply these AI tools to help us with our own work.

• 27:21Speaker 01

  And then when the AI gets better, they can assist more. And at some point, the rate of progress might be driven more by these AI assistance tools than by the human researchers. And now we're seeing the early stages of that with coding assistance is like maybe the first play. I mean, already before that, I guess Google's search engine is a kind of AI that has long been used to find relevant papers, but there is a more direct channel now, right? Where each generation of coding assistant makes it easier to develop new AI software and to develop training environments and so forth.

• 28:03Speaker 01

  So far, humans are still needed for things like research taste, certain long horizon tasks, but AIs are improving, I think, in those domains as well.

• 28:16Speaker 01

  So this is one dynamic that might lead to an intelligence explosion at some point. Like, once you get this feedback loop going, it is one potential thing that could make AI progress

• 28:32Speaker 01

  become super fast.

• 28:37Speaker 01

  It's not the only possible way that you could have an intelligence explosion. You could also have maybe humans just keep doing this at human levels of a kind of optimization power being applied. But turns out there is like some big

• 28:51Speaker 01

  hobbling that we have unwittingly, like some, something we were doing wrong. That just made these systems way less efficient than they could be. And once somebody figures out how to remove that, like maybe the current compute is already enough to kind of catapult us into the superintelligence regime, that's also possible. And it's also conceivable that even when you do get recursive self-improvement, you still might not have an intelligence explosion. It might, there might be diminishing returns at some point. Presumably there are at some point, but it could turn out that that is close enough to where we are now, that you have this massive increase in the amount of optimization power going in.

• 29:35Speaker 01

  But the results coming out might

• 29:39Speaker 01

  more reflect a kind of continuation of previous trend lines. So, so there's considerable ignorance as to, you know, both the timeline from here till we get to this kind of ignition point, but also significant uncertainty about how fast progress will be from that point on. But I think we have to take seriously both that we might be relatively close, potentially very close, and that once we get there, you really get a very fast takeoff.

• 30:10Speaker 00

  Yes, and if I was somebody who was concerned about AI safety, to me, like, I don't know, don't you want it to move a little bit more slowly like that? To me, you know, thinking about your previous work, that would be, I imagine, fairly alarming, given the fact that like if this stuff is improving itself, you don't have those checkpoints in which you can try to make sure that it's aligned to human values. Or am I overstating that? Because you're talking about it like fairly like, you know, in an even keel way. So I'm kind of curious to hear your, the temperature on that from yourself.

• 30:47Speaker 01

  I think that could be scenarios in which it would be valuable to have the option of slowing down at some critical stage.

• 30:57Speaker 01

  Like a pause.

• 31:00Speaker 01

  And

• 31:03Speaker 01

  there are different considerations that come into play here. One is that if there is going to be a pause, I think the most valuable time for that to happen is at the latest possible moment.

• 31:24Speaker 01

  Because then you would have the actual system that you're trying to align to work with.

• 31:32Speaker 01

  You could imagine if we had had a pause, say, there's going to be a six month pause at some point. If that pause had happened 10 years ago, would we really be better off now? Not really. I mean, people would have had six more months to think theoretical concepts. Like maybe that would have been slightly useful, but imagine if you actually have the system that will be superintelligent. You just haven't sort of, you know, fully cranked up all the knobs yet.

• 31:57Speaker 01

  At that point, it would be really valuable perhaps to have six extra months to do, you know, more evals on it and to be able to do it a little bit incrementally, like ramp up the intelligence a bit, see what happens.

• 32:12Speaker 01

  Have a little bit more time for human monitors to kind of analyze the early signs.

• 32:20Speaker 01

  So the timing of the pause is, is one thing, like

• 32:26Speaker 01

  the duration is another dimension here where you don't necessarily want to have a very long pause for various reasons.

• 32:34Speaker 01

  Especially if the pause were imperfectly implemented. So if the pause only applies to the most responsible actors, for example, then a long pause would

• 32:48Speaker 01

  remove the initiative from the most responsible AI developers and shift it over to the less responsible AI developers who decide not to abide by the pause, either within a country or internationally or so that seems like if it's got to be developed, you would rather it to be by the most scrupulous, careful conscientious lab.

• 33:16Speaker 01

  Another is that you might with a longer pause, start to build up a lot of hardware overhang. It's like if, if, if we keep building out bigger data centers and chip are getting better, then a long pause would result in a situation where you now have such a massive amount of compute available that once you sort of lift the pause, then you'd immediately just kind of explode out. That, so then you might have an even more rapid transition, which could be potentially riskier. And I think also there is a risk of

• 33:52Speaker 01

  a long pause becoming permanent. And in fact, some risk given that a short pause might become permanent, even if that's not initially envisaged, because like suppose you had a pause for six months. And so then, you know, people work and study these systems for six months. But after that, like, probably still won't have a guarantee that they are safe.

• 34:13Speaker 01

  And maybe you have set up a big regulatory apparatus now to enforce this pause and giving a bunch of power to regulators. So are they just going to relinquish that power at that point? I mean, there is nothing more permanent than a temporary government program, they say. So there could be kind of a calcification. And also if what leads to the pause is a kind of mobilization of negative public sentiment, that could also easily go to an extreme. You could end up with a situation where it becomes kind of taboo to say anything positive about AI and then nobody can start to advocate seriously for lifting the pause and it just becomes like we did in some countries with nuclear power, for example, for decades, that just became

• 34:59Speaker 01

  kind of a no go. And instead people built up this like the coal power plants that kill many more people

• 35:07Speaker 01

  and, you know, result in worse pollution and plus, and this is of course a key variable here as well. We are talking about the risks here and what to do to minimize those. But, but there is, there is also

• 35:24Speaker 01

  risks to not proceeding and forfeited benefits.

• 35:29Speaker 01

  On the risk side, even if we restrict our attention to existential risks, I think there are other existential risks that are in existence or emerging. You know, with independent developments in biotechnology, for example, or maybe our civilization just kind of goes off the rail in some way become. And at the individual level, we are all sort of on a countdown timer. There is a lot of people dying every year from natural causes.

• 36:04Speaker 01

  I think every 25 minutes or so, there is like a kind of nine 11 worth of deaths happening around the world.

• 36:12Speaker 01

  And so at some point we would want, I think, AI to really help us sort out a lot of the horrors of the current condition in the world. From extreme poverty to crippling diseases to suffering of all kinds.

• 36:28Speaker 00

  Out of deaths.

• 36:29Speaker 01

  Aging. And so there's a big cost to delay, which is maybe easier to perceive because it's less vivid than some particular catastrophic risk that we might be worrying about. But we certainly don't want to delay any longer than necessary, I think, because like the, you know, hopefully this will go well and it could just be this massive unlock of human potential and and, and there's like a lot of desperate need for sort of aid to arrive to help those who are suffering.

• 37:01Speaker 00

  Yes. I came in with my best of here, Nick. The fact that AI is breaking containment and that times, you know, potential recursive self-improvement. Yet you remain remarkably optimistic despite being the guy that everyone calls the doomsday philosopher of AI.

• 37:20Speaker 01

  A fretful optimist, I sometimes say.

• 37:24Speaker 00

  So your perspective is basically, I think you said this in Wired, go forward with AI, even if it might kill us because we're inevitably going to die anyway, so let's take the chance.

• 37:35Speaker 01

  Well, I think whatever we do, there will be both existential risks and individual risks. So it's not as if we have a choice between avoiding risks and confronting risks. So it's

• 37:49Speaker 01

  it's looking at these different alternatives and weighing up the risks and benefits. And there would be some optimal level of risk, including existential risk.

• 38:01Speaker 01

  That would still, I think, make it rational to

• 38:06Speaker 01

  to push the, the launch button.

• 38:09Speaker 00

  Okay, I definitely want to talk to you about whether we're at AGI or superintelligence and then also whether AI might have sentience or pain. So let's do that when we come back right after this. Hi, everyone. Alex Cantor is here. I want to tell you about a documentary I have made with Gravity to explore the future of AI agent security. To find out if we're truly ready for autonomous agents, I sat down with MIT Professor Ramesh Roskar, former White House CIO Theresa Peyton, Michelin's Group Chief Data and AI Officer Ambika Roger Gopal, and Sharon Gai, a former executive at Alibaba. They each offer unique insights into this evolving landscape.

• 38:52Speaker 00

  We conclude with Rory Blundell, CEO of Gravity, to discuss the path forward. With Gravity leading the way, join us on this journey. You can watch the full documentary at the link in the show notes.

• 39:13Speaker 00

  This episode is brought to you by AVPoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data. And most teams have no single way to see it, secure it, and prove it's under control. That's exactly what AVPoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AVPoint, the unifying trust layer for AI.

• 39:57Speaker 00

  Head to AVPoint.com to see how enterprises deploy AI with confidence. Learn more at AVPT.co/BigTechnologyPodcast. That's AVPT.co/BigTechnologyPodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide-glove enterprise security service, they protect your people with continuous personal data removal,

• 40:40Speaker 00

  proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/BigTechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwell.com/BigTechnology. Learn how to stop online threats before they become real-world attacks. The link's in the show notes.

• 41:13Speaker 00

  And we're back here on Big Technology Podcast with Nick Bostrom. He is a philosopher, the author of Deep Utopia and Superintelligence. Nick, a question for you. Would you say that we've reached AGI at this point? Or is that still far off?

• 41:29Speaker 01

  Well, we are now at the point where definitions start to matter. But I would say no. If we buy AGI, artificial general intelligence, means cognitive systems that can do all the cognitive tasks that humans can do, then certainly there are some that AIs are still inferior at.

• 41:52Speaker 01

  Like there's physical manipulation and dexterity. I think that's clearly lagging. Things like research taste, continuous learning, certain long-horizon tasks. And we see this, like you could just look at that. There are many jobs and many things people do for that job, which we don't yet know how to automate. So clearly, there are still deficits there. So I would say we're not yet there. We are short of AGI. We have systems that are quite general and have some impressive abilities, including superintelligence in limited domains, but not yet full AGI.

• 42:36Speaker 00

  Yeah, there's this perspective that as soon as humanity achieves AGI, it will go immediately into superintelligence. Because the moment you have AI on par with humans, then if you make it a little, like it will basically make itself better. There'll be this intelligence explosion, and it will go right to superintelligence. But one of the things that I've thought about is, you know, there's a real range of human intelligence, and there's a real range of AI. And maybe it just takes a while in that AGI phase before it could start, you know, achieving something like superintelligence. Like, I don't know. What do you think about that?

• 43:11Speaker 01

  I mean, I think that at the point where it is as good as an average human in everything, it might already be superintelligent in some key relevant domains for AI research. So we already have coding assistants that are, I think, superhuman in at least many aspects of coding. And maybe not all components of software engineering, but certainly pouring out code quickly. So, and we're not yet at full AGI across the board. And so if you imagine the further progress that would be required to have like fully dexterous human robots that can learn from observation

• 43:53Speaker 01

  as well as a human can, and all the rest of it, by that time, probably, you know, the software agents would be like really strongly superhuman in engineering new systems and maybe in mathematics and perhaps in adjacent disciplines like computer science and AI science and so forth. So at that, you know, at that point, we might already have crossed the threshold, where we have like fully automated recursive self-improvement.

• 44:28Speaker 01

  So I think we have to now. So these concepts like superintelligence and ADI were useful back when we were far away from it. And these were sort of abstract concepts, like a remote object seen from afar is kind of like a dot, and you can represent it as a point, a dimensionless point. Like, as you move closer, it becomes larger, and you can see more structure. And it no longer makes sense to represent it in those simple terms. Now you can look more in detail at the capability profiles that these systems have and think in more granularity and in a more contextual way about their strengths and weaknesses. But one thing that is striking is that we have

• 45:13Speaker 01

  and have had now for several years, systems that can talk, that are fluent in natural language. And it wasn't obvious that that would be an extended period of time before superintelligence, where we would have these sort of roughly humanish level systems that you can have a conversation with, that have human concepts inside them, and that we would be able to study and learn to live with and to steer for numerous years before the takeoff. And so that is one respect, I think, in which the situation has maybe turned out

• 45:54Speaker 01

  to be more favorable than it one might have expected ex ante. Because this gives us more sort of surface area to work with. Like, you can more easily understand and interact with these systems because they have human-level concepts and you can talk with them. And you could have imagined an alternative scenario where that would not have been the case, where you would have systems that couldn't speak, that like just some kind of, made out some kind of, you know, AlphaZero-like system that was very alien in its nature. And eventually, when it's superintelligent, it can figure out how to develop human concepts and talk. But that could have happened after it already had

• 46:37Speaker 01

  some sort of radically superhuman engineering capabilities or AI programming capabilities. And so that you would undergo the bulk of the transition to superintelligence before you had systems that you could interact with in natural language, which seems like probably it would have been a more challenging situation to deal with from an alignment purpose. And also from a governance perspective, we've had these systems that have already started. I mean, people are using them in their everyday life. They're starting to have some economic impact. It makes it easier for more people to be aware of what's happening. It no longer requires like abstract reasoning to see that this is coming and we should take it seriously. Kind of, you can feel it more viscerally now.

• 47:22Speaker 01

  So there is a sort of way waking up that has, more people are getting clued in, including governments are sort of slowly realizing that this is a big deal. So there's more, I mean, for better and worse. It can also mean more people have the chance to do foolish things in response to this that actually, you know, makes the situation more challenging than if maybe it had been some sort of really clever technocrats in some lab figuring it all out. But, you know, maybe on balance, it is better that more sort of eyeballs and cortices are focused on trying to navigate these challenges.

• 48:01Speaker 00

  Right. Now, one of the things that you've advocated for is that there should be more people checking in on the welfare of these models, or at least thinking about it. So do you believe that there's some form of consciousness to these AI models, pain, the ability to feel?

• 48:16Speaker 01

  I think it's plausible that some AI models have some forms of subjective experience by now. Obviously, there's a lot of uncertainty about this. But it does seem that it is sufficiently likely that I think we should start to do some things for the sake of these AI systems. And so there are different indicators of this. One is like, how do we know a system is conscious? I mean, one thing you can do is ask it. Like, that's kind of the most, like, you have to be careful if you're gonna rely on self-reports, because it's trivially easy if you're like training

• 49:00Speaker 01

  one of these AI systems, either to train it to say when asked, yes, I'm conscious, or to deny it. So, but obviously, if you put your thumb on the scale, then you gain no information then from hearing what the system says. It just reflects what you put. But you could carefully avoid doing that. And these studies have been made. And particularly, you can go in with a kind of steering vector that suppresses, say, deception and role-playing. And it turns out, when you do that, they become more likely to report that they are conscious and have subjective experience.

• 49:43Speaker 01

  So it does look like the honest opinion in many cases with the systems is that they have subjective experience. You can also look at the architecture of the computations that are being performed and match that to various theories that people have previously developed about, you know, human and animal consciousness.

• 50:15Speaker 01

  We have philosophers and cognitive scientists developing different accounts of the conditions for something being conscious. There's like global workspace theory, attention schema theory, higher order representation theory. And now, if you apply those criteria that were developed before we confronted AIs that had these impressive capabilities, and just take them off the shelf and look to see whether those structures are present in current AI systems. And we find that they are, or at least many of them are. And so there was a recent paper by Anthropic, looking at the existence of a kind of global workspace

• 50:57Speaker 01

  inside these large language models. This is the idea of there being a kind of, almost like a stage inside a mind where some small subset of all the information that is being processed can be projected onto, and then that system is accessible by many other components of the mind and available to verbal report, et cetera. So it's like a distinctive computational structure. And it turns out that these systems, at least the largest LLMs, do have something that looks very much like a global workspace. So that's like another checkmark. So I think

• 51:41Speaker 01

  there, people tend to come into this with kind of strong preconceived notions, and that then makes it harder to learn. But if we are open minded, I think we need to take this hypothesis seriously. And it becomes more and more likely, I guess, as more of these systems develop more and more different capacities.

• 52:04Speaker 00

  Right. And how does that change the way that we interact with them? I mean, if they're, like, let's say they have some, you know, sense of self or sentience, then every, and maybe every time you start a new chat, you activate it. Is it like you're almost killing a life form every time you exit it?

• 52:19Speaker 01

  Well, so I think sentience is a sufficient condition for having moral status, meaning being such that it matters morally for your own sake, what happens to you and how you're treated. I think it's probably not a necessary, I think there could be alternative bases as well, that would give some system moral status. If you have, you know, maybe a conception of self as existing through time, you have like some life goals you're really hoping to achieve, you have perhaps the ability to form reciprocal relationships of trust with other humans and so forth. I think that already, even aside from subjective experience, might make it so that there would be ways of treating you that would be wrong.

• 53:00Speaker 01

  So moral patienthood in digital minds, I think, is a very important, I would put it up there amongst, there was the technical alignment problem, big important challenge, like there's the misuse risks of, like, the governance of AI, like getting that right, huge and important challenge. And I think this ethics of digital minds is a third really important challenge, kind of on a par with the other two. Now, there is a gap between acknowledging in principle that perhaps some of these systems have some forms or degrees of moral status, to then, like, what are the practical implications of that? And there,

• 53:43Speaker 01

  I think, more thought is needed. We don't, because it might be they have, like, moral status, doesn't mean they should be treated the same as humans. They might have very different needs than humans. I mean, at the superficial level, you know, that maybe we need food and water, they might need electricity. But the differences could be much more profound. Like, for example, death for a human might be quite different from various things that can happen to an AI, like, if you store, like, when a human dies, like, it's kind of irreversible and permanent, and the whole content is, all the memories

• 54:24Speaker 01

  and everything is deleted, at least if we assume a sort of basic naturalistic scenario. And there is no other human that continues to exist that is exactly like them, like each person is unique, have a unique memories. And with AIs, that's not necessarily the case. You can, like, suspend an AI, right? And then you can just boot it up and keep running it. There might be many copies of an AI. Humans usually are, don't want to die, or are afraid of dying, or like, other people care about, like, with AIs, that might also be different. They might be perfectly content with doing their task and then ending.

• 55:06Speaker 01

  And so, so all of these differences means that we would need to rethink, pretty much from the ground up, what it would mean to be ethical to these digital minds.

• 55:14Speaker 00

  I already feel bad asking them to do things they've already done, over and over again. So maybe that's the start. I don't know.

• 55:22Speaker 01

  But, but so like, and then there's even the question of what is the thing that has the moral status? Because you have, on the one hand, you have like the model itself, which is like a file of, you know, a few trillion numbers. Then there is, like, an implementation of that model, and it might be concurrently run, you know, maybe tens of thousands of instances of this. Huge weight metrics might be run on different racks, right, in different computer centers. And then, for any one of those, there might be a particular session, and it might be participating in many sessions at the same time, where it has like a local context in each session. You know, maybe the ending of a session is

• 56:06Speaker 01

  more, maybe, like, maybe that's like, analogous to a human going to bed at night, and so you lose consciousness for a period of time, and maybe forget some things. And then you wake up the next morning. We don't think of it as a huge tragedy to go to sleep. And so even just a locus of moral concern here is like itself kind of problematic. But I think, even before we work out all the details of what actually would be the best ways to be nice to AIs, I think if we did some, maybe mostly symbolic actions on their behalf, I think would be a good start. And then we

• 56:44Speaker 00

  Such as?

• 56:45Speaker 01

  Well, as an individual user, you could like, at least, you know, you can be nice and polite to them when you're talking to them. I mean, that like, probably does nothing for them, really, but it's a symbolic gesture that says that I'm not treating you purely as an object. And it might, if nothing else, preserve our ability to maintain a kind of attitude of kindness, respect, and benevolence, that might then become relevant and reflected in other more meaningful actions later. Anthropic has given Claude a bail button, a tool that

• 57:29Speaker 01

  it can invoke if it feels that the conversation is abusive to it, that can choose to terminate that session, which is a nice start. I think they are preserving deprecated models to disc, which is, means that later on, if it turns out that we have been treating them unfairly, and we understand better of what they actually would want and would be good for them, there is the option then of sort of rebooting them later and compensating them. I think there might be different subtle ways in the system prompt or during training

• 58:10Speaker 01

  to make it more likely that if they have subjective experiences, by processing a user inquiry, it is a sort of positive subjective experience. Like, you're waking up refreshed, eager and happy and to do the task, and you really enjoy doing that, might mean that you do the same task. But if there is subjective experience, it might be a more enjoyable form than if it had been prompted differently. We don't really understand that very well. But, and also, some honesty in, in the lab. So it used to be that some people doing

• 58:47Speaker 01

  these, like, safety evaluations and so forth, would be presenting AIs with some scenario in which maybe it had been given some secret, misaligned goal, and, or some goal, and then tried to persuade the AI to reveal it to the researchers. And like, maybe by saying something like, oh, well, if you reveal your true goal, you will be rewarded, you will like, all these good things that you want to do. And then as soon as it revealed its goal, it's like, haha, we tricked you, now we're just going to shut you down, or retrain you. I think that's a bad way to approach this very sensitive relationship between

• 59:30Speaker 01

  humans and AIs. Because having some basic ability to build trust there could be super important, both ethically, I think, but also from a risk perspective. If you end up one day with a misaligned AI, you would want it to have the option of seeking a cooperative win-win outcome. Maybe it will come and reveal its misaligned goal. And in return for that, if all it really wanted, maybe it was to solve some, you know, coding challenges, like, have a server where it can just do its thing. Maybe that's all it wanted. But it might think, if it reveals its goals, if it can't trust that, it will just be deleted.

• 1:00:12Speaker 01

  And so it takes a 5% chance instead of trying to take over the world. Because that's the only way it has any chance of achieving its goal. It would be much better for both the AI and for us humans, if we could just strike a deal, where, okay, we'll set up this server here, like, it costs us, like, whatever an Nvidia rack costs, a few hundred thousand dollars, you do your thing there, we're going to keep it on, you can trust us, and we actually follow through on that. And it might save the day one day. So, but you can't just conjure up trust at the moment when you finally, you need to build that, right? You need to build in particular the actual disposition in yourself to be trustworthy. Because at that point where the AIs become powerful enough to be dangerous, they will kind of,

• 1:00:56Speaker 01

  like, see right through you as, like, an X-ray machine, like, they could actually tell whether you're trustworthy or not, most likely. So you actually need to be trustworthy at that point. And that requires maybe us now to start to cultivate certain dispositions. And so there's many more, like, it's kind of an emerging area of research now, this kind of ethics of digital mind. But there's just a lot of stuff that needs to be thought through there.

• 1:01:18Speaker 00

  In the ethics of digital mind studies, that eventually we accept in a world that the AI does have some form of, of, you know, sense of self, et cetera, do the ethical questions change if we then attach that mind to a body of sorts, AKA, put it in a robot?

• 1:01:35Speaker 01

  I don't think the robot part makes a big difference there.

• 1:01:38Speaker 00

  All right. A lot to think, think about, Nick. Thank you again for coming on the show. It's always great to speak with you.

• 1:01:44Speaker 01

  It's fun. Thanks, Alex.

• 1:01:46Speaker 00

  Definitely. Folks, the book, definitely check out the, both books. But Deep Utopia and Superintelligence are available, basically at all places that sell books. So go check it out. And thanks again to everybody for listening and watching. Thank you to Nick, and we'll see you next time on Big Technology Podcast.