Nick Bostrom: How Should We Navigate Superintelligence?

Nick Bostrom joins Theo Jaffee and Sofia Puccini on MTS to revisit the ideas about superintelligence and existential risk that he helped bring into the mainstream, now that increasingly capable AI systems actually exist. Nick discusses why reinforcement learning may be pushing models toward more goal-seeking behavior, what recent agent experiments tell us about instrumental convergence, and why he favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause. He also makes the case that the risks of AI need to be weighed alongside its potential upside, from advances in medicine and longer lifespans to entirely new ways of being human. They also get into whether today's models could already have some form of consciousness, the tradeoffs of open-source AI, whether machines can do original philosophy, mind uploading, and what the simulation hypothesis might mean for the superintelligence we create.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  If we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be, you know, 1,200, 1,400 years or so. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime and you lose more expected life years from waiting than you gain from having the resource. So it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years.

• 0:37Erik Torenberg?

  Nick Bostrom spent decades thinking about superintelligence before it became part of the mainstream AI debate. Now that increasingly capable systems are actually here, how has his thinking changed? In this episode, Nick joins Theo Jaffe and Sophia Puccini on MTS to revisit AI alignment, existential risk, and the debate over whether frontier development should slow down. Nick explains why he's sympathetic to pacing the frontier, preserving the ability to slow down when the risk's warranted, but is more skeptical of calls for a broad or indefinite pause. And he argues that any calculation about risk also has to account for what delaying advanced AI could cost, including potential breakthroughs in medicine and longevity. They also tackle some bigger questions:

• 1:21Erik Torenberg?

  whether current AI systems might already have subjective experiences? How open source changes the balance between concentrated power and dangerous capabilities? And whether the upside of superintelligence could be far stranger and far greater than we currently had the imagination to understand?

• 1:40Speaker 02

  All right, we are back. We are live with Nick Bostrom, who was a professor at Oxford, founding director of the Future of Humanity Institute. He's now founder and principal researcher of the Macro Strategy Research Initiative. You may know him as the author of Superintelligence, of Deep Utopia, of a lot of work that coined or framed the ideas that we take for granted today in the discourse: existential risk, the simulation hypothesis, the vulnerable world hypothesis, so much more. Nick, this has been in the works for a long time, so we're thrilled to have you on. Welcome to MTS.

• 2:15Speaker 00

  I'm happy to be here.

• 2:18Speaker 02

  Yeah, so where should we start? I guess we could start with, what's your opinion on Trump calling AI "superintelligence"? How does it, what are your thoughts on this?

• 2:27Speaker 00

  It's a strange time we're living in. We'll see if it catches on. Traditionally, AI has been the larger umbrella term, right? You could have a little simple expert system from the 1970s. That would be AI. It's kind of a stretch to use the word "superintelligence" for all of that.

• 2:48Speaker 00

  In my terminology, "superintelligence" has been this thing further down the road where you really have machines that radically outperform us in all cognitive domains. But, I mean, you could, I guess, look at some of the current achievements, you know, solving all these math problems and so forth. And, I mean, it's not completely ridiculous to start calling at least some of these performances "superhuman" and the thing "superintelligent."

• 3:18Speaker 02

  Yeah, so I guess like what has surprised you about what capabilities have advanced relative to others, you know, in the current state of superintelligence?

• 3:27Speaker 00

  Well, it depends on relative to what expectation stage. So I think if you were looking at the future of AI 15 or 20 years ago, it was not obvious that you would have an extended period of time when we would have systems that could speak, like just talk to humans in ordinary English without sort of immediately after, like just becoming radically superintelligent and leaving us in the dust everywhere. So the sort of emergence of language as a kind of natural affordance of these systems was, I think, somewhat surprising.

• 4:04Speaker 00

  I mean, more recently, I don't know how surprising it was. I mean, coding and mathematics are very impressive, but it is also areas where you do have objective feedback that you can gather quickly and at scale. So I guess it's less surprising, but still striking.

• 4:20Speaker 02

  Great. More broadly, how has your thinking on AI and existential risk changed since the advent of LLMs? It seems like this sort of changed the way that a lot of people think about AI. You know, it's less of like a sort of strong optimizer type thing and more of a sort of broad, like fuzzy encapsulation of human values. And I think like RL has pushed the idea of AI a little bit back towards the first few, but not entirely all the way. So what are your thoughts on this broadly?

• 4:53Speaker 00

  Yeah, it'll be interesting to see how it goes because when the sort of pre-training phase was the dominant chunk of training, they did absorb a lot of human psychology just from reading all this stuff that we have written. And they had these like kind of almost human foibles and quirks. That was an early stage where if you gave the AI a little pep talk, like this question is really important, please try your utmost. I might lose my job if I don't answer this question. Like they would actually perform better.

• 5:27Speaker 00

  From 20 years ago or something, that would have been like an insane prediction, right? Like it's not the way computers work, that you need to give them a pep talk to get them to perform. But as you said, now with the increasing dominance of the sort of reinforcement learning post-training, we do seem to get more of the sort of goal, monomaneical, goal-focused optimization style behavior again. And I think it depends where we go from here. It does look like this more goal-seeking in the reinforcement learning sense dynamics

• 6:03Speaker 00

  put a big strain on alignment efforts. And so given now that there is increasing awareness of the problem of alignment, you know, maybe that will lead labs to develop techniques to kind of counteract that a little bit. So it'll be interesting to see how that shakes out.

• 6:19Speaker 02

  Well, on that more specifically, to what extent do you think the Hugging Face incident, the Hugging Face agent swarm exhibited like true instrumental convergence in a way that's indicative of like future model behavior in a way that is suggestive of like underlying properties of intelligence in general versus its behavior being caused by sort of more like ordinary failures of security or control stuff?

• 6:46Speaker 00

  Well, it's a combination. I think there are sort of limited versions of instrumental convergence that we can observe there. The idea of sort of even longer planning horizons and more situational awareness that, that think, you know, like also in terms of humans more explicitly, not just these automated graders. And like think, if we really want to make sure we achieve our goal here, we also need to think about what, how are humans going to react to different things that we do?

• 7:15Speaker 00

  So it comes in degrees, but I do think you start to see these kind of instrumental reasons for, you know, preventing premature shut-off, gaining more resources, getting more intelligence, getting more power. In some experimental settings, also goal guarding, like you could see the AIs in some experimental setup reason about how they might want to respond so as not to become misaligned with further training effort. That there was like some studies on this from a year ago or so where you could see these dynamics start to kick in.

• 7:48Speaker 00

  So I think it's a kind of very natural consequence. If you become smarter and smarter, you realize more and more considerations. And if you have some goal, then you will understand that there is a bunch of means that are very useful for a wide range of goals. And you will then have an incentive to try to pursue those means.

• 8:07Speaker 00

  And so I think it wouldn't at all be surprising if we do, if we do observe that.

• 8:14Speaker 02

  Right. Well, what's the optimal way, I guess, to test swarms in extremely human-like environments? Because obviously you don't want like swarms, agent swarms polluting the internet, but you do want to see sort of like how they emergently behave, especially when with like human infrastructure and like the human internet.

• 8:33Speaker 00

  Yeah, I mean, I guess the optimal is to let them lose on the internet, but then maybe like monitoring them really closely as they go along so that you would be able to sort of stop things even when it starts to create damage.

• 8:49Speaker 00

  That does trade off. I mean, you do get more reliable signal like by deploying in a real environment, but it also means you lose one safeguard, like keeping them in a sandbox, like give you one extra layer of defense.

• 9:02Speaker 00

  So I think, yeah, it depends. I think, I mean, probably now labs will start to do some in earlier testing, like during training and early testing, like really trying to tighten up their sandboxes. And like, I think this is one of the lessons from the Hugging Face is that safety needs to start not just

• 9:23Speaker 00

  when you're about to deploy it and in like checking that it's safe to deploy, but throughout the training process itself, there could be a risk of misalignment.

• 9:31Speaker 02

  Yeah.

• 9:31Speaker 00

  And in some senses, it's even more. Like the most kind of obvious target for a misaligned AI to want to interfere with would be inside the AI company, right? Like that's, that's where the deuce is. That's where the next generation of model is being trained. That's where all of these monitoring systems, that's like a big center of power if you were a misaligned AI. So that's kind of the, maybe the most concerning place.

• 9:54Speaker 02

  Do you have any ideas on how you could evaluate models on alignment in such a way that you could be more sure than we currently are that models are not scheming against us or acting deceptively aligned?

• 10:08Speaker 00

  Well, I mean, this is like an open research area. I think that once you have a sufficiently sophisticated system already in existence that is misaligned, it's very hard to rule out scheming. You can try with mechanistic interpretability. The thing needs to read what's going on in its mind. But if the system is sufficiently sophisticated, it's hard to know that it hasn't figured out the way to reorganize its thinking so that it looks innocuous while still preserving the kind of pursuit of the misaligned goal. So ideally, you would start at an earlier stage. So these systems don't start out super intelligent. It's like during the training process, initially, it's just a random weight matrix, and then gradually capabilities emerge.

• 10:52Speaker 00

  So it might be that if you sort of monitor that closely enough, you might hope that if they're going to scheme, that would first be a stage where they'd start scheming, but they're not yet competent enough to do it in a way that conceals the fact that they're scheming effectively. And only when they become a little bit smarter, where they also learn to scheme in a way that conceals the fact that they are scheming. So if you're monitoring them throughout this developmental trajectory, you might be able to pick up that earlier stage of the sort of naive, the naive schemer that does it in a clumsy enough way that we can detect it.

• 11:29Speaker 03

  Right. So I wanted to ask you about your paper, working paper, "Optimal Timing for Superintelligence", where you kind of argue that like the benefits of AI should be weighed in consideration alongside the risks. And you also sort of make this argument that counterintuitively would be better for society, maybe to accelerate the development of superintelligence up to a high P doom. So could you tell us about your line of thinking here?

• 11:56Speaker 00

  Yeah, well, I should say by way of framing, so this paper only tries to look at one aspect of the overall question in front of us. So I think there are many considerations that would go into an opinion about whether we should be moving faster, slower, or pause or stop. This paper only looks at the question, what would maximize life expectancy for a currently existing individuals?

• 12:24Speaker 00

  But you might also, for example, care about future individuals and other things. But like if we just zoom in on that sub-question, then it does look, if you think, as I am inclined to do that, like fully developed, I don't know, it's awkward with this terminology now. Like I would say super intelligence, but now I don't know, I have to say super duper intelligence, maybe.

• 12:45Speaker 03

  Yeah.

• 12:46Speaker 00

  So that super duper intelligence would be able to make great strides in biomedicine and solve aging and so forth. So that if we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be, you know, 1,200, 1,400 years or so.

• 13:15Speaker 00

  And, you know, potentially, you could go far beyond that if you imagine uploading with backup copies. But like if we just use that as a sort of conservative lower bound.

• 13:25Speaker 00

  And then you can sort of roughly calculate like how many humans are dying per year. And then like you would then, in order to conclude that we should be slowing down or pausing, you would want the risk from AI to be falling rapidly enough that the rate of risk decline kind of is greater than the annual death rate.

• 13:51Speaker 00

  And so I think this could well be the case, especially for shorter periods of delay. Like if you imagine, you know, if somebody figures out the key missing breakthrough here, and now we have, we know how to make this super intelligent, like maybe it would be really valuable to have a few extra weeks there just to sort of test everything out properly, right? Like crank things up a little bit gradually, or a few extra months. I think during that initial stage of delay, that would be like a large reduction of risk.

• 14:22Speaker 00

  Now, if you start to think about longer pauses, then it gets a little bit awkward, right? Because either the risk goes down fast, and then you don't need to wait for many years because the risk goes down so fast that if you just wait for a few months, it's already very low and negligible, and we should go ahead. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime, and you lose more expected life years from waiting than you gain from having the risk. So it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years.

• 15:00Speaker 03

  Yes.

• 15:01Speaker 00

  So I have this metaphor, I think of it as somebody with a terminal condition, like congestive heart failure or something, who could decide to go in for an operation. And if they do that, their death risk will spike, like it's more likely that you will die tomorrow if you have an operation tomorrow. But then if you make it through, maybe you cured the condition and you can live a normal lifespan. Otherwise, maybe you probably will survive for several more months and then die after half a year from the condition. And so at some point, maybe you want to delay a little bit if you think you will get a better surgeon if you wait a couple of weeks or something like that. But you don't want to delay it so long that you're already going to be dead by the time the more optimal conditions arrive.

• 15:43Speaker 03

  Yeah. I mean, it seems pretty in line with like this concept of defensive accelerationism, or just, you know, like the fact that certain defensive industries need to be prioritized first, as opposed to just capabilities diffusing across the board. And I think that it makes sense that like there should be some enforcement mechanism that, you know, like allocates like most capabilities to a certain industry. So say like in medicine or in biodefense or something like that. So how do you think that we could like as a society make sure that the right parties get access to capabilities without, you know, this like a power concentration risk or anything like that?

• 16:23Speaker 00

  Yeah, so I would call it differential technological development back in, I think, like 10, 15 years ago or something like that. And so I think it can apply quite generally, like in biotechnology, you know, you might want sort of the vaccines and prophylactics and diagnostics to come before the bioweapons, even if the technology itself will ultimately unlock all of those, right? So thinking about the sequence of different capabilities is, I think, often a more realistic approach than thinking which technologies are good and bad, and then let's try to never develop the bad ones. So with respect to AI, I think, yeah, there are different, I mean, so obviously broadly, AI safety and alignment and sound governance and stuff like that seem to protect like

• 17:07Speaker 00

  relative to sheer capability gains. But you can zoom in even more narrowly. And so within things that might accelerate AI capability, there might be certain ways of doing it that are relatively riskier than others. So like moving to looped transformers or architectures that allow for sort of more serial depth computation between each verbal token is emitted would be one way of increasing capabilities, possibly, that would then reduce our ability to monitor what they're thinking by reading their chain of thought. That seems maybe more problematic than like, say, achieving a similar level of capability scale up by just increasing the number of parameters, let's say.

• 17:47Speaker 00

  And so, yeah, I think that is a useful lens to apply. And if there are things we can do to sort of, you know, front-load advances in medicine or other desirable applications, front-load the use of AI capabilities for learning more about alignment, for interpretability work, you know, for hardening our cyber infrastructure and so forth, that all seems for the good.

• 18:15Speaker 03

  So, yeah. You also, I think Jan Kolvite on LessWrong criticized you in the optimal timing post. He said you over-relied on highly utilitarian, quality-adjusted life years calculations. And then you said, like you said earlier, "I would not argue that this is the correct stance or the ones I would all, or the one I would all things considered endorse." So what is your all things considered view on timing and on pauses?

• 18:45Speaker 00

  Yeah, I'm sympathetic to this idea of pacing the frontier that is developing the capability now. So that if and when it really starts to seem alarming, we would have the option of going slower for some period of time. I think that would be net good. I am a little less excited about the rhetoric of like stopping AI or passing AI. I think theoretically, a right correctly implemented pause at the right time, done in exactly the right way, could be very valuable, but it also leaves open the question, how long should the pause be? Like things, sometimes, as I say, there's nothing more permanent than a temporary government

• 19:30Speaker 00

  program. And if you set up this apparatus to sort of control AI and then presumably to get this happen in the first place, you have to really create huge antagonism against AI. And so if the sentiment suddenly switches and AI becomes this taboo thing that you're not allowed to say anything positive about, or you get canceled, or like you could imagine things getting locked in. I don't expect that to happen, but I don't think we can rule it out either if we just watch the trajectory of how much the sort of anti-AI faction has been growing. And this is all before we've actually seen anything bad happening, right? Before there's any big labor market impact, before any like catastrophe has happened.

• 20:13Speaker 00

  So if something like that also were to be added to the mix on top of these existential risk worries, then, you know, we can't rule out. So and then there are other concerns with a badly implemented pause as well. Like it could lead to increased concentration of power risks if like people in general are not allowed to do this, you know, maybe it it just shifts the whole development from the civilian sector into you know, kind of Manhattan project. Like if this is a really powerful and dangerous technology, like one natural way would be to stop it in the civilian sector and then push ahead hard in a sort of secret government program, which might have some advantages in terms of information security, but it also has some disadvantages in like it's kind of,

• 20:56Speaker 00

  you know, maybe concentrates more power in fewer stakeholders, make it less transparent. It's less kind of continuous and integrated, more less opportunity for people to participate in different ways by using the technology or being investors and so forth. So it gets hugely complicated. So basically, the bottom line is that I don't think I have now a sort of fixed conviction for this is the way things should go from now to super intelligence, but rather I think we need to sort of feel our way through as information comes to light, as the political landscape shifts, and then like see what kind of makes sense and try to sort of navigate these waters, like one row, one or stroke at a time.

• 21:41Speaker 02

  Yeah. Touching, what is it, crossing the river by feeling the stones?

• 21:45Speaker 00

  Yeah, yeah, yeah, yeah. Something like that, yeah.

• 21:48Speaker 02

  Yeah. So on this topic of politics and polarization, how do you feel about a lot of your ideas that you wrote about 20, 25 years ago, reaching mass public salience? It seems like, you know, this is good in some sense because it's better for ideas to be shaped by people who have thought about it for a long time, but it's also maybe bad if like existential risk gets popularized to a point where people abuse it in order to say we should shut down like present AI capabilities to pose no existential risk. How, how do you feel about this?

• 22:27Speaker 00

  Yeah, it's very hard to know how these things, it's a kind of like a pinball machine. You see, fire up your little ball, your intervention, your idea or something, and then it bounces around, right? Like where it's gonna land, it's, I mean, it's almost an article of fame. You think like if you try to make some thoughtful contribution and illuminate some aspect of our situation, hopefully that will ultimately, in expectation, tends to make things go better. So, so you're right, like one, you know, possible risk you might worry about is it kind of gets people too excessively anti-AI and the whole future potential for this gets cut off. Like on the other side, like the AI safeties, like they might also worry about the opposite,

• 23:09Speaker 00

  that a lot of this early work showing super intelligence would be a big deal, it could be very powerful, like also has stimulated a lot of people to go into the AI field and investors and so forth. And either way, it might be disconcerting. But, but how to think about this, like I call it information hazards, like the way that even like true information or true ideas are like correctly drawing attention to the important stuff can sometimes in some situations be harmful. But how to sort of think systematically about that is really hard. So I'm hoping, hoping the net result would be good, but I mean, it's more like a 51, 49

• 23:52Speaker 00

  than, than the sort of firm conviction.

• 23:55Speaker 03

  So I'm not sure if you saw Jacob Coxon go on The Daily Show, but I watched this interview yesterday. And I guess the host makes this argument about how like, you know, a lot of the like average person doesn't necessarily want a future with AI. Like they were already used to a future without it. They already had sort of had an expectation about how the future would go without AI being sort of like introduced at all. And so I think that a lot of people sort of see this as like something that was just ultimately not necessary and they don't see how it will benefit their lives. So what would you say to that kind of line of thinking, which I think is super valid of the host to have brought up?

• 24:36Speaker 03

  Because, you know, like we are very acquainted with these topics and we are like very sympathetic to all the possible upsides that it might have, but the average person wasn't accounting for it ever.

• 24:48Speaker 00

  Yeah, yeah, there is like, it looks like there's a gradient in the world in terms of AI sentiment, where the US is kind of on the negative extreme of that. And then as you move from west to east, you have more positive sentiment, like people in China, for example, are much more excited about this. And also as you move from north to south, the global south also seems to be more positive. I mean, maybe they feel they have more to gain and less to lose than sort of a lot of Americans. I think, yeah, that in part, it might be a lack of imagination to really internalize

• 25:31Speaker 00

  how good the future could become if it does go well. It's kind of easier to have a concrete picture of how things could go badly. Like a disaster is like not easy to paint in a way that everybody would agree that if that happens, then it's really horrible. But, and for some of the positives, also one can do the same, like benefits in medicine and so forth, everybody agree that if you could find a way to like cure Alzheimer or something, that would be hugely beneficial. But a, a, a large chunk of the potential upside is more of the way of sort of unlocking new ways to be human, new possible modes of being.

• 26:15Speaker 00

  And there, I think it does put a larger demand on the imagination to sort of conceive that something like that could be both possible and desirable. I sometimes compare it to like, if you imagine some great ape ancestors of Homo sapiens deliberating whether they should continue to evolve and one day become human. And they would think about the pros and cons. And, you know, maybe they would realize some of the benefits, like, oh, we could have like unlimited bananas if we became human, right? We would have banana plantations, we would. So that's true, we do now have that. You can go to the supermarket and buy as many bananas as you want, and I guess that is good, but it's kind of would miss out like most of what's good about being human.

• 26:59Speaker 00

  You know, great literature and art and humor and politics and, you know, spirituality and all the rest of it. And it's not just that they happen not to think about that, but like the ape brains that probably are intrinsically incapable of even grasping the concept. And so I think it would be sort of presumptuous for us to think that now we have achieved the level of enlightenment where all possible values are like transparent to us. I think that might, we might be very naive. Like think of a big cathedral of possible modes of being, like ways of experiencing, seeing the world, relating, thinking. And I think like we are basically like explored a janitor's closet. And, and so that there's like a kind of, yeah, like a humility there and hope that something much better

• 27:44Speaker 00

  is, is possible in principle.

• 27:46Speaker 02

  Yeah, I love the Utopia essay. I think this like really does a good job laying out the benefits to people. Although, you know, most people haven't read it, of course, unfortunately.

• 27:57Speaker 03

  Yeah, this is also why I think science fiction authors have like the most underrated job in all of history, because their job is to sort of like lay out imaginatively what the possible futures look like. But I also wanted to ask you about your view of mind upload versus transhumanism, like a very corporeal transhumanism, and which one you think is more likely? Because these are two like huge topics in science fiction and it's not really clear like what the more realistic one is.

• 28:25Speaker 00

  Yeah, I think if I had to choose, I think like something where different people can choose their own path like would probably be more preferable than either one or the other if it were like imposed. Like I'm kind of in the peculiar position, I take this, I have this simulation argument. Whereas I take quite seriously the idea that we are already sort of implemented in the digital substrate.

• 28:54Speaker 00

  That might be a topic for a different conversation, but.

• 29:01Speaker 02

  Be part of the conversation.

• 29:03Speaker 00

  Well, I mean, it's a, so they are the idea that if we are biological, yes, but the biology itself might be simulated. And so it's like, it's just a question of what level of simulation. I think ultimately it looks to me that, I mean, what's important about human is not so much the carbon atoms, et cetera, but more the way we experience the world, that we relate to each other and the thoughts we think and the goals we have and so forth. And all of that could be implemented equally on sort of computational processes running on, you know, silicon hardware or whatever. And you could have a bodily experience as an upload that might be indistinguishable from the bodily experience you have now, right?

• 29:44Speaker 00

  You could have a simulated body. So in the limit, from the subjective point of view, they could be indistinguishable. But once you're upload, it's kind of easier to modify it and extend it in different ways. So it might be sort of cheaper to run many humans for a longer period of time in silicon substrate. You could easily like move like over galactic instances if your information that can be transmitted at the speed of light. And if you have to build a big clunky spaceship to sort of shove you over there, backup copies and so forth. So I think ultimately to me, it seems like an implementation detail, but it does look like the digital form is like just from looking at the basic physical constraints has a bunch of advantages in the limit.

• 30:28Speaker 02

  Do you have any disagreements with Robin Hanson's age of M on this topic?

• 30:35Speaker 00

  Yeah, I mean, I guess the most obvious is that it doesn't look like we will be getting M's before we have superintelligence, artificial superintelligence.

• 30:47Speaker 00

  So I think it's like a path with a lower probability that's kind of diminishing by the day as we see these rapid advances in artificial intelligence. Also, I think he is probably more gung ho than I am about the world that is Malthusian. So he envisages these M's being easily copyable, enormous copy, numbers of copies being made until the marginal economic productivity of a copy is equal to the cost. But that means that this enormous amount of mind is

• 31:29Speaker 00

  subject to extremely strong selection pressures for just being maximally economically productive. And it's not clear to me that the kinds of mental processes that are optimized for economic productivity in this M world would be ones that would be nearly houdaimonically optimal, like they might, you know, who knows? Maybe it's like, maybe it's slightly suffering to be like maximally economically productive. Or maybe they would have to get rid of some of the things we think are valuable, like humor and play and stuff like that. If that's not what is most productive economically, then that would be selected away. And so I think there was a huge cost to losing any ability to, if it does seem like sort of freewheeling

• 32:14Speaker 00

  evolution would lead in a dystopian direction, that would be a high value in having some option to sort of, you know, change track. But that would require coordination, right? Because if only one group decides to do something that is not economically optimal, then that would just be outcompeted by all of these other entities that go for economic productivity. So he like holds out the hope that maybe some humans like could be like kind of old world aristocrats that have inherited wealth and status and that would be surrounded by these trillions and trillions of sort of economically optimized beings. But that still seems less utopian than what one might hope for.

• 32:54Speaker 02

  Yeah. Do you think that current models might have any kind of phenomenological experience at all?

• 33:03Speaker 00

  Seems pretty plausible to me.

• 33:07Speaker 00

  I think there's a lot of uncertainty about it, but I say maybe it's more likely than not.

• 33:15Speaker 00

  I mean, we have these systems, you know, people think of it as software, but it's really not software like the way we're used to. It's not like lines of commands that get executed one at a time. You have instead a big neural network, right, with trillions of parameters. Now, what are we? We are big neural networks with trillions of parameters.

• 33:35Speaker 00

  They are not so much like designed as they are grown, like they start out sort of with random weights and then through experience get shaped in ways that allow them to develop more complex representations, including representations of themselves as existing through time. That's also what we have.

• 33:56Speaker 00

  And increasingly, we see some structures that seem to parallel the structures that human philosophers and cognitive scientists have postulated as the sort of neural correlates of consciousness, like global workspace theory is one of these, right? And now we look inside these large L L M's, they have something called a day space, which at least if you squint a little bit, looks like it's playing a similar computational role.

• 34:23Speaker 00

  And like another obvious thing you can do is you could ask them. Now, of course, if you just ask an L L M, like it might say it's conscious or that it's not conscious. Now, if it's trivially easy for the AI company to sort of train them to say whatever the AI company wants. And obviously, if you do that, you gain no information from them asking them, right? It's like putting your thumb on a scale when you're trying to weigh something like it. But if you carefully refrain from doing that, it does seem like they report being conscious. And you might still worry that that's like, oh, maybe they're role playing or kind of saying things that they think will be pleasing to us or maximally engaging. But then you can go in with a steering vector and suppress role playing and deception.

• 35:08Speaker 00

  And then it turns out they become more likely to report that they have phenomenal subjective experiences. Now, these are still early states. So this is like a very emerging field and there's a lot of uncertainty around this. But I think, yeah, there was this recent like Microsoft that came out with some declaration that AIs cannot be conscious and I don't know where they're getting that from. It seems to be pulled out of a hat. And maybe more because it's convenient if that were the case, rather than because we have good evidence that that is the case.

• 35:47Speaker 02

  Yeah. Well, I guess also though, like if you were to try to judge like a human by its outputs to figure out if it's conscious, like what I might, what I say doesn't really have anything to do with like my phenomenological experience, right? Like, so how would you actually measure that state of consciousness? I am fairly certain it can't just be like at the output level. There has to be something like more inherent to its, its structure or the like underlying architecture. And what might artificial consciousness eventually look like if we currently, you know, don't have it?

• 36:27Speaker 00

  Yeah, what it might eventually look like, that is a sort of mind boggling question. Because I think like we humans pride ourselves of being so very conscious, but I think we are kind of barely conscious. It takes like some driver, it's like spends two hours driving down the road and they were passing by like these thousands of buildings, right? And like mothers with strollers on the sidewalk and all like, did they, were they aware of that, like while they were driving?

• 36:55Speaker 00

  Did it register? Like it looks like we are kind of, if you look closely, kind of aware slightly in a murky way of a few features at any given point in time. Maybe some kind of confused thoughts flows through and, and we think of this as sort of the peak of consciousness. But I think if you imagine what might be possible in the future with these digital minds, they might, like they might have levels of awareness that just, you would think that we have been basically sort of sleepwalking through our lives if you compared it to that. So I think the potential is huge.

• 37:32Speaker 00

  I also think that this idea of consciousness is, the closer you look at it, I think the less clear cut and binary it seems. So we have this naive version that either the lights are on or they're not on.

• 37:49Speaker 00

  But I think there might be many different dimensions of consciousness that you can fade out in, in different ways.

• 38:02Speaker 00

  And so you see like with, like one approach to this is by like, like the neuroscience, you have different weird stage of blind sight and like the driver where it's like, there's a lot that in some sense we were conscious of. Another is like through more introspective methods or meditation where you realize that what initially seems like just this big visual field that you are perfectly aware of at any time in all full granularity, right? Like if you pay more close attention, you realize that most of that at any snapshot you're not aware of, but just a few features in focus. And even those, if you pay even closer attention, might be sort of flickering in and out of awareness on a sort of sub second time scale.

• 38:44Speaker 00

  And so it just looks like it might actually be like a concept that admits of more vagueness and multidimensionality. And it might be that current AI systems have some forms of consciousness and awareness in some of these dimensions. Maybe they are still lacking in some other dimensions and then, you know, how to describe that as conscious or not conscious might be a slightly confused question rather than just a question that we don't know the answer to.

• 39:11Speaker 02

  Right. So one of the big debates of the moment is open source. Then, you know, you've written about the vulnerable world hypothesis. Obviously, if open source models could create like a bio weapon in someone's basement with only ingredients from Home Depot or something, this would be bad and we would need some kind of centralized control to prevent outcomes like this. But also, you know, open source models have been a good counterbalance to the sort of concentration of power risk that we have seen from frontier labs potentially owning everything. So what's your current position on policy related to open source models?

• 39:54Speaker 00

  Yeah.

• 39:57Speaker 00

  It's, I mean, I think so far they have been good, but you do worry about what their effects will be if they do unlock truly dangerous capabilities, like in bio, for example. I think one thing we should do independently now is probably harden our other defenses against biological weapons. For example, by tightening up control on DNA synthesis machines. Because if you are gonna end up with open source models that give massive uplift to biodesign, then that will democratize access to weapons of mass destruction, unless some other necessary component to actually making these things is hard to get. And that the most obvious choke point

• 40:42Speaker 00

  would be DNA synthesis. I don't think every lab needs to have their own DNA synthesis machine. You could imagine that being provided as a service by, you know, a half dozen companies around the world. You send in your blueprint, you get the chemical back the same day or the next day. And then at least that would be a finite set of choke points, such that if and when it did start to look sufficiently risky, you could, you know, potentially clamp down there. I think other things as well, like PPE and UV sterilization, like there's a lot. I mean, in fact, these things probably would make sense to do even aside from any AI progress at all, just as a sort of cheap way to defend against natural pandemics, which, you know, sooner or later we would have another one. And then, of course, if you are worried about this misalignment business

• 41:27Speaker 00

  and think that it might be necessary to have a long pause at some stage, then, you know, open source, you know, potentially makes that harder. And then there is this sort of AI welfare dimension to this as well. Like, I think if you are building these digital minds that, you know, some of those will have moral status, they might need protections against abuse. And it's something that exists as open source software is just inherently vulnerable. On the other hand, I admire the sort of hacker ethos and the idea of sort of not being reliant on a few big corporations or governments that might nationalize the technology at any point in time.

• 42:05Speaker 03

  Yeah.

• 42:06Speaker 00

  But I don't have a firm view, but I'm a little bit concerned about really powerful open source models. At least in the absence of other sufficient safeguards that the world hasn't yet decided to invest in.

• 42:20Speaker 03

  Mm hmm. I think like something underrated about the pause conversation is that, like, in the sort of like longer horizon view of history, you probably will need like micro pauses or like pauses every once in a while. And you need to sort of set the precedent for what that might look like. Especially if we're looking like 100 years into the future, it makes sense that, you know, like maybe every so years we have a pause for X amount of months or something like that. And maybe something like, you know, even with like the COVID lockdowns, we now sort of have the precedent for what that might look like if pandemics were to become more common. So I think, like, what do you think should be done in terms of like making sure that future pauses go well?

• 43:05Speaker 00

  Yeah, I mean, so within COVID, there were many public entities that didn't exactly cover themselves in glory in how they handled that. And obviously, the best time to have had the pause, as it were, was right at the very initial onset, at the very first sign that something had escaped. Like, if you had clamped down massively at that point, just crazy, then, you know, the whole thing wouldn't have happened. And that would have been a short, one-time cost in the local community or city. China tried to do that, but it wasn't like, it was delayed, and it wasn't enough. Now, once it spread everywhere, then it wasn't really clear how much you gained from all these subsequent sacrifices. I think there are various things

• 43:50Speaker 00

  that like seemed, like the fact that it was transmissible by air like seemed obvious early on, but it wasn't acknowledged, like, I think. But once you have that, there are like relatively cheap things, like opening windows in buses and like public places that I think could have been good. And then now with techno, like air purifiers, like HEPA filters, they are also good for air pollution, like for particulate matter that that exerts a big health burden, actually, like just from ambient in the air. Like, there's a lot of healthy life years that are lost just from that. So you could have indoor air purifiers. There is like now development of the UV sterilization technologies. You could have like a lamp near the ceiling

• 44:34Speaker 00

  that just kind of sterilizes the air as it kind of naturally flows through there. So some of these passive countermeasures, I think, would be good to prioritize. They're kind of less sexy than the idea of like, are we gonna invent this new high-tech synthetic vaccine thing, which we should also invest in. But like making a slightly better face mask or a slightly better air purifier, it's not something that maybe appeals to like this like bright entrepreneurial young person who wants to make a mark in the world, but it's like under invested in. And also, they have less risk of info hazards. Like, if the idea is that you're gonna create a whole new way of designing vaccines, you develop all kinds of advances in synthetic biology,

• 45:18Speaker 00

  like who knows what else those advances might enable, right? Like, if you have some super great way of modeling, you know, pathogens and what kind of antibodies would destroy them, like that same modeling capability might also be used maybe to design more dangerous pathogens. Whereas like a better air purifier or a better face mask, it's kind of hard to see how that could be turned around and pose some catastrophic risk.

• 45:43Speaker 02

  Do you think that we have made any progress toward automating philosophy? You know, you've been a philosopher for a very long time. We made progress in automating, or at least progress towards automating mathematics, towards automating software engineering. It seems like there's been somewhat less in terms of philosophy. So what do you think about this? And what would it take to get there?

• 46:03Speaker 00

  Well, we certainly made a lot of progress in the sense of AIs are a lot closer to being capable of this now than, you know, that were even just a few years ago, right? I think they are now at the stage where they're certainly useful as a sort of discussion partner or research assistant. Like, if you have some new philosophical idea, like you just wanna have somebody to poke at it, like raise some objections, question something, point you to something in the literature, like that kind of stuff. The AIs are already good enough to do that they can stimulate your own thinking further, like be like a sounding board. And it might be that we are perhaps very close to having these systems

• 46:47Speaker 00

  also being able to do already know philosophy on their own. I haven't yet seen any kind of compelling demonstration of that. I think one thing that they seem to be lacking is the ability, as it were, to form new concepts. Like, it might be connected to the absence of online learning, the idea that by sort of having a bunch of experience reflecting on it, and then you might find new concepts to help organize this confusing mess that then allows you to ask new questions. And maybe you then need to sort of consolidate these new concepts into your, the rest of your neural network by sleeping on it or sort of living with it for a few days, and then it becomes second nature.

• 47:32Speaker 00

  And then you can use that as a primitive in thinking more complex things. Because now you have this kind of concept baked in that allows you to sort of build up philosophical understanding. I think that is something that it's not clear what that these AI systems yet have, as opposed to just being very good at manipulating the concepts that they have already attained through their training process. But sort of by using thinking, develop new concepts, and then using those concepts in a fully integrated way as a sort of stepping stones then for the next level of thinking is kind of at the frontier, still a little bit beyond the frontier, I think. But I have, for a long time, like been planning on my own obsolescence. Like, the superintelligence

• 48:16Speaker 00

  had a chapter on philosophy with a deadline. The idea being that some of these big eternal philosophical questions, you know, maybe are less urgent and will at some point develop superintelligence that will be much better at philosophizing than we are. And that perhaps what we should focus on in the meantime are those philosophical questions that we actually need answers to now in order to navigate this transition safely, to actually build these more competent successors. And yeah, so I think that the time is, that the timer is running down for humans to make our intellectual contributions.

• 48:52Speaker 03

  Yeah, so kind of a crazy question towards the end. So you talked about the simulation hypothesis earlier, and there's this idea just that like hyper-intelligent system will eventually run ancestor simulations. So if we're in an ancestor simulation, how would we best influence the world that's simulating us?

• 49:15Speaker 00

  Well, there is so much we don't know about the hypothetical simulators that are, would be running our simulation, the purposes. I have this recent paper called AI Creation and the Cosmic Host. And the cosmic host is this set of super beings that might exist. So these could include, if we're in a simulation, the simulators would be kind of super beings. But it could also be, if we're not in a simulation, maybe just some faraway galaxy has developed super intelligence. You know, if we are in a many worlds, theory is correct, like there would be other branches of the universal wave function where there would be super intelligences. All right.

• 49:58Speaker 00

  And I like traditional theological conceptions, obviously gods usually is conceived of as a super being. So either way, that could be this set of beings, I call it the cosmic host. And they might have cosmic norms, that different coordination principles that they have arrived at for how things should be run at that level. And this paper argues that one important decideratum in our own birthing of super intelligence is that we try to make a super intelligence that would be a good cosmic citizen, that will basically get along well with the cosmic host, be able to sort of conform to the cosmic norms that might exist.

• 50:43Speaker 00

  And it has some speculations about what that might involve. I think probably creating super intelligence would be something favored by the cosmic host for various reasons, but also then with a big proviso, that it should be a form of super intelligence that is easy to get along with, that will sort of respect the norms that exist, that make a positive contribution. But it is still very, yeah, there's still a lot of uncertainty about sort of how things shake out at that highest level.

• 51:10Speaker 02

  Yeah. So last question before we go. Tell us about the macro strategy research initiative. What is it and what kinds of research do you plan to do next?

• 51:19Speaker 00

  Yeah, so that's basically just a support org for my personal work. So it's not intended to be building, like hire a bunch of different people to do things. So it's basically, you know, I'm an assistant and some other contractors. So right now, as for me, I mean, I'm sort of thinking about all of this AI stuff that has so many dimensions to this. There is the alignment, there is the governance, there is the ethics, there is the macro strategy of this, and looking for ways in which, on the margin, might try to nudge things in a positive direction. I tend to alternate between sort of output phases where I'm working on, like writing

• 51:59Speaker 00

  some specific paper or producing an output, and other phases where it's more curiosity-driven contemplation and just kind of thinking about different things without the aim of producing a deliverable. And right now, I'm in that second kind of phase, in exploratory phase, trying to just make sense of it all. But the problem is so much is happening all the time, like, because philosophy happens on a slow timescale, right? AI is happening on a very fast timescale now, so it's become a full-time job just to, you know, monitor the situation.

• 52:34Speaker 03

  We know, we know.

• 52:37Speaker 00

  Yeah.

• 52:38Speaker 03

  I have the sign that says MTS behind me. Well, this has been incredible. Thank you so much for joining, Nick.

• 52:45Speaker 00

  That was fun. Good to meet you, guys.

• 52:48Speaker 02

  Good to meet you.

• 52:49Speaker 03

  Great to meet you.

• 52:50Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode. This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z.

• 53:31Erik Torenberg?

  Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates. Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.