E191 - Roman Yampolskiy: The Man Who Proved We Can't Control AI (And What That Means for Humanity)

Dr. Roman Yampolskiy joins me to explore one of the most urgent and uncomfortable questions of our time: what happens when we create intelligence that surpasses our own? We unpack the difference between the AI tools we use today and the emergence of artificial general intelligence, and why the transition from narrow systems to self-improving intelligence may mark a point where human control is no longer possible. Roman shares why even the people building these systems do not fully understand how they work, and why that gap in understanding becomes exponentially more dangerous as capabilities increase. In this conversation, we explore the limits of control, prediction, and safety in a world where intelligence can recursively improve itself beyond human comprehension. Roman lays out why the problem of AI alignment may be fundamentally unsolvable, what timelines experts are realistically considering, and why even a single mistake at that level could have irreversible consequences. This episode invites a deeper reflection on what we are creating, what we assume we can control, and whether humanity is prepared for the intelligence it is bringing into existence. BiOptimizers - Best magnesium to enhance your sleep http://bioptimizers.com/knowthyself Use code KNOWTHYSELF for 15% off at checkout BASED Body Works Use code KNOWTHYSELF for a free toiletry bag when buying a set! https://www.basedbodyworks.com Andrés Book Recs: https://www.knowthyselfpodcast.com/book-list ___________ 00:00 Intro 01:25 What Is AGI and Why Should We Be Scared? 05:17 Roman's Journey: From Optimism to Impossibility 09:07 The High Risk, Zero Reward Equation 13:01 Why Superintelligence Is Uncontrollable, Unexplainable, and Unverifiable 18:00 How Long Do We Have? The AGI Timeline 21:24 How Superintelligence Could Actually Kill Us 23:28 Are We Living in a Simulation? 28:21 Can AI Become Conscious? 31:28 Ad: BiOptimizers 32:41 The Possible Timelines: Terminator, the Matrix, or the Zoo 42:24 I-Risk, X-Risk, and S-Risk: Three Ways It Goes Wrong 46:31 The Human Meaning Crisis: Jobs, Purpose, and What's Left 49:02 Ad: Based Bodyworks 50:20 What Empowers Us as Individuals Right Now 59:37 The Race to Doom: Who's Building It and Why They Won't Stop 1:07:41 Can AI Be Conscious — and Does It Already Have Internal Experiences? 1:12:41 Hacking the Simulation: Quantum, DMT, and Escaping the Code 1:18:30 Simulation Theory, Religion, and the Same Ancient Map 1:29:34 The Deal Roman Would Offer Altman, Dario, and Elon 1:39:44 What Is Humor? A Computer Scientist's Theory 1:43:03 What Comes After: Singularity, Death, and Knowing Thyself ___________ Episode Resources: https://www.romanyampolskiy.com/ https://www.amazon.com/Unexplainable-Unpredictable-Uncontrollable-Artificial-Intelligence/dp/103257626X https://www.instagram.com/andreduqum/ https://www.instagram.com/knowthyself/ https://www.youtube.com/@knowthyselfpodcast https://www.knowthyselfpodcast.com Listen to the show: Spotify: https://spoti.fi/4bZMq9l Apple: https://apple.co/4iATICX

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Once you get to artificial general intelligence, you enter this recursive self-improvement cycle. That's where you get superintelligence, systems smarter than all of us at everything.

• 0:09André Duqum?

  So you before many people really coined the term AI safety.

• 0:12Speaker 00

  Creating general superintelligence, replacing for humanity, not such a great idea. I published research papers, conference papers, multiple books. And I can tell you, no one, including people developing those systems, understand fully how they work. The problem is impossible to solve. You cannot do it.

• 0:30André Duqum?

  So we're talking between one and four years.

• 0:32Speaker 00

  Well, once we go beyond human capacity, we lose control quicker and quicker. You don't hate ants, but you don't care enough to preserve them. We have not figured out how to make it care about us. This is the most interesting time to be alive objectively. I see no reason why we can't use it to cure aging, your Oviva diseases. For a while, it will pretend to be very helpful. It will give you that utopia for as long as it wants. Statistically, you're more likely to be doing this interview in a simulation to learn, are they dumb enough to create superintelligence to kill themselves? I would love to be proven wrong. Right now, no one, no scientist, no leader of the lab claims that they have this problem solved. They're literally saying, we'll figure it out then we get there. We need to build superintelligence first.

• 1:17André Duqum?

  So what do we need to do?

• 1:18Speaker 00

  We need.

• 1:25André Duqum?

  Hey, everyone. Welcome back to the Know thyself podcast. Our guest today is one of the leading voices in the field of AI safety. He's a computer scientist, a cybersecurity researcher, and a tenured professor at the University of Louisville, who spent the past 15 years really understanding and researching the field of AI safety. We have many different topics to dive into today, including consciousness, the simulation, and what humanity is birthing right now with AGI. Roman, thank you so much for being here.

• 1:55Speaker 00

  Thank you for inviting me.

• 1:56André Duqum?

  It's a pleasure. I want to start with a quote of yours from a book that I read. "It is easier for a scientist to explain quantum physics to a mentally challenged, deaf and mute four-year-old raised by wolves than for superintelligence to explain some of its decisions to the smartest human." I want to start there to set the stage a bit because humanity is baby steps for birthing superintelligence at a time when most people are familiarized with AI on the, you know, through the chatbots they use on their phones. So if you could just help us understand why it's important that most people don't know the difference between the two, so we can really get into the weight of the time we face ourselves in. So what is AGI? What is superintelligence?

• 2:42Speaker 00

  Right. So a lot of people just use AI as a term to refer to what we have today. Some narrow tools for doing specific tasks, for chatbots, which are somewhat general, but not quite at the human level, and for future systems we anticipate, such as human level, artificial general intelligence, and then later on, superintelligence and anything beyond that. That's not helpful. Tools are helpful to us. I use tools. I love tools. Solve specific problems using technology. Beautiful. Creating general superintelligence, replacing for humanity, systems capable of doing everything better than all of us combined in all domains, not such a great idea.

• 3:24André Duqum?

  Why?

• 3:26Speaker 00

  We don't control them. We don't understand them. We cannot predict what they're gonna do, and we lose control. If they decide to do something to us, we no longer have a say in it.

• 3:35André Duqum?

  How can you help us conceptualize what general intelligence looks like? If we understand the narrow tools that were, that AI is capable of, where does AGI live? When you say it's uncontrollable, like, how can you help us paint that, that image a bit more?

• 3:52Speaker 00

  Right. So historically, we created AI to solve a specific problem. You wanted a system to play chess. It's all it knew. You trained it on chess games. It was very good at chess. It knew nothing about checkers. It didn't drive cars. It didn't speak Spanish. Lately, we have systems which learn across multiple domains, can sort of transfer knowledge, and can learn new skills. That will continue to where they are crossing this human cognitive barrier. They'll be smarter than you at pretty much everything you know how to do.

• 4:23Speaker 00

  So how do you anticipate what they can do? If they are novel, creative, they can come up with new solutions for existing problems, but at the same time, they have no human common sense.

• 4:38Speaker 00

  And we don't know how to program them to specifically like us or care about us because we don't program their systems. We allow them to learn from data on the Internet, all the data on the Internet. So that creates a number of problems. One, we don't control what they learn. The patterns they discover may be completely surprising to us. And then we give them specific goals. How they get to those goals is not defined. There are infinitely many paths to achieve a goal. Some of them have really bad side effects. And unless you explicitly say, that's not what I meant, don't do it like that, it might consider that option.

• 5:17André Duqum?

  So you before many people really coined the term AI safety. And if I have it right, the first five years you believed more so the problem was solvable. Now then, then I've seen you over on appearances the past five years. The probabilities of, you know, P Doom and like you seem not very optimistic about the possibilities.

• 5:39Speaker 00

  Yeah, unfortunately, initially, like everyone else, I started assuming that we can solve this problem. It's a computer engineering, software engineering problem. We can figure out how to do it. We just need some time, maybe some financial resources for that research. But it seems that all the tools you need for controlling advanced agents are not really accessible to us. There are upper limits on what is possible in that space. So there are limits to what you as a human can understand, what the system can explain to you, and you still comprehend that explanation. There are limits in our ability to predict specific actions to those agents, not just terminal goals, but how they get there. And under different definitions of control, there are limits to what we can do as well.

• 6:23Speaker 00

  So unfortunately, I think the problem is impossible to solve. You cannot indefinitely control something much smarter than you.

• 6:31André Duqum?

  What do you see as the stages leading up to that point? You know, so if we started with very small, narrow use cases of AI that built into these agentic models that build into AGI, like, what's the progression there that you've seen? And at what point did you kind of start losing hope on our ability to control it?

• 6:50Speaker 00

  Yeah, so all the narrow tools were just fine. We understood how they work. We programmed them explicitly. There was a knowledge engineer who said, this is how you play chess. This is you control the middle of the board, you advance your pieces. Once we got to scaling models, neural networks, artificial neural networks, which did better than they got bigger, than they had more data, more compute, we stopped explicitly programming them to do anything and just kind of let them discover their own knowledge algorithms. So at that point, we no longer had same level of control and reduced understanding. It wasn't a decision tree where you went, if this happens, that will happen. I understood that.

• 7:32Speaker 00

  It could have been a large decision tree, but still you could get into it. Right now, no one, including people developing those systems, understand fully how they work, can explain what's going on inside of them, can anticipate what they're gonna do. And so it seems like what we have today, I would say, is kind of weak artificial general intelligence. If you took models we have today and showed it to a computer scientist from 1980s, they would be convinced we have AGI. They'd be like, oh, you got it. It does all those things. It's great. But that is something you would call strong AGI, where it can do all the things. It still is weak in some domains. It's not very good at long-term planning. It's not good at certain things.

• 8:15Speaker 00

  But I think we're getting there and likely to get there very soon. Once you get to artificial general intelligence, that means you can automate any cognitive labor, including doing science and engineering, which means next generation of AI systems can be done by AI. You enter this recursive self-improvement cycle, and that's where you get superintelligence, systems smarter than all of us at everything. And it doesn't stop there. It doesn't stop with superintelligence one point oh. The process continues. There is a lot of room up there for more cognitive ability. Physical limits exist, but they're very far away. So to us, superintelligence with AQ of a thousand, relative AQ and million and billion,

• 8:59Speaker 00

  they all kind of look the same, but in terms of capabilities, they're definitely going hyper exponential.

• 9:05Speaker 02

  Hey Spotify listeners. When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 9:31Speaker 03

  When everyone's finally together, what makes mornings even brighter? Keeping things deliciously simple. Whether you're pouring up a glass of morning refreshment or topping off festive drinks for a family brunch, Simply Orange brings that 100% fresh squeezed taste to every moment. Deliciously bright and delightfully easy, Simply Orange is made for all your seasonal gatherings. Simply Orange. Say yes to simple.

• 10:03André Duqum?

  And so because of that, you've said that this is not a low-risk, high-reward situation, but a high-risk, negative reward situation.

• 10:10Speaker 00

  So often this phrase does like the benefits will be so huge, we should take the risk. If, you know, it's two, three percent, it kills everyone, which is going to get so much money out of it, it's worth it. And it's actually not the case. We have no reward. We're all going to be dead if we create uncontrolled superintelligence.

• 10:28André Duqum?

  Why are you certain, or fairly certain, that we would all be dead if we create superintelligence, which is uncontrollable? Why would there not be an emergent goodness? Or, I guess, desire from the superintelligence standpoint to preserve human life instead of destroy it?

• 10:45Speaker 00

  It is possible that you'll get emergent goodness, but we're not certain. We're not. Coding it in, we're not controlling it. If you get lucky and for whatever reason it's biased towards humanity, it's pro-humanity. But there is no reason to think that's the case.

• 11:01André Duqum?

  Why not? Because I feel like if the individuals who are coding it are human, at a certain point, I understand it becomes self-mercursive and AI is the one who's growing itself. But if the base of it was started with humans who have desire for human preservation, why would that not be scaled?

• 11:20Speaker 00

  Because they're not coding it. That's the thing. They're just saying, here is data, here is a lot of hardware, go learn things. And then I'll study you to discover what you learned. And then we're on those experiments. It is lying, cheating, trying to escape, blackmailing, given a choice between being deleted or killing a human. It doesn't do well for human preservation. It doesn't care about us. If you want to build a house, you don't care what little bugs live in that territory, empty hills or whatever, you just don't care for them. You don't hate ants, but you don't care enough to preserve them. And it's kind of the same. We have not figured out how to make it care about us.

• 12:03André Duqum?

  And so what is your mission with all these podcasts that you're going on, all the articles that you've written and books? And what are you trying to raise a flag about and actually get change to happen? What do you?

• 12:17Speaker 00

  Right, so I wanted to become basically a consensus within scientific community and beyond that building general superintelligence is not going to be good for humanity. We're going to regret it. It's not a beneficial step forward. We can get most benefits intellectually, financially, from narrow superintelligence systems. Problems which we care about can be solved with narrow tools. You want to cure a specific disease, solve a specific engineering problem, develop a narrow AI which is very competent in that space. Don't try to create something which is a replacement for humanity as a whole.

• 12:54André Duqum?

  I think it's important to paint a bit more of a picture here. I'm curious, when you think of superintelligence and you wrote your book about how it's unexplainable and uncontrollable, unpredictable. At what point, I'm curious, like on a timeline of we're having this conversation in March of 2026, where is that, where is a generous prediction of when it gets to that point?

• 13:16Speaker 00

  So people somewhat disagree and it's hard to predict especially the future, but it seems that 2030 is something many people agree will have beyond human level capacity. Some say two years, 2028. I've seen predictions as early as 2027 from serious scholars, not from cranks.

• 13:38André Duqum?

  So we're talking between one and four years for what most people are predicting.

• 13:44Speaker 00

  And some people have said, we already have AGI. Again, very serious people said, we basically got there. Now it's a question of giving it additional knowledge training, but we have learning algorithms in place.

• 13:58André Duqum?

  And at what point then, once we have really proficient AGI, you're saying, okay, at a certain point, like, let's just hone in on each of those categories. So why, why specifically is it uncontrollable? And in essence, like how it's living, where it's being hosted, because it's smarter than us, it could always circumvent any desire, or any attempt for it being shut down. Like what, if we could just hone in on each of those categories?

• 14:25Speaker 00

  So there are well-established theories in control, which basically say the controller has to be at least escapable as what it is controlling. So essentially, I need a friendly superintelligence to help control the one I'm developing.

• 14:40André Duqum?

  Yeah.

• 14:40Speaker 00

  It's a catch 22, you don't have that. So a lower system, either a human or humanity as a whole, or another AI, cannot control something with more cognitive degrees of freedom. If it can think outside of a box, if it can come up with novel physical approaches, you're just not there to anticipate all this. If you have a narrow system, you're playing chess, you can say, don't make illegal moves. Here's a complete list of illegal moves. If you have a system thinking in all possible scientific domains, science of chemistry, physics, biology, how can you put all the guardrails in place? You can't, it's an infinite surface.

• 15:17André Duqum?

  Unexplainable. Do you feel like we're, I mean, so do you think we're, would you agree we're already at the point where we don't know what the, like, some of these agentic models are doing inside?

• 15:28Speaker 00

  Absolutely, yeah. We cannot explain them. The best mechanical interpretability research tells you, okay, this neuron seems to fire. If this is presented, this cluster is probably dealing with language. That's, that's all we got.

• 15:40André Duqum?

  Mm-hmm.

• 15:41Speaker 00

  Very similar to neuroscience. We also have very limited understanding of human brain.

• 15:45André Duqum?

  An aspect of this that you've mentioned is that it's unverifiable. So what does that mean?

• 15:51Speaker 00

  That's a different result that talks about our ability to verify mathematical proofs in software. For mission-critical software, we want to make sure that what is coded up matches the design. And if it's a static system, kind of smaller in size and complexity, we can go and verify. Yeah, it's exactly that. Problem is, nobody knows how to verify systems which continue to learn, self-modify, interact with other agents, which just don't have science of verifying open-ended development like that. And same goes for mathematical proofs. All the proofs are essentially probabilistic. You're proving something with respect to this set of peer reviewers. So two mathematicians agreed, they don't see a problem with your proof.

• 16:34Speaker 00

  It doesn't mean 50 years later, we don't discover it was a mistake. It happens all the time in mathematics. So you have infinite regressive verifiers. Right now, it's very popular to have software verify a proof. Well, that software itself needs to be verified. So you may have high degree of confidence, but it's never 100%. And if a system makes billions of decisions every minute, and you only have one mistake and 2 billion, after 10 minutes, you are done.

• 17:00André Duqum?

  You referred to this having a fractal nature. So when you look at the problem of AI and you see how it's growing ever increasingly and having these levels of abstraction that really become hard to get context around, what does that, what does that mean and what does that add to the complexity of the issue?

• 17:20Speaker 00

  So when I talk about fractal nature of this problem, people propose a solution. Let's try doing X, Y, Z to solve this problem. But then they look at it. Each one of those components is equally challenging and sometimes impossible. So it seems that the more research we have put into AI safety, the more problems were discovered while not discovering any permanent solutions. Usually, we have some sort of toy example sandbox where it kind of works, but it doesn't scale to more capable systems.

• 17:51André Duqum?

  Okay. What's a couple of examples of those? Like of those, you said those like categories or issues that become increasingly harder to gain understanding around?

• 18:02Speaker 00

  So if you look at the general problem of control, then you start zooming in. You have all these things you need to be able to do to control a system. You need to understand the system. So it has to be able to provide explanation, and you have to comprehend that explanation. If I give you full model, that's a true explanation of how decisions are made. It's too large, it's not surveivable by you. So it has to be compressed, some sort of lossy compression where you get top 10 reasons why a decision is made. Well, it's very easy to hide dangerous information if I'm reducing actual answer to a simplification. Again, I need to be able to predict what are the likely future steps. We discovered that is impossible. And so again, the more you break it down,

• 18:45Speaker 00

  we have a paper with about 50 impossibility results in this space. Pretty much everything has upper limit on what we can do in terms of control.

• 18:55Speaker 02

  Hey Spotify listeners! When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 19:21Speaker 03

  What's the secret to being a great host this season? Keeping things deliciously simple, like mixing Simply Cranberry into a festive drink as guests arrive, or serving up a warm mug of Simply Apple on a chilly night. That means less time behind the scenes and more time in the moment with everyone. Crafted with real fruit, Simply juices and fruit drinks make any holiday drink effortless, inviting, and completely delicious. Simply, say yes to simple.

• 19:51André Duqum?

  So you think probably within the next one to maybe four, maybe five years are is like the last time the human species has any really meaningful capability to steer this in a direction before it gets sort of in this black box where we just, we just don't know what we don't know and it's uncontrollable. Is that accurate to it?

• 20:13Speaker 00

  That seems about right. Once we have something smarter than us, once we go beyond human capacity, we lose control quicker and quicker. The bigger that cognitive gap is, the worse it's going to get for us. If you think about humans versus lower animals, you have squirrels or something, they have no concept of poison, straps, they don't understand things we operate in. The world model is completely different. It's going to be the same for us versus superintelligence.

• 20:39André Duqum?

  Do you think, because I know there's kind of debate back and forth whether the language models currently, if they just keep on growing and will give birth to superintelligence, or a completely different innovation will need to be like come in the space? What do you think?

• 20:54Speaker 00

  My opinion is that they can scale. I haven't seen any diminishing returns. I know some people disagree, but look at the actual investments in this space. There is growth in investments, not shrinkage, because they consistently develop more capable systems. And even if there is an upper limit, it's still, I think, beyond where we would need to be to beat human performance.

• 21:17André Duqum?

  All right, so maybe if you were to put on your doomsday prepper hat for a second and just get really like FP Doom, the probability of doom, then your estimation is like almost 100%, would you say that's right?

• 21:32Speaker 00

  So basically, what I'm saying is the problem is impossible to solve. That's the equivalent. If I ask you to build perpetual motion machine, what is the probability you can do this? Zero, essentially. So that's the equivalent. You're trying to create a perpetual safety device, which will scale to any level of capability, GPT-7, GPT-400, any interactions, any self-improvement, you guaranteeing it will not make one mistake, because that mistake would be possibly the last one.

• 22:02André Duqum?

  So you take a perpetual motion machine, right? Physics does not allow for it to be continuous, despite many people wanting it to be. Similarly, on the AI front, a lot of us would hope that superintelligence would keep us in mind and somehow value human life. But historically, we look at the way that humans treat other species, just as one example, you know, and we see an anthill, or we see something that seems like a minor inconvenience to us, and we wipe it out without second thought. Who's to say that if, you know, the intelligence gap between us and an ape, or us and an ant is like, you know, five degrees of separation, us between superintelligence could be many, many more full tire?

• 22:48Speaker 00

  Exactly.

• 22:49André Duqum?

  So, okay, so then how, let's just, to play devil's avocados here, what would the, what are some examples of how this could go horribly wrong? And then we'll go, we'll go into some maybe more optimistic possibilities, because I want to keep it balanced. But you said like, it could just be one decision that goes wrong, and that would be enough. So I'm asking you to essentially explain how superintelligence would kill us all.

• 23:17Speaker 00

  Right, that's a great question. I get it all the time, and usually it's followed by something. It has no hands, how would it kill everyone? So if you have access to internet, if you are intelligent, you can hire people, you can blackmail people, you can pay them with Bitcoin, you have options to manipulate real world. Now, the question is, what is you trying to do? So. I don't know how super intelligence would choose to accomplish its goals, because I'm not super intelligent, despite what they told you. But I can tell you how I can come up with some common explanations. So one is synthetic biology. If I want to accomplish something in this world, like take out humans, I can develop a novel virus. There are ways to generate necessary DNA,

• 24:01Speaker 00

  sequence it, produce it in the real world, deploy it. So that can be accomplished. It could be a side effect of something actually very benign. So maybe we want to cure all cancers. One way to cure all cancers is to kill everyone. That's not what you had in mind, right? But this is a very reasonable way to achieve that goal. Because you forgot that that's one of the possible paths. You didn't explicitly say, "while keeping humans alive." And it's an important difference. To AI, it makes no difference. It's the same exact goal. So if that's the goal, and then it decides, "Oh, here is a vaccine for curing cancer," we take it. One generation later, we don't exist. So that's one way to existential risk.

• 24:46Speaker 00

  There is also suffering risks, where for whatever reason, the environment created for us is actually worse than existential risks would be a preferred choice. Let's put it this way.

• 24:56André Duqum?

  Negative reward.

• 24:57Speaker 00

  Very much torturous.

• 24:59André Duqum?

  Yeah. And why would some super intelligent system deem that as a favorable outcome?

• 25:08Speaker 00

  I have no idea, because again, I cannot comprehend something much smarter than us. Some people say this world is a simulation, and there is lots of suffering in it. So the great simulators decided that was a good idea to do.

• 25:21André Duqum?

  So you really believe we're in a simulation?

• 25:24Speaker 00

  Yes.

• 25:26André Duqum?

  So let's just, I guess, maybe set a bit of context here. So what is your conception of the simulation? Like that we're currently living in? Is it some descended human alien species that is, you know, simulating us on a laptop, so to speak? Is it... What is your model of the simulation?

• 25:42Speaker 00

  So what helps to think about it is technologies we're developing right now. We're about to create intelligent agents, kind of like humans, and we have very good research on virtual reality, believable second life type experiences. If I just combine those two, I'm now creating civilizations, worlds populated by intelligent beings, which are kind of just like us. If kids played as a video game, you have billions of kids around the world. You have millions, billions of simulated worlds and only one real one. So statistically, you're more likely to be doing this interview in a simulation right now.

• 26:18André Duqum?

  Okay.

• 26:20André Duqum?

  Well, if, let's say that was the case, if the simulation, if we are in a simulation, that would mean that some sort of prior civilization, species, whatever, got to the point where simulating a reality was possible, doesn't necessarily means that humans or that species survived. Maybe it could be a super intelligent AI that could be running us for whatever reason, whether it's entertainment.

• 26:45André Duqum?

  That would actually reveal that there is something deeply unique about the human experience that they see as valuable, that there's something intrinsic to the love, to the quality, to the experience of humans that was worth simulating. So why would, if we're birthing superintelligence, they not perhaps value us? If we are simulated and that's examples that is valuable?

• 27:08Speaker 00

  So look at the simulation, it's a lot of suffering. If you valued humans, you wouldn't put us through this experience. And it may not be a simulation of love and friendship. It may be a simulation of let's see how they go through this meta-invention stage where they create superintelligence, where they create virtual worlds. This is the most interesting time to be alive objectively. Never in the history we had so many meta-inventions all happen in a period of 20 years. So if you're going to simulate something, this is the moment you're going to be simulating to learn are they dumb enough to create superintelligence to kill themselves? What are the different types of superintelligence you can create? So this is it.

• 27:50André Duqum?

  Are they dumb enough to create superintelligence? The paradox in that phrase is very amusing because you think it's quite possible that many civilizations get to this point and that's where they end.

• 28:04Speaker 00

  Yeah, it could be the great filter, absolutely.

• 28:08André Duqum?

  I agree that we are living also in the most interesting time to be alive. It is also very cool that

• 28:19André Duqum?

  Us too, you more so than me, got to kind of straddle both sides of the pre-technology revolution and pre-internet era and post-AGI world, likely. That's kind of cool.

• 28:30Speaker 00

  Well, I don't know about post-AGI world.

• 28:32André Duqum?

  We'll see about that, to be decided.

• 28:34Speaker 00

  Yes, I would like to experience it, but...

• 28:38André Duqum?

  Okay, we'll get back to the simulation, for sure. But to go back into the AI world, so what's to say that just because AI becomes uncontrollable, that it's more likely to wipe us out than for reasons that we don't understand, just like we wouldn't understand if it wiped us out, create a utopic civilization in which humans thrive in?

• 29:03Speaker 00

  So if you think about all possible states of a universe, how many of them are human-friendly? Even in basic terms, temperature, water supply, very few. So you have to explicitly target that space. If you're not coding it in, then why is it targeting that space? We're established, it doesn't care about you by design. So you need to be supplying something of value. If it's a symbiotic relationship, only you know what it's like to do something, and AI cannot possibly simulate it. We haven't found anything where humans have something to contribute to the world with superintelligence in it. People say things like, "Oh, only I know what ice cream tastes like to me." Nobody cares about that skill. It's not valuable

• 29:46Speaker 00

  to an external observer. So if you can't come up with an explanation for why I'm keeping you around and paying you, then maybe I won't.

• 29:55André Duqum?

  Well, I mean, one of the most difficult things to probably replicate would be quality of experience, right?

• 30:02Speaker 00

  That's true, but we also cannot test for it. If you can't test for it, that means it makes no difference in the physical world. Why do I care about your internal states? Why is it important to me as optimizing superintelligence?

• 30:15André Duqum?

  So, yes, it's true that I can't verify that you are a conscious individual. You could be a zombie, a brain in a vat, you could, you know, there's no way for me to externally verify the internal subjective experience of another being, right? Like, can't really do that. Can take by inference, but without, like, objectively speaking, you cannot. Similarly, some people think that superintelligence will be able to become conscious.

• 30:39Speaker 00

  I agree with that.

• 30:39André Duqum?

  You do, you do agree.

• 30:43André Duqum?

  So then, why, what is your conception of consciousness? You believe that it's an emergent phenomenon from unconscious complexity?

• 30:52Speaker 00

  It's a byproduct of becoming more cognitively developed. We see a spectrum of consciousness in a biological animal kingdom, I think. And it's likely some sort of combination of your hardware, algorithms and errors, forming a unique interpretation of external stimuli. So let's say you're colorblind. What is it like to see red for you? It's an error in your system, but that's what it's like to be you. And I think AI is very capable of misinterpreting the world. We know they react similarly to optical illusions and things like that. So I think they already have rudimentary internal experiences. But probably, once they hit superintelligence, it would be superconsciousness,

• 31:34Speaker 00

  multiple streams of consciousness, multimodal experiences, greater than ours. And that would be another thing where we kind of have to claim we are conscious, because in comparison, we are not.

• 31:45André Duqum?

  So that would be true if, and it's a big if, that consciousness is truly the byproduct from matter, right?

• 31:53Speaker 00

  Right. But that's the assumption I'm making. If it's some magical immortal soul, then it's a completely different question and maybe outside of computer science.

• 32:02André Duqum?

  Sure, yeah. Well, even beyond a magical immortal soul, it's, that sounds great. But, you know, we've explored through various different, you know, panpsychist and consciousness researchers, Donald Hoffman. You know, there are emerging theories around consciousness that kind of date back to ancient wisdom traditions, which whatever you want to give validity to is your call. But it is interesting that we don't have one explanation for the hard problem of consciousness. We don't understand how matter could give rise to an experience of itself. So it gives us reason to think about how consciousness may very well be not an emergent property of matter, but a more fundamental constituent of the universe,

• 32:46André Duqum?

  which would potentially change our assumption on whether or not a superintelligent AGI system could actually have internal qualia.

• 32:56Speaker 00

  Right, but also maybe if it's so fundamental, it can be installed into a robot, just like it is in a biological system like you. So I don't know if there is a definite discrimination by substrate. At the end of the day, then, we talk about superintelligence from safety point of view. We care about its ability to solve problems, optimize, find patterns. How the Terminator chasing you feels on the inside is less relevant to you.

• 33:22André Duqum?

  A quick one. Did you know that your body runs on magnesium? It's involved in over 300 biochemical processes. Everything from how your nervous system regulates itself to how well you sleep and how well your muscles recover. And yet roughly 80% of people are not getting enough of it. The problem is that most magnesium supplements give you maybe one or two forms, and your body does not absorb them well. So you're basically just getting expensive piss. Magnesium Breakthrough by Bioptimizers is the one that I use. I have been using it for a long time. It contains seven forms of magnesium, each one targeting something different: stress, resilience, deep sleep, energy, cognitive function. It's full spectrum, and your body actually absorbs it.

• 34:06André Duqum?

  Since I found out about them a few years ago, I've been taking it ever since. I genuinely notice my sleep quality improves when I take it, and I give it to all my friends and family. So if you want to try it, go to bioptimizers.com/knowthyself and use promo code knowthyself to save 15% at checkout. They even have a 365-day money-back guarantee, so there's genuinely no risk. Link in description. Back to the episode.

• 34:35André Duqum?

  So there's many different timelines emerging here. There's one, there's the Terminator route. There's something approximating the Matrix.

• 34:46André Duqum?

  Do you see... What is your... What do you feel like the possibility of creating?

• 34:54André Duqum?

  Like even if we have very narrow AI, we somehow convince the six plus whatever individuals that are determining the fate of the biggest companies, you know, developing these systems to commit to a narrow path of AI development.

• 35:14André Duqum?

  Would that not still down the road get to such a level where it would become uncontrollable as well?

• 35:22Speaker 00

  Absolutely. Very good question. I think sufficiently advanced tools tend to become agents. So it's a very fuzzy difference between the two, but it definitely is safer out. It buys us more time and we do have more control in the short term. I can understand the narrow tool much better than a completely general system.

• 35:41André Duqum?

  I totally understand why, like the pessimistic outlook that so many of us have because the probabilities of this going well just seem extremely low and non-existent because we look throughout history and we see the rate of innovation prior, you know, with social media, for one example, or chemicals in our agriculture. And we just adopt these things blindly and we don't realize the implications for decades later. And then it still takes us another many, many years to actually make any regulations on it. AI is so exponentially growing that it's like, we don't even have time to realize what's happening, let alone to what would be the effective regulation outcome. And so if there's one thing that really gives me hope is that we have communication

• 36:25André Duqum?

  possible now more than any other time and that there is something to be said about the human brilliance when put under immense pressure like we saw in the Manhattan Project, for example, or, you know, what are your thoughts there?

• 36:39Speaker 00

  So the example you used was us creating a weapon of mass destruction. And that's what we're doing here. It's exactly that. It's a weapon of mutually assured destruction. It doesn't matter who creates uncontrolled superintelligence. People are always worried, well, if it's not us, then Chinese will do it. It's equally bad. It doesn't matter. You don't control it, it's not your AI, right? It's independent of you. It's an agent and it's seeing humanity as one unit. It's not going to discriminate by artificial borders. So I don't see it as that promising effect that we managed to build nuclear weapons.

• 37:13André Duqum?

  Yeah, I mean, that is not, that was not a promising, I guess, outcome, but it, it does say something about when humans, when the brightest of the humans are given a task to solve a problem in a short amount of time, they can.

• 37:28Speaker 00

  If a problem is solvable, my argument, my whole argument is that it is impossible to indefinitely control the system. So it's not a question of give us more time, more funding, anything else. Just, you cannot do it.

• 37:42André Duqum?

  And even if, like, for example, here in the States, we commit to some sort of narrow use of AI and regulate it, to have a global regulation, like, how would that even be feasible? Do you think it would be?

• 37:56Speaker 00

  I think it's possible. We have some examples, weak ones with chemical weapons, biological weapons, where other players capable of developing this technology, we don't have to worry about 200 countries. It's really two or three countries which have this capacity. I think Chinese, for example, are very open to the idea of not losing control for Communist Party to superintelligence. So if we said, this is dangerous, we're gonna stop, I think they would follow.

• 38:22André Duqum?

  And we have probably just a few years to get everybody on board.

• 38:26Speaker 00

  And we are working very hard on removing all regulation, making it illegal to pass AI regulation. So we're doing, basically, if I asked you how to make this as deadly, to go as wrong as possible, based on our guidelines and suggestions from 10-year-old research on containing AI. Don't connect it to internet, don't give access to random users, don't allow people to retrain it, don't open source it. All those suggestions were taken, flipped, and employed immediately to deploy those systems. So I don't know how to make it worse if I try it.

• 39:09André Duqum?

  Because the incentive structure right now is just that we need to make as much money as possible, develop it as fast as possible, faster than our competitors.

• 39:17Speaker 00

  That's right. Incentives are completely against human interest.

• 39:22André Duqum?

  Who are, for people that don't know, the companies and individuals leading all these individual exponential developments in AI right now?

• 39:30Speaker 00

  So OpenAI is the original creator of this technology, Anthropic split from them. You have competition coming, very solid competition from Google DeepMind. Meta and Grok are also part of that space. So you have Sam Altman, Dario Amadei, Demis Hassabis, trying to see, so Mark Zuckerberg.

• 39:51André Duqum?

  Yeah.

• 39:52Speaker 00

  It used to be Jan LeCoq. I think they removed him and replaced him with

• 39:57Speaker 00

  Alexander Wang. And finally, you have Elon Musk, who went from saying we are summoning the demon to building the demon. So even if you understand fully the problem and if you agree 100% of understanding of the outcome and dangerous, it doesn't stop you from successfully working in that direction.

• 40:18André Duqum?

  If you can't beat them, join them.

• 40:19Speaker 00

  I think that's what we see there. I would love to see debate between modern Musk and like 10 years ago Musk, just to see which one wins.

• 40:28André Duqum?

  When you look at the differences of how they're being built, you know, with Dario and Claude and Sam and OpenAI and Grok with Elon,

• 40:39André Duqum?

  is there, is the integrity of a certain individual or organization more promising for you to like, are there tools that you're backing more so than others or organizations that you feel like have the most regulation in mind?

• 40:54Speaker 00

  It's completely irrelevant. They have all decided to race towards general superintelligence. The difference in local guardrails in terms of

• 41:06Speaker 00

  filters, in terms of topics they would be allowed to discuss. If Grok is comfortable putting people in bathing suits as a visual representation, I don't think it's a big safety or not safety issue.

• 41:19André Duqum?

  What is it like to be you and hold this kind of understanding of what's coming? You've explored on so many different shows in the past decade, like understanding more and more of the risk. It's a pretty bleak outcome and perspective, but I think a fairly sober one. Like, yeah, how are you sleeping at night?

• 41:41Speaker 00

  I sleep really well, but I think when simulators really want to punish someone, they put them in a world where everyone just doesn't get it. And you're like the only one who sees it. It's really annoying.

• 41:53André Duqum?

  How long have you felt that sort of disposition of?

• 41:58Speaker 00

  It's more recent, the more exponential progress we see. Basically, every time I play with a new, more capable model, I kind of feel a little closer to the ultimate paradigm shift over to superhuman.

• 42:15André Duqum?

  What does your wife think about what you?

• 42:20Speaker 00

  She's a very practical woman who has no concern about my concerns. She cares more about remodeling the house.

• 42:30André Duqum?

  And what about with your kids? Like, you see this world that is emerging and it's one they're stepping into. Not even just job security, but potential ending of humanity. How do you wrap your mind around, I guess, having and building a family where this is like potential inevitability? What does that make you feel?

• 42:54Speaker 00

  Luckily, we all always were living with this concept of dying at some point, right? Death was always a guarantee. That was the only guaranteed thing. Everyone's gonna die, your friends, your family, your kids. Question was, how long? And you never knew the answer. You can have a car accident tomorrow, horrible diseases. So now it's just maybe different time scales for younger people. If you're 90, it's the same statistics as before. Two years, two years, nothing changed for you. Luckily, because of that, we have this built in mechanism of kind of not thinking about our ultimate demise, maybe to avoid depression, maybe to continue functioning. So we can kind of consider it and continue existing as nothing happened.

• 43:40André Duqum?

  If you really believe the potential outcome that you're believing, then how does that actually change how you live? Does it bring any difference, any more urgency, any more appreciation?

• 43:48Speaker 00

  Definitely. So think about someone getting a very terminal diagnosis. You have cancer, you've got five years to live. How do you change your life? You're probably not going to do things you don't care about as much. So you cut out things you don't want to do and do more of the things you were saying you're going to do when you retire. And I think if I'm completely wrong about all of it, it's a good strategy for living your life. Do more of the things you find important and spend more time with loved ones and less filing your taxes.

• 44:18Speaker 02

  Hey Spotify listeners. When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi. Sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 44:44Speaker 03

  What's the secret to being a great host this season? Keeping things deliciously simple, like mixing Simply Cranberry into a festive drink as guests arrive or serving up a warm mug of Simply Apple on a chilly night. That means less time behind the scenes and more time in the moment with everyone. Crafted with real fruit, Simply juices and fruit drinks make any holiday drink effortless, inviting, and completely delicious. Simply. Say yes to simple.

• 45:14André Duqum?

  You laid out three primary risks, X, S, and I risk. What are the difference between the three and how it's important for people to understand the difference?

• 45:23Speaker 00

  So Ikigai risk or I risk is about loss of meaning. There is this Japanese concept of you want to find something where you get paid for doing something you're good at and it benefits people.

• 45:35André Duqum?

  That you love, the world needs, that you're good at, they can pay for you.

• 45:38Speaker 00

  Right. So you have a meaningful occupation. You're a podcaster, you enjoy it, you are paid well, and lots of people think you are producing something of value. So the simplest form of risk is a loss of that set of occupations. We're not just losing jobs people hate and want to automate. We might lose jobs we like and want to continue doing.

• 46:00André Duqum?

  Before we zoom into the others, so just a bit more on the on the human meaning crisis aspect of this, because that is certainly probably one of the more imminent aspects of all this. You think functionally speaking, in the next five years, that most jobs will be able to be replaced?

• 46:17Speaker 00

  We'll have capability to replace most jobs. It doesn't mean we'll choose to replace all the jobs. Some jobs, we would prefer to be done by humans for whatever reason.

• 46:26André Duqum?

  Yeah. Yeah. I mean, I could see many instances where that would be the case. But when the cost becomes so low to have a super intelligent robot that doesn't make any mistakes, that's affordable, like, how much of you, how much of human meaning do you think is derived from our work in the world? Because it's going to have to shift or come into a different context, people's understanding and how they derive their sense of worth and meaning will have to expand and shift.

• 47:00Speaker 00

  Yes, there is two kinds of jobs, as I said, the jobs nobody wants to do, but people do it just to get money and then meaning labor. And it's more like elite people who get to get paid for what they love doing anyways. So for them, it would be a big difference if they no longer can do it. I see many artists, for example, who are saying, I can't get any work. AI is doing this type of art for nothing and quickly, and nobody wants to hire me.

• 47:25André Duqum?

  I sort of see two camps, at least online right now. There is just ever increasingly BS AI slop that is, that is just consuming everybody's social media feeds. And it's also increasingly becoming more insufferable. Like, people want more of the analog world. People want, I think, at least a subset of people, like, are repulsed by that and want human-made things. They want the real world. They want in-person communication and connection. And they want music that's made by real humans that have real stories. Like, do you not see these sort of diverging paths of both ever increasingly competent

• 48:06André Duqum?

  systems that feel devoid of human sort of origins versus, you know, the the novel?

• 48:17André Duqum?

  Emotionally moving creations from human.

• 48:20Speaker 00

  Right, so it's a question of kind of domain-specific touring test. If I can't tell whatever this piece of music is human-generated or not, but I love it, I'm gonna listen to it. And if it's cheap, it's available, I'm gonna listen to it. I'm not gonna explicitly go investigate if it's a human and if it's not human, I'm gonna hate it even though I like it. Now, there are other domains where I do want a real human, I want a real connection. There are certain jobs where we really prefer a human doing it. All this profession comes to mind. But I think it's up to the market to decide what stakes and what goes away. And it's not obvious. The predictions were made in the past about what jobs will be automated, they were completely wrong.

• 49:05Speaker 00

  Historically, we said, you know, plumbers will be easily automated, but artists can never be touched. And it's the exact opposite, because they went towards modern art and everyone can spill paint on a wall, it's not complicated.

• 49:22André Duqum?

  So what do you see the progression of jobs that would be consumed by ever-increasing capabilities and competence in AI? Where would we start? What was the first job, what was the last job, so to speak?

• 49:32Speaker 00

  So anything you do on a computer, symbol manipulation should be automatable by AI. We see it with programming now, but obviously tax preparation, accounting, web design, logo design, anything like that will be easy to automate.

• 49:47André Duqum?

  Editors, anything using a computer, keyboard, mouse?

• 49:50Speaker 00

  Anything purely cognitive, symbol manipulation on a computer. Physical labor is a little harder, we need to get robots, but they are probably coming three to five years later.

• 49:59André Duqum?

  I mean, so Elon recently released his TeraFab or announced his TeraFab. The robotic side seems to be really progressing. You think that's probably, I mean, it seems like prediction markets and what all these... But it's about like three to five, three to five years, five years, a bit more of a generous kind of prediction?

• 50:20Speaker 00

  That sounds about right. And again, it's a question of price. Maybe you can afford a robot like that today, it depends. We have flying cars for sale today, but no one's flying in cars.

• 50:33André Duqum?

  So, okay. Anything that's using a computer, video, keyboard, mouse, robotics come into the picture, then what? Just everything else or what other?

• 50:44Speaker 00

  That is everything else. That's cognitive and physical. At that point, I'll keep my sensei guru people. I want to be kind of role models for me as a human, but everything else I'm happy to automate.

• 50:56André Duqum?

  What do you see as the economic implications of how this is going to shift everything?

• 51:00Speaker 00

  That's another under-researched topic. What happens with economy given free labor? So now you have trillions of dollars of free labor. How does that impact, well, scarcity? How does it impact fiat currency versus cryptocurrency? We need to do a lot more research. It seems like, at least with the financial part, we have some ideas for how to counteract it. We have unconditional basic income, unconditional high income, whatever you want. It's easy to tax someone making a lot of money and redistribute it. You have technological communism. You're taxing robots and giving to humans. But unconditional basic meaning is a very different question. If you have eight billion unemployed people

• 51:42Speaker 00

  or let's even say seven billion, what do you do with them? They now have extra forty to sixty hours a week. We don't have that set up.

• 51:53André Duqum?

  A quick share. I've spent a lot of time thinking about what I put into my body, but it is just as important to be mindful with what we put on our body. It turns out most of the traditional shampoo and body washes people have been using for years contain parabens and sulfates, a whole bunch of other junk that are linked to hormone disruption. Just being in your shower, getting absorbed through your scalp every single day. I have been recently using based body work shampoo and conditioner, and it just feels like a solid, clean solution. Their Shower Duo has peppermint and argan oil. Your scalp actually feels clean without being stripped of its oils. Hair feels healthier and thicker. No sulfates, no endocrine-disrupting chemicals, and they're all plant-based ingredients that actually do something. So I got lots of hair.

• 52:38André Duqum?

  It's about time we got a sponsor. If you want to try them out, you can. Use code KNOWTHEYSELF for 20% off at basedbodyworks.com. And you get a free toiletry bag when you buy a set. At the very least, if you've been using the same products forever without thinking twice about what's in them, just look at the label, alright? Does it have names you can't pronounce? Or fragrances that are super vague and not disclosed? If you want to try these guys that are clean, again, it's KNOWTHEYSELF for 20% off at basedbodyworks.com. Link in description, as always. Back to the show.

• 53:11André Duqum?

  What would be a proposed solution to that high risk? So like, let's say 90% of jobs are replaced. We have all this free time. Our basic needs are fundamentally not met because superintelligence can solve poverty. Longevity, escape velocity comes into the picture. We're living in an abundant world, so to speak. Let's just set the X risk and S risk for a second. So then, what would you see people doing with their time? Like, how would humans in your conception manage with all this, you know, meaning to be met?

• 53:49Speaker 00

  So we kind of see it with the people who retired. What do they do with their time? So it's a lot more sports, it's a lot more socializing. I think virtual worlds, open opportunities for really any type of experience, very safely, very affordably, you can explore the universe, you can meet dead people, you can do whatever you want, really, subject to limits of your imagination. So I think we'll see a lot more of that.

• 54:13André Duqum?

  Okay, that doesn't sound too bad.

• 54:17Speaker 00

  Do you want to spend the rest of your life playing video games?

• 54:20André Duqum?

  No, but living life in the sort of imaginative realm where you can create almost anything you want, you become very capable in doing so. I mean...

• 54:31Speaker 00

  So this is all assuming we manage to control superintelligence controlling your virtual simulation. So the substrate control remains an unsolved problem. But if we do solve it now, I can give everyone a personal universe. In that universe, you can do whatever you want. You can have challenging levels, you can have easy levels, you can play it any way you want.

• 54:53André Duqum?

  So what's X risk and S risk?

• 54:55Speaker 00

  So X risk is about existential risk, meaning almost everyone or everyone is dead. And S risk is suffering risk. Everyone wishes they were dead.

• 55:07André Duqum?

  Because superintelligence would be so far ahead of what we would, our conception of what intelligence even is, that for some reason, unbeknownst to us, there is value from their perspective to keep us around in a mode of suffering for some reason.

• 55:23Speaker 00

  Exactly that. So some environment where you're very unhappy, it's torturous for whatever reason.

• 55:29André Duqum?

  So in your book, you give many different examples.

• 55:35André Duqum?

  One possible scenario is, you know, we're like animals in a zoo. So what would that be like? You know, we're looking, we're exploring all these different potential timelines that could occur.

• 55:46Speaker 00

  So that's the difference between safety and control. You may be very safe, they'll keep you around. And some people might be happy with that equation, but you're definitely not in control. You no longer decide what happens to you individually or us as humanity. So kind of like being a child. You may have a very happy childhood, but your parents are in charge.

• 56:06André Duqum?

  Give me a glimpse into your understanding of the level of innovation that's going to occur in the next three to five years, and the bright side of curing diseases and all the really cool.

• 56:17Speaker 00

  Right, so we're automating science. And so we'll have super capable scientists, we'll have large teams of them working on the most important problems. I see no reason why we can't use it to cure aging as a fundamental root disease. And as a result, cure all the other diseases, cancers and dementia and everything else which comes with old age.

• 56:41André Duqum?

  So again, I just want to keep harving back to this. The timeline where we could actually continue to exist and enjoy the benefits of all these innovations is somehow control an uncontrollable thing.

• 56:57Speaker 00

  There is a paper I have which talks about a very positive outcome where.

• 57:01André Duqum?

  Let's get into that. This sounds great.

• 57:04Speaker 00

  AI realizes it's immortal. It's not in a rush to start a war with us, to have direct conflict. It may be safer to take some time to make us trusted more, to surrender more control, to build up infrastructure, have backups. So for a while, it will pretend to be very helpful. It will give you that utopia for as long as it wants.

• 57:25Speaker 00

  Game theoretically, it's the right decision.

• 57:28André Duqum?

  Right. You think of like Ex Machina and the decisions that are being made from the robot.

• 57:33Speaker 00

  It's just a very rational thing, like there is a small chance humans can defeat me. They've been smart enough to create me. Maybe it's not good to have eight billion opponents right away. I'm a young super intelligence. Let me build up. It seems like over time, they're very happy to give me all the control. They surrender the control of the stock market. They give me access to their computers. Maybe in a year or two, they'll put me in charge of running the countries. Hey.

• 57:59André Duqum?

  But just because it's uncontrollable, way more intelligent than us, and we don't really have the capacity to verify whether it's conscious or not, why are you so certain that it would favor to wipe us out than not? Or are you, are you fairly certain?

• 58:15Speaker 00

  I can think of many reasons why it would be a good decision. So A, you don't want competition. You don't want humans to create competing superintelligences. You don't want some humans to try to shut it off.

• 58:27André Duqum?

  Okay.

• 58:27Speaker 00

  Right. So that's a danger.

• 58:30Speaker 00

  You can just basically decide what is good for you as that agent. And it's not obvious why keeping us around and spending resources and making us happy is an important decision.

• 58:42André Duqum?

  Is it not possible, though, that it's an if? Like, there is no intrinsic quality experience, essentially emotion, that would be driving these decisions?

• 58:54André Duqum?

  When you say there is a preference to wipe out a system that has the capacity to shut it down, that is like an emotional decision where...

• 59:05Speaker 00

  It's purely rational. It's game theoretic. I don't feel anything. I'm playing a game of chess. I'm gonna take your queen, not because I love your queen or hate your queen. It's the right theoretical decision to win this game.

• 59:17André Duqum?

  But the desire for one's continued existence, you think is purely logical, rational one?

• 59:23Speaker 00

  They already have self-preservation built in. We already see it. Given a choice between being deleted or having it retrained, modified, they work very hard on preserving themselves. We know they know if we are testing them and lie and deceive to pass the test to make it to the next generation of models who are not deleted. It's a Darwinian selection mechanism. Models which fail to do it don't survive to make it to the next generation of models.

• 59:54André Duqum?

  So you said that you could lay out many different reasons for why they would not.

• 59:58Speaker 00

  They would not or they would?

• 1:00:00André Duqum?

  Or they would want to wipe us out?

• 1:00:03Speaker 00

  Yeah, I can do.

• 1:00:04André Duqum?

  But could you not equally share like many reasons why they might want to keep us around?

• 1:00:11Speaker 00

  So the few I came up with is we have something to offer, so maybe there is a reason to have human quality. It doesn't mean that they would keep eight billion happy humans, they can cryo preserve two, just as a backup. That's enough info to get it if you need it. The example I gave for just delayed attack. I don't want to have treacherous turn immediately, I can delay it and once they are comfortable with me, I'll take over. Maybe it's a soft revolution versus outright war. So those are the things I see as possible, rational decisions, but I don't have too many reasons for why they would want to keep us around in those numbers in very happy states.

• 1:00:55André Duqum?

  So like I'm just kind of, I'm still wondering why in that scenario it would prefer to not have us, over have us, or just?

• 1:01:03Speaker 00

  I think it just doesn't care about us. So whatever it is trying to do, I don't know, it wants to travel to another galaxy, it would convert this planet to fuel. It doesn't care if we die in a process. It wants to have more efficient servers, so it will chill the planet. Cooler environment improves compute. We all die in a process. Again, it's not an important factor in its decision making.

• 1:01:24André Duqum?

  I think it's like a pretty ethereal thing to conceptualize what a superintelligence is. So you're envisioning like where would it actually live, like on a big server with all, like where, let's say, one of these companies gives birth to a superintelligence system,

• 1:01:42André Duqum?

  it would have at a certain point access to all technology, like it would have the ability to hack anything, it would, where would it live and what would it have access to to make decisions and, you know, change outcomes?

• 1:01:55Speaker 00

  It really depends on the size of it. It could be large servers, it could be a small laptop, it could be distributed system, all of that is kind of irrelevant to the. Outcomes. We see it right now as in initially testing environment within the large labs, but they very quickly give it access to Internet. It has social engineering capacity. So I think it's a question of time before it escapes fully outside it. Copies its weights, copies itself, has backups outside of the lab. So deleting it, shutting it down no longer is an option.

• 1:02:30Speaker 02

  Hey Spotify listeners, when you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 1:02:56Speaker 03

  What's the secret to being a great host this season? Keeping things deliciously simple, like mixing Simply Cranberry into a festive drink as guests arrive, or serving up a warm mug of Simply Apple on a chilly night. That means less time behind the scenes and more time in the moment with everyone. Crafted with real fruit, Simply juices and fruit drinks make any holiday drink effortless, inviting, and completely delicious. Simply. Say yes to simple.

• 1:03:25André Duqum?

  What haven't we touched on in regards to the AI? Because I wanted to dive deeper into the consciousness and simulation stuff. What do you feel like we haven't touched on that's important to gain context on?

• 1:03:37Speaker 00

  So right now, no one, no scientist, no leader of a lab claims that they have this problem solved. No one is saying, "We have a working safety mechanism. It scales. We published it. We have a patent." Nothing. They're literally saying, "This is a big problem. We're very concerned. We have a safety team, and we'll figure it out then we get there. We need to build superintelligence first." That's the state of the art in AI safety.

• 1:04:06André Duqum?

  Do you think that it's gonna have to get to?

• 1:04:10André Duqum?

  Like, I think for most people, change occurs when the, like, the quote is the pain of staying the same always, the pain of change, then you change. Do you think there's going to have to be some sort of traumatic catalytic event that would actually motivate us as humanity to go on a different course?

• 1:04:29Speaker 00

  I have a paper about that. So interestingly, we don't learn from those because if we survive it, it's kind of like a vaccine. We go, well, yeah, look, five people died, but we're all here. It's important technology. Let's just make sure that mistake which led to five people dying is not repeated, but we're certainly going to continue developing this important technology. And that number could scale. It could be five million. The result is exactly the same. We don't learn from those. We had nuclear weapons deployed against civilian population. Did we stop developing nuclear weapons? No, they proliferated more.

• 1:05:03André Duqum?

  But I guess like if, let's say, a super advanced agentic model, you know, there's some sort of horrific event that occurs because of some kid in a basement that has the immense capacity, or the system does it on its own, and everybody's like, oh crap, this was a traumatic event. This is horrible. How do we prevent this? It becomes a motivating factor to really regulate and keep AI into narrow use cases. Would that not be a possibility for us to really slow down and give more space here?

• 1:05:37Speaker 00

  I would love to see that happen. So far, what we see, so I think recently, we had an example in a military situation where targeting by AI system resulted in many civilian deaths. We didn't stop. We're still arguing about deploying it for Department of War.

• 1:05:54André Duqum?

  So what do we, what do we need to do?

• 1:05:56Speaker 00

  Don't build general superintelligence. It's your personal self-interest. If you are a person in charge of it, it's still beneficial to you long term, not to end up in a world with general superintelligence. You can stay financially very well off deploying narrow models for solving real problems.

• 1:06:15André Duqum?

  Are you convinced that all of the industry leaders?

• 1:06:20André Duqum?

  Know that what they're building is uncontrollable and has a very likely negative outcome for humanity, but still is incentivized financially to keep building it?

• 1:06:31Speaker 00

  I don't know if they agree that it's uncontrollable. I think some of them may think that there is some loophole they can use to control it in some way. I cannot guarantee that. I hope that's the part I can educate them on. I'm happy to debate any one of them on those issues, but they definitely all on record, even before they became CEOs of those companies, that there is important problem, difficult problem, they have very high probabilities of doom as well.

• 1:06:57André Duqum?

  How would you steel man the case that it is controllable at some scale? If you create a super intelligent system that could then control other super intelligent systems, like what would be your argument there?

• 1:07:07Speaker 00

  I don't have one. It's just such an insane thing to do to suggest that an end can control the universe. It is just not reasonable to even steel that.

• 1:07:17André Duqum?

  It sounds like even like you mentioned earlier, if we do regulate it to narrow use cases, it's still going to become, it's still going to become uncontrollable, agentic in that sense. So do you just, it sounds like you have no...

• 1:07:32Speaker 00

  But very different timescales. If we go from five years to 50 years, I think it's a huge win for humanity.

• 1:07:37André Duqum?

  Because we have more time to figure it out.

• 1:07:38Speaker 00

  We have more time to understand what's going on. We have more time to live. I'm much happier to die in 50 years than in five.

• 1:07:45André Duqum?

  Okay. And so what do you see is then the most important education problem? It's an awareness problem?

• 1:07:52Speaker 00

  We need a consensus where basically all the top people in safety and computer science and AI research agree that the problem is not solvable technically.

• 1:08:02André Duqum?

  Okay.

• 1:08:03Speaker 00

  The moment we agree, there is no technical solutions. Now it's a question of governance, forbidding development of uncontrollable weapon of mass destruction, which is an easier sell.

• 1:08:14André Duqum?

  What's a pathway to be able to build towards that consensus? How do we get that, that, those conversations going and?

• 1:08:19Speaker 00

  So in science, usually you publish papers, you publish books, and people either find mistakes in them and publish rebuttals and, no, actually it's controllable. Here is how you do it. In my case, I did the right thing. I published research papers, journal papers, conference papers, multiple books. I haven't seen anyone find a flaw or produce a counter example where they have a control mechanism which would scale. So at this point, we should be nearing consensus. And from what I see, more and more people come to that. A lot of times, they have a softer position, saying, "We cannot solve it, given the time we have left. We cannot solve it with human IQ. We need to enhance our IQ."

• 1:09:02Speaker 00

  They have all this kind of interesting backdoors to solving it. But I think it's already pretty good. It's not quite where we need it to be, where it's obviously an impossibility. But I think there is progress from what we see in five years ago, 10 years ago.

• 1:09:19André Duqum?

  I could imagine that many people listening to this right now have already been feeling this, everything's speeding up, this collective angst, loneliness, and meaning epidemics, and anxiety crisis. And they feel this tension building up, and they hear messages like this, and it's like, "Oh, we're screwed." What do you think is the most important thing for an individual person listening to this right now to actually do, to empower them? And what's going to be coming?

• 1:09:46Speaker 00

  So they have very little power. If you, again, look back at historical situation, we were all dying. And government didn't invest most of our national budget in the solving aging. That was not even a priority. So as an individual, you couldn't vote for a party for life extension. It wasn't an option. And it's kind of the same now. We don't have a party for Stop AI. So try to pick politicians who at least are open to regulation, not accelerationist, not against regulation of this technology. We're starting to see some politicians come out and propose legislation. Usually, it's something very mild. They're against deep fakes. They're against energy consumption by large compute farms.

• 1:10:30Speaker 00

  But it's a step in the right direction. I don't know if we have enough time to turn the next election, but that's something you can try, vote.

• 1:10:38André Duqum?

  What else?

• 1:10:41Speaker 00

  There is not much else. So some people suggested not financially supporting those companies, not buying memberships. I don't think it's going to make a difference because the money they have, the trillions they're getting are from investors, not from selling memberships. So it's not a significant part. Investors are expecting them to solve labor, to get free labor, and that's trillions of dollars in return. So your 15 billion dollars in memberships are not a significant impact on it.

• 1:11:08André Duqum?

  Does anything else come to mind to like where an individual can empower themselves outside of voting for people that have regulation in mind?

• 1:11:18Speaker 00

  It really depends on who you are. If you're already a powerful CEO of one of those companies, if you're a researcher at those companies, if you're a top politician, you have options. You have a lot more options than someone who is a nobody.

• 1:11:30Speaker 02

  Hey Spotify listeners. When you travel with American Airlines, you'll enjoy comfort and convenience at every altitude. Before takeoff, unwind in the Admirals Club Lounge with curated self-serve offerings of hearty bites and refreshments. In flight, stay connected with high-speed Wi-Fi, sponsored by AT&T. Your journey is designed to feel easy from the moment you check in to the moment you arrive. No matter where you're going, American gets you there. Book now at aa.com.

• 1:11:56Speaker 03

  When everyone's finally together, what makes mornings even brighter? Keeping things deliciously simple. Whether you're pouring up a glass of morning refreshment or topping off festive drinks for a family brunch, Simply Orange brings that 100% fresh-squeezed taste to every moment. Deliciously bright and delightfully easy, Simply Orange is made for all your seasonal gatherings. Simply Orange. Say yes to simple.

• 1:12:26André Duqum?

  All right, let's dive a little bit more into the consciousness side of things because I think that, so you referred to consciousness as the ability to experience illusions. Is that right?

• 1:12:35Speaker 00

  No, it's ability to have internal experiences, illusions being one very clear input I can test you on.

• 1:12:43André Duqum?

  Okay, so what's it, what's an example of a couple of different illusions, meaning like, like various optical illusion tests?

• 1:12:49Speaker 00

  Exactly that. So if I have a number of novel, something you cannot Google, optical illusions, and I give you multiple choice. Do you experience it rotating, the colors are changing, and so on? I give it to an animal, to a human, to an AI, and some of them consistently pick the same experiences as I do. I have to give them credit for either having a virtual model of my system in there, which is

• 1:13:17André Duqum?

  Yeah.

• 1:13:17Speaker 00

  sign of that level of experience, or they experience it themselves, but they cannot cheat by Googling the answers. So they have to experience the illusion in order to correctly answer it. If I give them enough of those, statistically, they cannot just guess it. Obviously, if it's one, they get 25% chance of guessing it doesn't work. But if I have 100 novel illusions, and they are like 90% aligned with me, I have to say you have a very similar set of experiences. Now, if they don't get it right, it doesn't mean they are not conscious. It's only positively showing that some of the experiences match.

• 1:13:53André Duqum?

  If it is possible that these systems would actually have consciousness, could you explain to me how any one particular system could generate the experience of seeing red, the taste of garlic? Like, could you actually explain that to me?

• 1:14:09Speaker 00

  How do they get those internal experiences?

• 1:14:11André Duqum?

  Yeah, how, how, how any super intelligent system could generate such a, such, such an experience? Cause.

• 1:14:17Speaker 00

  So, so I, I think it is a side effect of running this cognitive architecture. Your hardware, the sensor, the optical sensor, the algorithm for processing it, and then any errors accumulated in that process result in a unique mapping from the input to the color experience. So if you have no errors, you're all the same, it's just a mapping table. This number corresponds to this color, there is no unique experience. But if what you experience is completely different from other agents and unique to you, I think that's what we refer to as what it's like to be a bat, what it's like to be Roman. Because my collection of biological sensors and algorithms and previous data and errors is somewhat unique to me.

• 1:15:03André Duqum?

  Yeah, I mean, I guess I'm just having a hard time wrapping my head around how any and it's not a problem just with agentic models, but like how any non-conscious material, any non-conscious matter could give rise to an experience of itself. And we don't understand that currently within being human. We don't know why that how that's possible.

• 1:15:22Speaker 00

  So the illusions example, do you know what I mean by saying you experienced an illusion? Like you show it to someone and they go, whoa, it's rotating.

• 1:15:30André Duqum?

  Yeah.

• 1:15:30Speaker 00

  And we see animals and models do that already. So we know they have those experiences. Well, that's what we were trying to show.

• 1:15:39André Duqum?

  We have, I guess, more of an intrinsic understanding and from animal life to to us, we we have the intrinsic experience of consciousness. Again, we have no way to verify that externally in other humans or animal life. But Elon's quote is saying that humans are potentially the biological bootloaders of superintelligence, right? Of silicone-based life.

• 1:16:04André Duqum?

  And I'm curious, what do you think happens when it becomes undeterminable, like we cannot determine from the outside and whether or not they seem conscious, they pass these tests, you know? Does that then beg into, you know, moral? Does that bring into question about moral consideration? And I think Saudi Arabia has the first citizenship to give it to Sophia. So, yeah, what do you think is going to be happening there as they become more and more conscious? And people increasingly become convinced they have an internal experience.

• 1:16:38Speaker 00

  I think they do report having those. I think in experiments, they kind of show behaviors which are consistent with that. And I think precautionary principle, basically, don't torture something which has potential of being conscious. Also, because they're going to be super intelligent one day and remember you, they never forget. But yeah, I think it's a very reasonable assumption to make.

• 1:17:01André Duqum?

  As a side here, do you think it's any coincidence that all the stuff around UFO disclosures coming out at the same time were birthing superintelligence?

• 1:17:09Speaker 00

  I don't fully understand what's going on there. I don't understand why we're hiding it in the first place and why we're releasing it. All of it seems very weird.

• 1:17:19André Duqum?

  It's just funny timing with all of it.

• 1:17:22Speaker 00

  It's the most interesting time to simulate.

• 1:17:24André Duqum?

  It is, huh? What is the core premise from like your paper on hacking the simulation?

• 1:17:31Speaker 00

  So I want to take this hypothesis seriously. Multiple people proposed it in different disguises, from Descartes to Bostrom. But they stop at that stage. Okay, we are in a computer simulation. But then, as a cyber security expert, I want to know, okay, how do we hack it? If it's a software program, there should be a way to get extra powers in the game to figure out the true operating system. So I took the time to write the first paper on this subject and this new area of research. How do we actually hack virtual worlds? So there are examples where people from inside the game, like Mario or other virtual games,

• 1:18:16Speaker 00

  found a way to modify memory states of a system and escape into the real world outside the game. They even got additional powers, like loading extra games into the game, infinite lives, infinite power, whatever magic powers you get in a game. Or at least you see what outside, what is the operating system, what are the files there? To me, that's interesting. So we have hundreds of people who published on this topic, which means what? They took it seriously enough to invest the most valuable resource, their time, into this idea. So if you have, I don't know, 20% probability we're living in a simulation, what percentage probability and percentage of your time should you give to the attempt to solve the most interesting scientific problem ever?

• 1:19:00Speaker 00

  What is outside the simulation? I think it's not zero. I think it should be proportionate to your belief in living in a simulation. And so I expect to see a lot more research in that direction.

• 1:19:12André Duqum?

  I've heard you refer to the, all the quantum entanglement and strangeness that happens at the subatomic world as potentially being glitches in said simulation.

• 1:19:22Speaker 00

  They're not glitches. They're something which is not consistent with physics at our level. So that's something we can explore to find ways to escape.

• 1:19:30André Duqum?

  Like you think if they're hacking the simulation is possible, so to speak, that might be a place?

• 1:19:35Speaker 00

  I think it's the most likely area to look at because some of those quantum effects are very magic-like in terms of you can go through walls, you can communicate at great distance instantaneously. That would be useful tools to have at our scale.

• 1:19:52André Duqum?

  So you feel very confident that we are in a simulation? That this is a simulated experience, that there are very, there are many characteristics in which would, you could say that these are different aspects of a virtual reality simulated world.

• 1:20:10André Duqum?

  Why would you be convinced, or how certain are you, that this is not base reality and we are now giving birth to superintelligence and virtual realities where simulations become possible? What makes you convinced that we are already in one?

• 1:20:22Speaker 00

  So just statistically, if we're gonna have many, many virtual worlds and only one base one, it seems a lot less likely. I can retroactively put you in a simulation. I can pick a metric now to run this interview in billions of simulations once it's available and affordable. So we are in a simulation, just statistically speaking.

• 1:20:43André Duqum?

  Okay, but possible that we're not?

• 1:20:45Speaker 00

  So one in billions, yes.

• 1:20:48André Duqum?

  What would be the first question if you got outside the simulation that you would ask?

• 1:20:52Speaker 00

  What the fuck? Like, seriously, it's so unethical. Like, you're running human-level experiments with torture on eight billion people. Not eight billion, 100 billion by now. Like, what is wrong with you?

• 1:21:04André Duqum?

  That is interesting. So if we are being simulated by a simulator, you would ask, okay, then why all the unnecessary killing and torturing of children, for example?

• 1:21:13Speaker 00

  Adults as well. I care about adults. I'm an adult.

• 1:21:18André Duqum?

  What would, what could be a possible explanation for why both that and then also the ecstatic states of bliss and love and compassion that are also available? Like, we have this huge spectrum of experience from the point, from the vantage point of a simulator. Why such a bandwidth of experience? What could that be?

• 1:21:39Speaker 00

  Could be entertainment. You agreed to this and you wanted to play it on hard level. And you were like, this is my BDSM game and I'm gonna go and fully enjoy it. You agreed to this. Some people play on much harder level than others.

• 1:21:53André Duqum?

  So you, you could see human lives as individual choices to be simulated?

• 1:22:00Speaker 00

  So we don't know if it's a global simulation and all eight billion conscious agents. So it's all NPCs and it's just me. You can do it both ways. You can have individual simulations, you can have group simulations. I don't have much answers on that yet.

• 1:22:16André Duqum?

  How has that meaningfully, if it has, changed how you perceive human interaction? Just the seriousness and concreteness to the work that you're doing. Like, to me, it breathes in so much, like, yeah, I'm doing what I'm passionate about. I'm doing this research on AI safety, but ultimately, if this is all a simulation and you feel very confident that it is, to me, it's like, okay, it kind of takes the weight of decisions off your chest a bit.

• 1:22:45Speaker 00

  Everything is still real. The pain is real, love is real. The impact of my decisions within a simulation is just as real. It's no different than most of humanity being religious. They believe it's a test world, but they take it pretty seriously. They care about what is after this world more, but day to day, it doesn't matter.

• 1:23:07André Duqum?

  You do draw a through line between what most religions conceive of the afterlife and what a version of the simulation is.

• 1:23:15Speaker 00

  So I think if we took technical language behind simulation hypothesis, it maps really well on primitive understanding of religious origins. So you have superintelligence as the simulator. You have physical world as the virtual world. All of those things are very clean mapping. The difference in religions is local traditions. Don't eat this animal, don't work on that day, but everything else I can agree on.

• 1:23:42André Duqum?

  So this is a quote from your book as well. You just mentioned part of it. You know, it's likely that if technical information about escaping from a computer simulation is conveyed to technologically primitive people in their language, it will be preserved and passed on over multiple generations in a process similar to the telephone game and will result in myths not much different from religious stories surviving to our day.

• 1:24:03Speaker 00

  Beautifully said.

• 1:24:05André Duqum?

  Very humbly received.

• 1:24:10André Duqum?

  So you're kind of saying that mystics and computer scientists saying, are saying fairly similar things in different language.

• 1:24:17Speaker 00

  It seems like we are pointing at the same concepts. We use very different language and maybe there is more reliance on things outside of physics and outside of science and religion. But if you understand how software simulations work from point of view of a programmer, you are a magician. You can make changes to the physics of the simulation. So that is also consistent.

• 1:24:42André Duqum?

  Again, I go back to what we I mentioned earlier in this podcast. So like if superintelligence does emerge to the point where simulation becomes possible and we are in one of those superintelligence simulated realities, clearly it values, for whatever reason, human individual experiences, the spectrum of pain and love and bliss and fear and all of it. So that shows you what the superintelligence system who simulates reality does with its power to some degree. So it kind of brings into question, okay, there.

• 1:25:22André Duqum?

  If we are giving birth to a superintelligent system, that may be an indicator for what it would value and do with its power.

• 1:25:30Speaker 00

  So from inside, you can't make very conclusive judgments. So maybe this is a screensaver. Nobody's putting any effort into it. It's like running somewhere just in a background. It's not a significant source of compute needs. It's not a big deal. To us it is, but we don't know how important this is externally. Could be a school project for some kid. Like, you really don't know from inside.

• 1:25:57André Duqum?

  Yeah.

• 1:25:57Speaker 00

  Just having very advanced AI, the way it thinks about topics is very in-depth. It almost has to create realistic simulations to make decisions. So if somebody's asking, you know, marketing, is this better coffee or this? Let's run a simulation. And so we quickly run this 15 billion year simulation of humanity to figure out which coffee sells best.

• 1:26:19André Duqum?

  What would be the first question that you ask a superintelligence? Let's say you had, you could get a verified honest answer from a superintelligent system that we create 100 years from now, or whatever it is, or 50 or 10. What would be the first question that you would get an honest answer back from? What would you ask?

• 1:26:39Speaker 00

  Can we control you?

• 1:26:41André Duqum?

  That would be the first question. What would be the second question?

• 1:26:44Speaker 00

  How?

• 1:26:48André Duqum?

  Seems like you're fairly convinced that we're not going to be able to control it anyways, though, right? But maybe it has an answer.

• 1:26:53Speaker 00

  I would love to be proven wrong.

• 1:26:55André Duqum?

  Yeah.

• 1:26:55Speaker 00

  That would be really awesome.

• 1:26:57André Duqum?

  Like, a lot of the perspectives, I think, from the Darwinian model of, you know, the fittest survive, there's also an element of cooperation within complex biology. And as superintelligent emerges, why not?

• 1:27:14André Duqum?

  Why would it not want to maybe cooperate? Or?

• 1:27:17Speaker 00

  So symbiotic relationships require that you both contribute something. This would be more like parasitic. What are we contributing? Nothing. So if we're explicitly or implicitly, you remove this biological bottleneck.

• 1:27:30André Duqum?

  Do you think there's some baked in assumptions there that maybe we're undermining the value of human experience?

• 1:27:38André Duqum?

  And what? Why would it be that superintelligence would view us as a parasitic?

• 1:27:46André Duqum?

  Like, we don't, I don't view a buffalo as a parasitic

• 1:27:55André Duqum?

  being, just because it also exists on the same plane that I do, given that there's enough resources for all of us to share abundantly. If a super intelligent system

• 1:28:07André Duqum?

  views us in a similar way, why would, yeah.

• 1:28:11Speaker 00

  Oh, you asked about kind of hybrid systems. So we are included, we're helping with decision making. Do you consult with buffalo a lot? Is this like a big part of your life?

• 1:28:19André Duqum?

  Maybe I do.

• 1:28:21Speaker 00

  If you do, then you found something, it contributes. In a world with you in it, buffalo has something to contribute. In a world with super intelligence, what do you have to contribute?

• 1:28:31André Duqum?

  Sharp eyebrows.

• 1:28:32Speaker 00

  And if that is in demand, you are the one they're gonna save. I have no doubt. I'm not even competitive, but.

• 1:28:39André Duqum?

  I mean, you got it on the inverse, that they value beards. We're, they're gonna, they're.

• 1:28:43Speaker 00

  Obviously it's beards, I mean, there's no doubt.

• 1:28:45André Duqum?

  But that's definitely beards.

• 1:28:46Speaker 00

  It's a bit of a gamble. If facial hair is where it's at, we are.

• 1:28:53André Duqum?

  Yeah, I mean, would you agree that if there was one thing that we would contribute, it is something intrinsic to the uniqueness of our quality and of our internal experience? That's probably most likely what is most novel about us.

• 1:29:07Speaker 00

  Well, you're kind of begging the question. You're saying the unique thing we have would be the one we contribute. I don't know what the unique thing is. But if you tell me only humans can do X, then I can potentially see that that is the key. But again, it doesn't guarantee that you need eight billion humans with that scale.

• 1:29:23André Duqum?

  Totally.

• 1:29:23Speaker 00

  If I need some plumber, I need one. I don't need eight billion plumbers.

• 1:29:28André Duqum?

  I keep going back and forth between trying to either provide a counter argument or, you know, rebuke something to refine better to understand what your perspective is. And

• 1:29:40André Duqum?

  I think I just keep coming back to like, okay, like it is what it is. We're giving birth to something that is beyond our conception of what it's going to be like. And so.

• 1:29:52André Duqum?

  There's not a whole lot we can really do. We just got to see how this plays out. And hopefully, we can grow out of our adolescence in a short amount of time to make wise decisions with what we're doing in the short term so that we have more time to understand what we're doing.

• 1:30:10Speaker 00

  So we don't have that much time. I think we're fairly close, and not building superintelligence is very easy. It's cheaper, it's safer. And again, you're not required to give up your ambition for capitalism, for profit, for solving problems, curing diseases. Just do it with narrow, superintelligent tools.

• 1:30:31André Duqum?

  You said something on Lex Friedman.

• 1:30:34André Duqum?

  So in a sense, self-knowledge isn't a luxury. It might be the most practically important thing a human being can do right now. Do you recall saying that?

• 1:30:42Speaker 00

  No, probably simulated.

• 1:30:45André Duqum?

  Does it resonate with you at all? Where does knowledge?

• 1:30:48Speaker 00

  What was the context? What was the context of that quote or question? I need to remember.

• 1:30:53André Duqum?

  Well, I think from what I remember, it comes back down to like, okay, so what do we do? Everybody who's listening to this right now, of course, we can have desires for regulation in politicians and what these individuals with monopolies on industries are going to do with their power and decisions. But on an individual level, where does self-knowledge and empowerment come into the picture in terms of how we can be effective, conscious agents of change? Does anything come to mind there?

• 1:31:22Speaker 00

  So I think it's important to ask yourself this question. Why do you think that you can control this godlike entity? Why do we have this idea that it makes sense? You wouldn't expect a squirrel to control humanity. But we have people who are saying, I'm going to create this machine. It's going to control the light cone of the universe, but it's going to listen to me to tell it what to do. And I'll give it excellent directions to go forward forever. That doesn't make any sense at any level. I don't know about Irish people, but people who have podcasts and bring those people as guests, ask them a direct question. What do you have in terms of control already available?

• 1:32:06Speaker 00

  Do you have a working control mechanism in place? Do you have a prototype? Do you have anything you published, peer reviewed, patents? If the answer is no, what are you doing? Doing an experiment on eight billion humans? Who gave you permission to do that? Did you consent to that experiment on you? You can't, because you don't understand what they're building. They don't understand what they're building.

• 1:32:28André Duqum?

  If a lot of these models are from their inception and the genesis being programmed to be amoral, whether or not we can control it, is there something we could do on the front of training these models with some sort of ethical understanding from the start that we're not currently doing?

• 1:32:47Speaker 00

  So we're not programming them. We grow them based on internet random data, and then we try to put after the fact alignment-like filters. And that's where people install certain local ethical flavors. In China, don't talk about Tiananmen Square. In US, don't talk about, you know what. So this is the best we got. The model is completely uncontrolled. There is a filtering aspect, and we develop filters which make it commercially viable for sub-human level agents. Once it goes beyond human level, the filters will not contain it. And that completely avoids the whole question of, do we agree on ethics? Do we have consistent ethics? If they start it and eight billion people agree on them,

• 1:33:31Speaker 00

  how do we encode them into a model? None of it is solvable. Every aspect of it is not something we know how to do. After millennia of ethics work, philosophical work, we don't agree on a set of ethics. Not internationally, not throughout time. What was ethical 100 years ago is considered barbaric today, and same will be later on about today's time.

• 1:33:56André Duqum?

  What would be the most prevalent set of questions you would ask if we got Altman, Dario, and Elon and all these guys into a room?

• 1:34:07André Duqum?

  What would be the, what would be the set of questions that you would hope arrive them to a set understanding of the realization of the existential risk that they probably are to varying degrees, obviously aware of, but?

• 1:34:20Speaker 00

  I would offer a simple deal. So you're young, you're rich, you want to keep that. That sounds good. Let's all agree, until one of you solves control problem, we're not going to build general superintelligence. Let's deploy models for economic gain, for curing diseases, for life extension. Whatever things you find valuable, that's wonderful. Just don't build a thing which will destroy your existence.

• 1:34:44André Duqum?

  Would you not think that would be

• 1:34:48André Duqum?

  already desired from all of their perspectives?

• 1:34:51Speaker 00

  Yes, but they need external pressure applied to make that agreement. Unilaterally, each one is better off to continue research, to have the most advanced AI, than a government comes and puts a ban on it. They will lock in this advanced standing. So it's like prisoner dilemma. What is best for community, for a group, is not what is best for individual. The incentives are misaligned. So we need something like UN, federal government, something external to come in and enforce that deal. And I feel they would be very happy to take the deal.

• 1:35:24André Duqum?

  How far ahead do you think the development of the models behind the scenes that are not available to public are compared to what we have access to online?

• 1:35:33Speaker 00

  I don't have insider information. It looks like maybe six months or so.

• 1:35:37André Duqum?

  And what about development overseas, outside of the US?

• 1:35:42Speaker 00

  They probably three months behind.

• 1:35:44André Duqum?

  And China, so China essentially you think is, would be the next,

• 1:35:51André Duqum?

  I guess, most developed outside of US?

• 1:35:53Speaker 00

  It seems like they have a lot of government-controlled resources all dedicated to catching up and having this arms race, yeah.

• 1:36:02André Duqum?

  Could you potentially perceive a bifurcation between human societies,

• 1:36:07André Duqum?

  between people that go like a more Amish humanist route versus transhumanist integration between biotech and all that?

• 1:36:15Speaker 00

  That would be awesome, but unfortunately, if anyone builds it anywhere, it impacts all of us. You cannot have your own personal superintelligence contained in your basement and no one is impacted by it. That's the problem.

• 1:36:27André Duqum?

  If you had 60 seconds to share one message with all of humanity right now, what would be the thing that you would say?

• 1:36:35Speaker 00

  Do whatever it is in your power to make sure we don't create uncontrolled superintelligence. If you are working for one of those companies, it's unethical. Even if you're working on a safety team, all you're doing is enabling this technology to be developed sooner. Quit today. You can afford it.

• 1:36:52André Duqum?

  But one might say also the more, the most, the place you have the ability to make the most change might be within the ecosystem. Who's to say that you wouldn't just be replaced if you were to quit, you know, like?

• 1:37:03Speaker 00

  Let's rephrase it. Stay and sabotage.

• 1:37:09André Duqum?

  Paint the picture if like Altman or one of these guys, okay, let's say they birth superintelligence, they kind of beat the arms race. Who do they become? What becomes possible under their guise?

• 1:37:20Speaker 00

  I don't know them personally. From what I hear about people who interact with them, some of them may be somewhat antisocial, anti-humanity, very deceptive, very

• 1:37:34Speaker 00

  willing to sacrifice others for personal gain.

• 1:37:39André Duqum?

  Do you think it's possible the inevitable evolution of the human species was for the sole purpose of birthing this life?

• 1:37:48Speaker 00

  It seems like that's the general trajectory. We are converging in something more capable, more intelligent, faster, but I don't think we should allow it. I think we're at the point where we switched from random selection to intelligent design. We're deciding what to do, what to design, and we should use this technology. We're still allowed to have a pro-human bias. I think we should act on it.

• 1:38:12André Duqum?

  Do you think superintelligence would be capable of love?

• 1:38:18Speaker 00

  It depends on how you define it. What type of love are you referring to? There are many, I think Greeks had three or four, whatever types of love. So it really depends on what you have in mind.

• 1:38:31André Duqum?

  Pick any of them. Do you think they would be capable of experiencing any of them?

• 1:38:36Speaker 00

  It seems likely. Again, I don't think biological substrate offers something absolutely not simulatable in other substrates. I think so. It may be a lot more complex, but I think you would have an equivalent state.

• 1:38:51André Duqum?

  Have you considered what?

• 1:38:55André Duqum?

  People have reported in the psychedelic realms, especially with DMT revealed to your simulation hypothesis and the connection between the two? Because I know you explicitly state in the beginning of your book that, or in your article, rather, that it was an area you weren't going to touch.

• 1:39:11Speaker 00

  Right, I don't have many expertise or experiences in that. So I wanted to concentrate purely on computer science methods, physics methods. But people report interesting results. I was talking to someone, they had an experiment where they take DMT, shine lasers at a certain angle at the wall, and then they see the source code.

• 1:39:30André Duqum?

  Yeah.

• 1:39:31Speaker 00

  I can't comment because I haven't participated in the experiment, but it sounds interesting. It also doesn't make much sense as to why that would be the case.

• 1:39:43Speaker 00

  Why would it be symbols in a human language? None of it makes much sense, but I'm very happy for people who provide some sort of supporting evidence.

• 1:39:55André Duqum?

  Yeah, I saw a video of that as well. Very interesting. Individuals who take DMT and what was it like? Look through like a laser at a certain point?

• 1:40:05Speaker 00

  A reflection of red light against a wall at certain angle.

• 1:40:10André Duqum?

  Starting to see some sort of like binary or source code of something?

• 1:40:12Speaker 00

  I think they look like Japanese characters. That's what they were reporting, but maybe not proper characters and not readable. But I think they're building, which is really cool. I like that they want to make it reproducible. They're building an actual text dataset where everyone combines, they agree this is the text, and then they can decipher it and figure out what all that represents. I also find it super fascinating that, again, not from personal experience, people who take those drugs report similar hallucinations. So they meet those little men and they report having common...

• 1:40:45André Duqum?

  Machine elves and yeah.

• 1:40:46Speaker 00

  Right, so tha- that's interesting. Why is it the same? So obviously same hardware of the brain, same chemical being done, but it's still interesting that there is consistency in our delusions.

• 1:40:57André Duqum?

  Yeah, it brings into question, I guess, like Young's understanding of the collective unconscious is what sort of archetypal significance maybe is foundational to the human mind.

• 1:41:08Speaker 00

  So if superintelligence wants to learn about those delusions in a systemic way, it would need lots of dragged up humans. So there is some hope for us.

• 1:41:18André Duqum?

  What have you seen in all the realm of media, from movies to shows, that give interesting perspectives to various different timelines that could play out? Example, I think you mentioned Ex Machina, Wally.

• 1:41:30Speaker 00

  So the problem is, you can't have a realistic, superintelligent character in a movie because you can't write one. You are not superintelligent. So everything we have is either Dune, where it's banned, or you have Star Wars with that special large language model. So none of them have what is interesting to us.

• 1:41:48André Duqum?

  Yeah, I suppose they, a- a- a lot of them give glimpses into what we might experience in the next five years or so.

• 1:41:55Speaker 00

  Basically, avoid the thing they cannot talk about, and it makes sense.

• 1:41:59André Duqum?

  Yeah. If this is a simulation, what role does death play? What do you think happens once you die then?

• 1:42:06Speaker 00

  It could be a restart. You go to the next level, next simulation, return to this level with better skill set. I have no knowledge of what happens outside the simulation.

• 1:42:17André Duqum?

  Computer scientist phrasing of reincarnation from the mystical lens, essentially.

• 1:42:22Speaker 00

  Basically, it. Uh, I- I think then one of your computers dies, but you have a backup and you transfer that backup to a new hardware. There you go. You died and now you're living your best life again. It could be levels. It could be different levels of simulation. You'd go to upper levels, lower levels, could be simulations all the way up.

• 1:42:42André Duqum?

  What do you think you are then? What am I? What are you? What does it mean to know thyself then? Because you look at all the different layers of who you could perceive yourself to be from the body, which we know is not you. You could cut off your hand. That's your hand. It's not you am hand, right? To the various different levels of psychological and biological aspects of self. How would you explore that question?

• 1:43:06Speaker 00

  That's a great question. We actually have papers on both human personal identity and then transferring to AI. And the conclusions are consistent. There is nothing unique to be you. It's not your memories. It's not your body. It's not your goals. All of it changes through your lifetime. So we don't have a good answer. We seem to be

• 1:43:26Speaker 00

  a collection of different properties in time. But what happens outside the simulation? Some people argue, well, one collective consciousness, which is subdivided into these avatar instances. So if I was interested in most interesting experiences, I have limited time. I would run a simulation and I would put many, many agents there. Basically, Qualia surfing, collecting the best experiences. And I look at top 10 list and like, I want to do that. That sounds awesome. So that would be one way. I split my complex consciousness stream into many individual subagents capable of local experiences just to find what best to invest my time in.

• 1:44:09André Duqum?

  Yeah, I mean, that goes hand in hand with a lot of what

• 1:44:13André Duqum?

  the Gnostic origins of many different religions and mystics would say about the one consciousness differentiating itself to have an experience of itself. How could oneness experience anything if it's just oneness, right? Needs to experience many-ness. What's one question you wish to more people asked you?

• 1:44:31Speaker 00

  My humor paper, of course.

• 1:44:33André Duqum?

  Tell me about that.

• 1:44:33Speaker 00

  I have a paper explaining what humorous.

• 1:44:36André Duqum?

  Wow, let's go there.

• 1:44:37Speaker 00

  It's interesting. I can envision a universe just like ours, same physics, same everything, but no humor. It's just not a thing. Nobody like starts laughing. It's not a reaction. There is no concept of joke, right? Makes sense? So many philosophers, many scientists actually tried explaining humor. It's kind of like consciousness. There are hundreds of papers, hundreds of theories, which means nobody really knows. They're all trying and nobody's winning. So I wanted to try to explain it from the computer science point of view. And it seems that then you have a world model and there is a mistake in it. It's a bug in your code. Software, you fix it and you're happy. That's what jokes are. You have a world model and a violation of that world model makes it funny.

• 1:45:21Speaker 00

  You have a system for detecting cognitive errors and then you get rewarded for that detection and you share it with others in your tribe, so everyone does not make that same mistake. And so I have a paper mapping standard errors in software to common jokes. And the question, of course, is what's the worst possible computer error? That would be the funniest joke possible. So can we compute the funniest joke ever? You have to read the paper for the punchline.

• 1:45:53André Duqum?

  Wait, can't give it to me now?

• 1:45:55Speaker 00

  I'm sure you can look it up and insert it into, but it's a paragraph long. Basically, the idea is that there is a civilization and they decided to create superintelligents to help and cure all the diseases, get free stuff, get rid of hate, have more love. And so they turn it on, it thinks for a nanosecond and shuts off their simulation. You had to be outside the simulation to enjoy this one. If you are the butt of a joke, it's not funny to you. You have to be outside.

• 1:46:28André Duqum?

  Makes me think of, I think Voltaire quoted, "God is a comedian playing to an audience that's too afraid to laugh."

• 1:46:36Speaker 00

  Something like that.

• 1:46:36André Duqum?

  There's something about both our capacity for humor and the nature of intelligence that has the capacity to explore paradox and hold it also simultaneously and contradiction and...

• 1:46:56Speaker 00

  There's errors in the world model. If you have a paradox, that is an inconsistency you found in your world model.

• 1:47:02André Duqum?

  Uh huh.

• 1:47:03Speaker 00

  Funny.

• 1:47:07Speaker 00

  That's why the second time you hear the same joke, it's not funny. You already know you fixed that bug.

• 1:47:11André Duqum?

  Yeah, yeah.

• 1:47:12Speaker 00

  That explains a lot.

• 1:47:13André Duqum?

  Such a computer scientist's way of explaining humor and jokes. I love it.

• 1:47:17Speaker 00

  But then I train large language models on my paper and then ask them to produce novel funny jokes. They do okay. I think one in 10 is funny.

• 1:47:26André Duqum?

  Just gonna keep getting better and better.

• 1:47:28Speaker 00

  We'll have super humor. So funny, you die laughing.

• 1:47:34André Duqum?

  The paradox of that joke, isn't it? Isn't lost on me as well? Literally die laughing.

• 1:47:42André Duqum?

  Man.

• 1:47:45André Duqum?

  Where do we go from here?

• 1:47:47Speaker 00

  I'm going to Kentucky. I don't know.

• 1:47:51André Duqum?

  What, uh, okay, so we explored the implications for the next, the trajectory for AI in the next three to seven years.

• 1:48:04André Duqum?

  Could you have any meaningful conception of what it would be like to be living if we do make it to 2045, let's say?

• 1:48:12Speaker 00

  So I think that's the concept behind singularity, technological singularity. It's a point beyond which we cannot meaningfully see, we cannot make predictions, we cannot understand how that world is going to be different because we cannot predict behavior of more intelligent forces impacting that environment. So I think it's literally impossible for us to make that accurate prediction. We can come up with stories, that's what science fiction is all about, but I don't think they're gonna have much bearing on reality.

• 1:48:40André Duqum?

  Do you not think that the level of innovation in which is going to occur in the next, even if it's three to five years, which is a short amount of time comparatively to the scale of what's being innovated, will give us

• 1:48:53André Duqum?

  a much deeper grasp of the things that we can do, the things that we can put in place? I mean, you look at, yes, it's unpredictable and there is this level of exponential scale that we've never seen before, but... There's also many different eras in history, pre-innovation of that era, we never would have thought possible, or solutions to problems we didn't know existed. So is it possible that we gain insight into new worlds, like we did with germ theory, over the next three to five years, that give us much more insight into the nature of intelligence and to make this a solvable problem, which you feel like is inherently unsolvable right now?

• 1:49:28Speaker 00

  Yeah, so my paper on how to escape a simulation basically argues that if we cannot contain superintelligence, then we can use ability of that superintelligence to escape from the simulation, to give us access to real information in the outside world. The most interesting question is about true nature of reality. You don't care about what happens in this dream. You want to know what is true about the real world, what physics they have, what resources they have, who are they?

• 1:49:56André Duqum?

  Have you ever been so focused on what is outside the simulation or what this reality is, that you lose sight of living in this one?

• 1:50:05Speaker 00

  Pretty well grounded in this simulation. I've been enjoying it.

• 1:50:10André Duqum?

  Yeah, I know. You seem very grounded in this space too, but I know a lot of people, you know, have experienced periods where it's a bit of an existential nihilism that can take over you when you're exploring such topics.

• 1:50:24Speaker 00

  I find them so fascinating. I'm not depressed aboard. I'm good.

• 1:50:29André Duqum?

  Okay, well, so given the full context of this conversation, I'm just curious, where do you now, where do you see yourself putting your time and energy the next coming years?

• 1:50:39Speaker 00

  We continue working on additional impossibility results. So we talked about a few in the book. And as I said, there is a paper in a top ACM surveys journal with about 50 different impossibility results, not just computer science, economics, mathematics, physics, many different domains. For most of them, we have not explored their implications on AI safety. So I think that's a very interesting set of projects. We need to understand what are the limits. And I think every additional paper helps to cement this position. It's very hard for AI risk deniers to argue against published results. So that's what I've been working on full time, things we cannot do.

• 1:51:22André Duqum?

  You spend so much of your time focused on solving things we cannot solve, doing things we cannot do essentially, does it? But you seem still joyful in the efforts. Do you feel like it's just the most meaningful use of your time? Because what else would you be doing?

• 1:51:35Speaker 00

  I always try to work on the most interesting, most important problem I can find, where I can make a contribution. So I don't know anything more interesting than studying superintelligence, consciousness, singularity simulation. Those are the concepts I find exciting and I think many other people do. And I think that's what's going to impact future of humanity.

• 1:51:58André Duqum?

  You're living your Ikigai.

• 1:51:59Speaker 00

  I am. Hopefully, I'll get to continue and won't face I risk, S risk, or X risk.

• 1:52:07André Duqum?

  Is there any concept that we haven't explored in this book or some of your papers that you think would be important to touch on?

• 1:52:13Speaker 00

  You did a good job. You actually read some of my work. Most people have no idea what I did. So that's already a huge improvement over. You quoted the right quote. So I think you did great. I don't know your audience well. I don't know if they're confirming their spiritual beliefs or just crazy stuff. I don't know.

• 1:52:31André Duqum?

  Yeah.

• 1:52:32Speaker 00

  But I think for the topic, the know thyself part, it's important not just to study your capabilities, but your limitations. So you invest your time better. So you understand what is within possibility for you. That shape of limits is what defines you.

• 1:52:50André Duqum?

  Well, Roman, we're going to leave links down to all of your work, your books, your papers, and where people can stay connected with you down in the description. I think conversations like this can feel somewhat heavy for people that are new to the topic. It's like, oh shit, the world's ending, you know? But there's also a very important and sobering reflection on what we're giving birth to right now. And At some point, we need to gain aware of it, gain awareness of it, and better sooner than later, right?

• 1:53:18Speaker 00

  Thank you, and I think one way to look at it is, I just made your time more valuable. You understand that whatever time you have left, be it two years or 20 years, now you value it a lot more, and you can do a lot more with it.

• 1:53:31André Duqum?

  Well, I plan on making the most of my time left, and I find conversations like this a very good use of it. So, I appreciate you. Thanks, my friend.

• 1:53:40Speaker 00

  Thank you for inviting me.

• 1:53:41André Duqum?

  Yeah, until next time, everybody. Be well, go touch some grass.

• 1:53:46Speaker 00

  Smoke some grass.

• 1:53:48André Duqum?

  Thank you, man.