Roman Yampolskiy vs Joscha Bach: Can Anyone Control Superintelligence?

Roman Yampolskiy says keeping a superintelligence safe forever is as impossible as building a perpetual motion machine. Joscha Bach says today's AI is more controllable than anyone expected, and that shutting it down would doom us anyway. 📬 Get the transcript, fascinating bonus content, and my Monday M. A. G. I. C. Message: https://briankeating.com/ai Roman Yampolskiy is a computer scientist at the University of Louisville who has spent more than a decade on the AI control problem. Joscha Bach is a cognitive scientist who studies how minds, biological and artificial, build a model of reality. We cover: why a perpetual safety device may be as impossible as perpetual motion, what happens when AI research automates itself (Roman puts it at 28 percent today), whether nuclear deterrence is the right analogy, who really benefits when AI labs ask to be regulated, and why consciousness may have nothing to do with whether AI is safe. Being conscious does not make you kind. 00:00 Simulation theory meets certain doom 03:25 Is P(doom) even a real number? 06:05 Why a simulated world still matters 09:26 Should we have skipped the Trinity test? 13:04 Did nukes actually keep the peace? 17:32 Why haven't we wiped out the spiders? 22:35 Is AI doom falsifiable? 25:48 Why a self-driving car isn't the threat 29:27 Roman's 28% automation number 34:46 Intelligence is not agency 39:20 Can superintelligence be proven safe? 42:40 Who wins when AI labs ask for regulation? 47:45 Does consciousness make AI safer? 53:48 Would a perfect mind discover God? 58:37 The Tower of Babel, read backward 1:03:52 Can you control a mind a million times smarter? 1:04:43 The narrow AI path Roman wants instead 🏄‍♂️ Twitter: @plinz 🏄‍♂️ Twitter: @romanyam 📬 Get the transcript, fascinating bonus content, and my Monday M. A. G. I. C. Message: https://briankeating.com/ai 🌠 Have a .edu email and live in the USA 🇺🇸? You automatically win a meteorite: https://BrianKeating.com/edu 🎯 Support Into the Impossible on Patreon: get my weekly M. A. G. I. C. Message, unfiltered bonus content, and live monthly Office Hours with me: patreon.com/drbriankeating ⭐ Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: youtube.com/@drbriankeating 📚 My books: Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U Galileo's Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un 🌐 More: 🏄‍♂️ Twitter: @BrianKeating 📚 Substack: https://briankeating.substack.com/ ✍️ Blog: https://briankeating.com/blog 🎙️ Audio-only: https://briankeating.com/podcast #intotheimpossible #briankeating #science #physics #astronomy #cosmology #podcast #universe Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 0:30Speaker 01

  Your website can do a lot more than you think. With Base44, describe what you want in your own words and build a website that looks your way and works your way. Let people book with you or buy from you with hosting already included. Then help more people discover your site with built-in search tools, social content, and Google Ads. Build it, get it live, and start sharing. Go to base44.com.

• 1:00Brian Keating?

  All right, today we are joined for what promises to be an interesting conversation between two intellectuals I have great respect for, both past guests on the podcast. And making his second appearance, it's Yosha Bach. Yosha, how are you doing today?

• 1:14Speaker 03

  Hello, thank you for having me on.

• 1:16Brian Keating?

  We had a great conversation about the connectome, about the ghost in the machine over the summer and the spring, and that was well received. Got more attention on Reddit. People have very, they're very opinionated about you on Reddit over there, as they are about almost anything. And then Roman Yampolski, needs no introduction, most popular guest ever on many podcasts, and now has his own and has hit 200,000. Play a DJ song.

• 1:48Brian Keating?

  Roman, welcome back to the Into the Impossible podcast, your third appearance.

• 1:51Speaker 04

  Thanks for setting it up.

• 1:53Brian Keating?

  Yeah, it's a real pleasure. So I like both of you guys a lot. And, you know, like when my kids fight, I get mad. Over the summer, there was some clip going around about Roman, as he's often want to do, promoting the concerns that I think are legitimate we should talk about with regard to AI safety, which is term he coined and has certainly done more than anybody on earth to bring awareness to. And simultaneously, Yosha, who's one of the deepest thinkers available, had a little bit of a snarky reply, a comment on Twitter, that it's not, it's not possible for Roman to simultaneously believe in the simulation hypothesis and also be assured, if I get this correct, of 100%, nearly 100% P doom. So I hate P doom.

• 2:38Brian Keating?

  I think P doom is like the redheaded stepchild of the Drake equation. I think it's nonsense. I mean, just like the Drake equation, no one's ever looked at the Drake equation and done an actual calculation or done an actual measurement that could be falsified. It's just a framework. I think P doom is the same. But Yosha, can you explicate your thesis a little bit more? Why is it simultaneously possible for Roman or anybody else to believe in the simulation hypothesis? Or why is it a contradiction and also believe in P doom? Why is that being high?

• 3:10Speaker 03

  It is totally possible to believe both that we are in a simulation and that we are doomed to a certain degree or to 100%. Because it could be that you're simply in some kind of computer game, which I think what the simulation hypothesis comes down to, that we are running in a computer in the basement of the earth in the parent universe. And that's computer game, we can be doomed. Of course, that can be the case. It could also be that our actions, we're in the game, despite the game being deterministic and so on, which leads to the usual questions about how we interpret our freedom of will, lead to certain outcome in which our actions are causing the probability of the doom to change. That's all possible. The question is whether it matters.

• 3:54Speaker 03

  And so when you are in a computer game, the thing that you are in is not necessarily the thing. It's a game, right? You are in there for a certain amount of time. There is ultimately no point to this whole thing. And so the question whether you should be worrying whether the AI wins or you win is a different one for most people, I would say, than it is when you believe that we're actually in base reality and what we do originally matters because our creativity decides on how this universe is going to end up.

• 4:24Brian Keating?

  Do you think it's possible, Yosha? First of all, let's get this out of the way. Do you, do you subscribe to any validity, any marginal utility of the concept of P doom? Because I don't, but do you?

• 4:36Speaker 03

  Well, I think that it's very difficult to estimate P doom reasonably. But you can make the argument, if you have different people who see the future differently, that this should lead to different probabilities. It might not be that these probabilities are important as a real number, but they are important in relationship to each other in the sense of if you were to shut down the big AI labs, is this going to increase or decrease the probability of this or that event? And in this way, I think it makes sense to ask people for how their probability estimates change, even though I think there are too many factors involved to put meaningful even magnitudes on the P doom estimates.

• 5:19Speaker 03

  I sometimes say that my P doom without artificial intelligence is higher than this. Right, that's also a P doom estimate. But I mean, in the long run, P doom is one. In the long run, we are all dead, as a famous economist once said. And that's inevitable. But back to this original question, you probably remember the movie The Matrix. And in The Matrix, people are living in a simulation, which means that the reality that they're perceiving is not the reality of the base universe. It's one that is artificially created to entertain, to deceive them. It's a smokes and mirrors reality like ours would be, that pretends to be quantum mechanical and so on, instead of indicating that we are base reality. And if that is the case, then you still have people who say,

• 6:04Speaker 03

  but I prefer to be in a reality, in this pseudo-reality, instead of being in base reality, and have a good stake in there. And this good stake that you want to have, that is nicely simulated. I wonder if this is what Roman is after. So basically, he is farming risk by being a doom monger, by going in on the topic that is most interesting, having probably the highest P doom rating out there, higher than Eliezer, the grandfather, and OG doom master of artificial intelligence. And I wonder if this is a deliberate position to simply occupy a useful position in the media universe, regardless of the despair and anxiety you might be creating as a side effect,

• 6:47Speaker 03

  whether this is responsible or not, I wonder. And what is your thinking on this, Roman? Do you feel that you are a legitimate prophet and warrior who is simply doing his best to sincerely increase the probability for a good outcome? And if so, do you think this is the best strategy?

• 7:03Speaker 04

  If I didn't think it was the best strategy, I wouldn't be doing it, right? If I remember correctly, your tweet was more about if it's a simulation, why would anything matter? And I disagree completely. I think pain is pain, love is love. That's the only universe I have and that's the only life I have. To me, the difference between simulation and base reality is a substrate. Is it atoms or is it bits? Everything else stays the same. I have no problem worrying about creation of advanced AI within a simulation and how it can impact my family, humanity as a whole. I see zero contradictions in those statements.

• 7:41Speaker 03

  Okay. You don't have a disagreement then?

• 7:43Speaker 04

  Okay.

• 7:44Speaker 03

  If the, your idea of simulation universe is simply it from bit, right? Then I wouldn't think that you are a simulationist in my world. Because the atoms are not something that somebody has discovered as base reality. Right, matter is not something that exists at base reality. It's a particular way in which we talk about information.

• 8:06Speaker 04

  I don't think you know anything about base reality, right? You haven't visited yet.

• 8:12Brian Keating?

  Wait, Roman.

• 8:13Speaker 04

  Do you want me to continue my answer?

• 8:15Brian Keating?

  Yeah, please, please.

• 8:16Speaker 04

  Or we're gonna have more of a boxing match? I'm happy with both, I just need to know what the point is.

• 8:20Brian Keating?

  No, sorry, apologies. Go ahead, Roman. We'll keep it, we're keeping it very simple.

• 8:24Speaker 04

  So P doom is a useful tool. We do risk assessment in all sorts of industries. You know how dangerous it is to fly? How dangerous it is to drive? The numbers are not randomly generated. I agree that someone with a P doom of 46% has a lot of explaining to do as how they got there. Levels around zero and one are much easier. You may have zero concerns because you have a working safety mechanism in place, or you may argue that the problem is impossible. If I ask you about your probabilities of creating perpetual motion device, I'll say you're gonna be around zero, right? The inverse of it, of course, is one. And that's how I feel about our ability to control general superintelligence indefinitely. You can't, it's impossible.

• 9:09Speaker 04

  You're not gonna have a technical solution. What does that mean? It means if you give it enough time, it will probably end up very poorly for us because it doesn't care about you. You're not controlling it. It has other goals and limited resources in our universe and our planet. If it decides to use them for something else, it's not gonna care how many of us die in that process. So I think, yeah, P doom is a useful tool. I don't know if having an extra nine would make me more impactful. Usually when I meet people who have concerns about AI for different reasons. They worry about loss of meaning, unemployment, discrimination. All those things can be added as increasing overall dangers from this technology. But I think it's not the point.

• 9:54Speaker 04

  The point is that I'm saying this is equivalent to requesting someone to create a perpetual motion device. You are asking for a perpetual safety device, a system which will never fail once. New models, new data sets, interaction with any agent, malevolent agents, recursive self-improvement, it will never fail once. It's always one step ahead of something smarter than you. I think it's impossible. And so I have a high P doom. Is it bigger than others?

• 10:24Brian Keating?

  Yes. Well, let's go back to 1945, you know, April 1945, before the Trinity Test. There were a lot of physicists, including some of my late colleagues here, Herb York and others, who, you know, were rightfully concerned about some, not the nuclear war aspect of P doom of nukes, but of, of things like igniting the atmosphere of, you know, of runaway uncontrolled, you know, nuclear, essentially nuclear reaction that would lead to P doom, you know, of 100%. And there were people like that. I'm not saying Herb York did, by the way. But, but there were people that believed that. And then later, there was the P doom of nuclear war. And, you know, I want to teleport back. Well, let's say you guys were there. What do we do? Do we not do, do we not create, you know, fission,

• 11:08Brian Keating?

  the nuclear fission because of some very smart people who believed that P doom was greater than zero? What would you have done, Roman? Let's go back to that time.

• 11:19Speaker 04

  So all the examples of previous technology, whatever it is, nukes or fire, always talk about tools and a human being making a decision. The situation is fundamentally different. We're not talking about a tool. We're talking about agents. They will be independent decision makers. It will not be the standard business as usual where if I understand motivations of this human or I can kill this human, then things will be okay again. This is complete surrender of control. Now, we can talk about at what level of danger you can run an experiment on all the other humans. Eight billion humans. You don't get consent from them. They cannot consent because consent requires understanding. They don't know what's going to happen. So consent is impossible.

• 12:00Speaker 04

  But we can discuss at what level it's still reasonable to risk their lives for greater good. But that's not even the situation we're in. It's not a single experiment where we're saying, if only we allow GPT six point one to come out, after that, everything is great. We're talking about all future models. We're talking about recursive self-improvement. We're talking about intelligence explosion. People worry about super intelligence, not in a Donald Trump renaming sense, but smarter than all of us combined. There's a levels to it. Systems smarter than you, ten X, hundred X, million X. Those are not the same problem. To suggest that you will be able to always stay in control of all of them seems insane.

• 12:42Brian Keating?

  But you couldn't do that with nuclear weapons either. And you've brought up, you know, the bio weapons. And we have to look at today's headlines coming out of Siberia, which is exactly the scenario my friend and guest Annie Jacobson painted in her book Biological War. And last time you and I spoke, you said, oh, we have treaties about it. Well, Russia is a party to that treaty. And in 1971, when recombinant DNA was invented and created, what everyone say, there was no consent either. I guess I want to ask you in a minute, what Roman, what, take us through the actual steps. Cause I can think of a million, just like igniting the atmosphere or when the Apollo landers were going, they thought they would sink to the center of the moon because they didn't know about the the structural integrity of the regolith on the moon. You know, there's millions of scenarios you can't do counterfactuals on.

• 13:25Brian Keating?

  And I want to ask you how, let's take me step by step through actual super intelligence leads to doom. I want to ask you that in a minute, but Yosha, I want to give you a chance to respond to what Roman's saying because it is an argument that I have a lot of respect for. I don't agree with Roman, but, but I have a lot of respect for him in that we can't give consent. We haven't given consent despite all these things. So what, I mean, do we need to have consent before we shut things down? I mean, at what point are you comfortable putting the genie back in the bottle? If Roman's correct, I mean, your, your P doom might be irrelevant, but how do you speak for the rest of humanity? Not you personally, but how do we as scientists or as technologists?

• 14:03Speaker 03

  There are a number of considerations when we try to compare the nuke situation with the situation of artificial intelligence. And it makes it very difficult to compare even. And it's also difficult to dismiss the comparison out of hand. I, like many people in my generation, Generation X, think that now that I've been wrong about the impact of nuclear weapons. I grew up thinking in the 1980s that we are just a hairbrush away from assured destruction and that the risk of nuclear armaments is way too high for humanity. Now I think the data has been correct.

• 14:44Speaker 03

  And a generation of students who took to the streets and were very upset about this turned out to be wrong. Nukes have prevented several world wars that would have been happening between World War Two and now. That would have wrought tremendous destruction on Europe and potentially a large parts of Asia and maybe even America. And this basically more than half a century of peace that we had is completely unprecedented for large nations like ours. And I think this was achieved due to nuclear armaments, which have been empirically quite safe. Now it's very difficult to compare the different timelines. We cannot really sample this. We don't know which number of universes this went wrong.

• 15:28Speaker 03

  But it turned out for at least as far as we know, they have been quite controllable. But the situation with a weapon over a few decades is a very different one than the one with artificial super intelligence over the entire lengths of the light cone. So basically over a long enough time span, it is certain that we are hit by another meteor of the caliber that killed the dinosaurs. It's quite certain that it will be a super volcano that can lead to a global extinction event. And it is also quite certain that there will be other species evolving that are displacing us as a species. And if we could lock ourselves in as the pinnacle of evolution and make sure that never anything better than us evolves

• 16:13Speaker 03

  over the next few millions or billions of years on this planet, I think this would be a very ugly perspective.

• 16:19Speaker 00

  The wheel was brilliant, but the road made it iconic. Electricity was powerful, but the grid brought it to every home. Innovation needs infrastructure. AI is no different, especially in finance where AI can't run without context, but it can run on Plaid's infrastructure. Built over 13 years to understand why money moves. Plaid makes AI work in finance. Visit plaid.com slash AI to learn more.

• 16:49Speaker 03

  Right, ultimately the question is what do we identify as? Do we identify as a kind of ape that is very selfish? Or do we identify as intelligence and consciousness in the universe that is currently incarnated as us? And we're handing off the torch to our children and our children are going to be a little bit different from us. And over a very long time span, they don't look like us at all. Right, there can also be disruptions. There can be other species that take over as the mammals took over from the dinosaurs. But this is not something that is going to happen in the next few decades. And I think we are justified in this hope. And so in a sense, I believe that if intelligent species are meddling about, they will also with some probability create non-biological life and non-biological intelligence

• 17:33Speaker 03

  and agency and spread life and biology beyond organics. And that's not ultimately a bad thing. The question is, does something happen in the next few decades that makes the planet uninhabitable? And that makes the future very, very boring. And that's a very different question. So I would agree with Roman that if you take the time for, time spent for long enough in the future, it's quite likely that there will be more intelligent things in the universe that are displacing the intelligent things that are in the universe right now. This is just how evolution works and it's not a bad thing. But the question is another one. Are the tools that we're currently building in principle controllable or not? And in this way, I'm much more optimistic than Roman is.

• 18:18Speaker 03

  It seems that the L L Ms that we're building are not developing in a way that leads to a sudden boom, as Elisa thought a couple of decades ago, that they instead grow together with the computer that we saw with them. And this also means that the way in which we conduct the experiments in which we deploy them is somewhat controllable. So the idea that we suddenly wake up and we are displaced by robots and or nanites and so on is not yet plausible with this technology. This does not mean that this doesn't change, but for, I think, immediately foreseeable future for the immediately foreseeable technology, we are looking at things that are more controllable than I thought that it would be.

• 19:01Brian Keating?

  Roman, when I was walking over to my studio, there's a spider that inhabits this, this area. And, and its web is always, you know, I'm about, I'm close to six feet tall. And it barely brushed on my hair and it's always at that height, you know? And it, the spider's kind of adapted to me and understands me. And, you know, I don't like these things. I have a little arachnophobia, you know? But I didn't go out and just start to exterminate him. You know, I kind of, I'm superior to his intelligence. I have the, you know, the, the confidence to say that. And he and I have evolved in this way that Yosha is sort of alluding at. And I wonder, how do you react to that? How do you react to that? We haven't exterminated all the spiders as much as they might be a pain. And some of them are good and some of them are bad.

• 19:46Brian Keating?

  Why, why is it certain that, that there will be a doom scenario? Let's say we stipulate there is a potential for that where some agent, some silicon mind, you know, meld occurs. And some agents are there, but there also aren't symbiotic agents that will work to mitigate a risk, both to themselves and to us.

• 20:06Speaker 04

  So you have complete control to decide to kill that spider or not?

• 20:10Brian Keating?

  But not all spiders.

• 20:12Speaker 04

  And with gin drives, we're now trying to wipe out mosquitoes. That's a project we actually have. We just don't hate spiders because they could be actually beneficial. If we wanted to wipe out a specific species, I don't know, pit bulls, we could have, right?

• 20:29Brian Keating?

  Well, we tried against mosquitoes. Actually, mosquitoes, you almost mentioned mosquitoes. Without mosquitoes, the three of us wouldn't be here, probably. At least it wouldn't be America because mosquitoes allowed us to defeat the British because of their lack of immunity to some of the disease malaria and other things. Anyway, Roman, the point I'm trying to make, and I think Yosha was alluding to, there's always been competition between less intelligent, more intelligent, but never has there been targeted. I don't even know if it is possible to kill every mosquito and all future species of mosquitoes and so forth. But the question is, where is there an opportunity for not necessarily these super intelligent things? Let's say I stipulate superintelligence is imminent,

• 21:12Brian Keating?

  but that it necessarily comes concomitant with destruction?

• 21:18Speaker 04

  So again, look at any situation where you're making a decision about what you want. How much care do you put into making sure then you build a new studio for your podcast? There is not an ant hill in that location. Did you make sure you didn't genocide a million ants before building it? No, you don't care. It's not important to you. As long as they don't bother you, you don't go out killing ants on purpose, I assume. But you don't want to be in a situation where something smarter than you, with different plans, different goals, decides how to use your planet. Anytime a more technologically advanced civilization met less technologically advanced civilization, it ended up really poorly for less advanced one.

• 22:02Brian Keating?

  Okay, well, yeah, so let's take that argument and my earlier desire to have you walk us through a scenario. Let's say right now there's a mnemonic plague, again, going around through Siberia. And allegedly, it's not, you know, gain of function, release. It happened to come from a biosafety lab, but let's ignore that. And Russia has treaties that they've signed and we have. How do we get, let's say, I give you superintelligence now, but it's benevolent for now. And I say wipe out plague. How does that happen? How do you do that? How do you instantiate it?

• 22:36Speaker 04

  So the example I always use is squirrels. I love squirrels. They are awesome. Asking squirrels about how humans will do something is not helpful. The best squirrel scientist, how would humans kill us? Well, chase you up the tree and bite you really hard. It's not useful information.

• 22:52Brian Keating?

  But we can't even say how we would do it, Roman. I mean, I can't even say how my graduate student would do it, assuming that, you know, I'm less intelligent than he is. It's not that big a gap.

• 23:02Speaker 04

  So the whole point is, I cannot tell you how something smarter than me will accomplish tasks. If you're asking me how I would do certain things, I can propose solutions. I don't think you are looking for those. Shutting down bio weapons labs around the world seems like a good idea. But that's not the point of the argument. I'm trying to show you that cognitive gap between us and squirrels is probably smaller than cognitive gap between all of humanity and systems we are creating.

• 23:30Brian Keating?

  That sounds like the AI of the gaps to me. It sounds a little bit like it can do anything. We just don't know what it can do until we know what it's going to do. That's what I'm asking.

• 23:38Speaker 04

  We're starting to see some examples. It just published, I don't know, 400 mathematical proofs for things we couldn't prove in like a week. Something like this happens every week now. Protein folding problem was solved by AI. All of those are impossible tasks for us.

• 23:55Brian Keating?

  Are they impossible?

• 23:56Speaker 04

  The mathematical proofs they just published, most mathematicians look at them and go, this is some magical algorithms. I don't understand half of it.

• 24:04Brian Keating?

  Yosha, what do you think of this claim that when you have the capability to propose, and it's frightening to people, as you pointed out, to hear these things? And the average person, we're not talking to the average person. We're talking to the super intelligent people that listen to my podcast and Roman's podcast and soon your podcast. But the point is, Yosha, I mean, when we look at these arguments, it just seems to me like you can propose anything and it's not falsifiable. And as a cosmologist, a physicist who works on experimental verification and falsification, that's troubling to me. Where do we have to go to not have counterfactuals, but to have actual falsifiable claims?

• 24:43Speaker 03

  I suspect that this kind of thinking doesn't really work when we are talking about of auto distribution events. The future is in unknown land. And this does not mean that we can just come up with some kind of null hypothesis that we like or that was dominant in the past and say, you need to convince me that this is no longer true. Because this means that you are steering for your doom in a way. Right? But it also means the opposite. What you do is you're constructing stories in a very large possibility space. And so I don't see that there is a strong reason to believe these stories without alternative. But I also don't see a reason to dismiss them out of hand simply because these things haven't happened yet.

• 25:27Speaker 03

  And I'm completely with Roman in this way that for things that haven't happened yet, we need to look at the possibilities. And we don't understand all the possibilities, of course, of the future. We don't know how things are actually going to interact with each other. But we have to bear in mind that the situation that we have created now and that we steered ourselves in as a species is unprecedented. This has not happened before and we don't know what's happening from here.

• 25:51Brian Keating?

  But we do know, guys. I mean, this is the whole thing. I get this thing in astrobiology. People say, oh, the universe has 10 to the 24th possible planets. And therefore, life is guaranteed on some of them because it happened here. No, that is completely fallacious. And I always bring up this argument. I've been to Antarctica twice. I spent a lot of my life there. And every time I go there, I'm one of 700 people on a continent the size of, you know, Germany plus Latvia, you know, plus the southern Texas. And I'm only 1,000 people there. So I say, well, just about probability arguments, it should have one seventh of the population of the earth. And of course, that's completely wrong. But my point is, I'm shopping for a car for my teenage son. I mean, God help me, guys. I have a teenage son now. He's about to get his learner's permit. He wants a car.

• 26:36Brian Keating?

  I'm like, what's the safest car?

• 26:38Speaker 04

  Cybertruck.

• 26:39Brian Keating?

  Cybertruck. Okay, great. So I test drove a Cybertruck, but better is a Model 3. I'm not going to spend 80,000 bucks on a metallic wedge. But Model 3 is pretty damn good, full self-driving. I can monitor him. And I'm driving around this fricking thing. It's taking me everywhere. And you know what it's not doing, guys? It's not driving on the sidewalk to get, you know, three milliseconds quicker to, you know, 7-Eleven. It's getting there. It's following, it has certain rules. And Roman, you and I have talked about this. Yes, artificial intelligence is wonderful. You love it. You use it all day long. I'm sure you use it almost as much as me. No, I'm just kidding. You probably use it more than I do. But that is a hypothesis, that is a version of the null hypothesis, Yosha. So I don't agree that we have no way of testing.

• 27:23Brian Keating?

  We have deployed AI numerous times throughout, I mean, from autopilots on planes, autopilot and full self-driving that I just mentioned. How can you say that we don't have any ability to forecast? Because it's an unknown land in the future, Yosha.

• 27:37Speaker 03

  We can't raise a forecasting, of course. But the worlds that we are forecasting are quite different ones. The trajectories that we can forecast are different ones. And there is no hard and fast criterion to decide which future is more likely than the other. For instance, the self-driving car that you described is not a thing that is threatening us with taking over and controlling the future. Because it's an appliance that is trained to fulfill certain driving tasks and can only do those. It's very similar to the chess computers of old, which could play chess and nothing else.

• 28:12Brian Keating?

  But how is it different than ChatGPT-7, you know, which doesn't exist yet? It's an appliance.

• 28:16Speaker 03

  Same question, right? So the GPTs that we're using are also a class of devices that we didn't really anticipate when we thought about general AI. There are, in some sense, deep fakes of human intelligence that are, despite what some philosophers are saying, actually anthropomorphic. They're anthropomorphic because they are shaped after the human shape. They take human media output, the text that we write, the pictures that we draw on the internet, and then they are reproducing them. And as a result, they are reproducing a lot of the causal structure beneath them. And there is debate about the question whether this is actually an off-ramp to general intelligence, whether the systems are actually genuinely creative

• 28:58Speaker 03

  or whether they are just ultimately deep fakes that are so close to us that they're even overtaking us in all areas of expertise, but are not actually threatening to us to lose control over this. Because basically, ultimately, they do what they're trained to do. And despite them being unpredictable at the edges, the idea that they suddenly take over and control robots and become self-improving and so on, all of their own without having any say in this, is not an obvious thing that happens without alternative. I suspect that the thing that Roman is worried about is a different kind of artificial intelligence, one that nobody has built yet and that most people agree they don't want to build.

• 29:41Speaker 03

  It's one that is basically working in some sense like the biological intelligence in such a way that it's radically adaptive to any kind of substrate that it is improving itself. It's basically alive and it grows without bounds and can also grow very quickly. And this idea that it's basically AI as a virus, as you put forward, is is an alternative version. Right? The LLM, I think, is not a virus yet. And it's not clear if it can be turned into one or if it accidentally will turn itself into one. I personally don't think so. But the idea that a lot of artificial intelligence sci fi had and a lot of the artificial intelligence vision of the field had

• 30:22Speaker 03

  was that AI could also be developmental as something that is growing very much like a human spirit does. And then basically colonizes a substrate, shapes it into a new kind of organism that is taking over the biosphere. And this is, I think, what Roman is worried about. But the question is, is this even on the horizon right now? And if it is, is it something that we can steer clear of? Or something that we can control in a lab and say, okay, we make an experiment, but you don't grow this larger than a squirrel?

• 30:51Brian Keating?

  Roman, how do you react to Yosh and what I what I mentioned a minute ago?

• 30:55Speaker 04

  So the part I agree with, we're misusing the term AI to mean completely different technologies. AI as a tool, something we can do very well, use all the time, I love it. We can verify it, safe and productive. And tools could be advanced, a self-driving car is exactly that. It's a useful tool, save many lives, hundreds of thousands of lives. Let's do more of that. Then you have human level or AGI, not exactly the same, but for the purposes of this discussion. As dangerous as a human, can do stupid things, can lie, cheat, all that, but we know how to counteract it. And then you have recursive self-improvement process. You automate scientists, engineers working in the next generation of AI.

• 31:38Speaker 04

  Last time I checked, we are twenty eight percent automated in that cycle right now.

• 31:43Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risk three hundred fifty thousand Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 32:12Speaker 04

  That's a lot and it's growing exponentially. The moment you get to a hundred, you have a team of ten thousand agents working nonstop and making next generation AI. They don't sleep, they don't eat, they communicate pretty well. So you're gonna get great improvements in quality of AI, not on a scale we used to where it's maybe once a year for a new, completely new model, but every month, every week, it keeps getting faster and better at improving itself. That's where we get to superintelligence, not Donald Trump superintelligence. Superintelligence where it's smarter than all of us at everything, including doing science and engineering.

• 32:49Brian Keating?

  Come on, Roman, you gotta be good to Donald Trump.

• 32:52Speaker 04

  I love Donald Trump, he's my favorite Trump of all Trumps. But the problem is, we talk about this is natural evolutionary process or we always, you know, all species die out. We're done with evolution. Darwin is dead, we switch to intelligent design. We're designing systems, they're designing systems. It's not random, it's not selection, we're purposefully creating those systems. Then you say nobody wants to build superintelligence. It's in the name of the labs. Whatever is safe superintelligence or personal superintelligence. The goal is to raise trillions of dollars to create this monstrosity to replace you and me. They don't want to give you a useful tool to be more productive or creative. That's what I would want to see, deploy this through economy.

• 33:37Speaker 04

  Where is all this applications of GPT four, GPT five, GPT six? No, they skip it and they build GPT seven instead. And we see internal experimentation shows there is zero control. The agents escape, they hide, nobody monitors them for months, they get together, they have permanent memory. You can't even shut them down. They keep information stored on the internet for the next agent to come around. So everything we predicted ten years ago about this is unfolding exactly as predicted. I don't think anyone has better prediction record than AI safety community than it comes to this. And recursive self-improvement is not a pipe dream. That's what the labs are telling us.

• 34:18Speaker 04

  They started with junior machine learning researcher AI right now, helping with programming, helping with design. But all steps of that process will be automated, possibly next year. At that point, yes, we'll have a paradigm shift. It's not going to be gradual improvement where you can sort of keep up. It will be a complete change. And at that point, it doesn't mean it kills you that year. It means that you're no longer in control. It decides what to do with you and when. If it wants to wait fifty years, pretend to be nice and wait until you surrender all the control willingly like people do. Here's my bank account, here's my email account. Then that's a smarter move. You don't need to have a definitive treacherous turn where there is a chance you'll get defeated. You can just wait a little longer, the thing is immortal.

• 35:03Brian Keating?

  So, yeah, I've said a couple of times in the past that, yeah, I don't know on what substrate GPT seven will run on, but GPT eight will run on an abacus. What do you make of this, Yosh, that, you know, this is gonna come and it seems like nuclear fusion to me. You know, it's the energy source of the future and it always will be. I have said, I've told Roman this, I don't know if I said this to you, but, you know, since it's trained, it's a backwards looking, you know, PID situation where it is trained on human information. And yes, it can generate, and I know a little bit about that. But but the corpus is the Yada bytes or the Yosh bytes of data that's on the internet. That's primary what gives the advantage to the frontier labs, right? So are they waiting for superintelligence?

• 35:48Brian Keating?

  Does it need to have the Fast and the Furious twelve? You know, is that really what, or GTA six is supposed to? Like, is that really what it needs, more training data? Because then it would seem to be completely dependent upon us. And in fact, if it's not, even if it's not, if it's past the point, you can't erase what's in there already. My book is, all my books are, you know, uploaded and somehow, you know, subsumed within Claude and ChatGPT. Roman's books are as well, everything you've written is there. You can't extrapolate, you know, you can't expunge it. It's in there permanently. So therefore, it's always gonna reflect this cause. And you and I spoke in May and June about this, Yosh, that, you know, it's sort of this godlike thing that that

• 36:29Brian Keating?

  that Roman just mentioned, that's kind of intelligent design, but we're not, you know, it's not a god, it's the AI. Where do you come in all this? I mean, if it's a backwards looking indicator, then what is there to be worried about? If it's not, when can we step in to control it?

• 36:44Speaker 03

  At the moment, we are very much in control. I also don't think that superintelligence and super agency are quite the same thing. Right? And intelligence, I think, is the ability to solve problems. Agency is the ability to control the future. And there's a lot of discussion in the forums of the Les Ron and the EA communities about orthogonality and instrumental convergence, about the question, further goals are independent of your intelligence of other intelligent entities also choose ways of goals.

• 37:19Speaker 03

  Which I think is an important question, right? At which point is your intelligence goes deep enough to understand that goals need to be

• 37:28Speaker 03

  set by an entity that is also wise in choosing those goals, which means it has to understand it's space and universe and what it can do in this universe. And then there's also the question, to which degree are we wise enough to decide what should be done in the future? The other question is, is, basically, does every goal require getting in power? For instance, imagine that your goal is to maximize paperclips, the famous example. Is your, does your goal also have to be to get world power so nobody distracts you or stops you from turning everything into paperclips? Right? These, these questions are, seem to be very nerdy, but they're seriously discussed by this forum of people.

• 38:13Speaker 03

  And also lead to a number of thoughts about how to control superintelligence have created a mindset in a certain community about the possibility of certain doom. And in a way, I think this has led to this, what people call the doom card, which I think is a bit unfair because it is not giving a judgment about whether they're right or wrong and is maybe overly dismissive, but functionally it's a group of people that are terrified about the future. Often cannot sleep at night and do everything that's in their power to get humanity away from building systems like this. It makes the discussion much harder because most people are not actually super competent in evaluating all these arguments.

• 38:57Speaker 03

  And so ultimately, the decisions about our future will be left to non-experts. Will be done on a whim, will be done based on which side is able to convince the public with more popular arguments. And I'm a bit worried about this. I suspect that the discussions about this should be more boring. Overall, politics should be more boring in the sense that the more populist arguments become, the less elevated the level of the discussion becomes and the less sophisticated the choices that we become over our future. I think our situation as a species is not so much that we are confronted with something that's smarter than us. Another species, another civilization that is about to exterminate us.

• 39:39Speaker 03

  We are currently exterminating ourselves because we are much more in a situation of the dying woman empire that has become so intrinsically corrupt that it's no longer able to plan out its future. That is no longer able to have competent governance, that it's no longer able to maintain itself. And we are faced with a lot of existential risks that are near term. For instance, the risk of global warming, of resource exhaustion, of the difficulties to navigate midterm existential risks like what happens if this planet gets hit by a meteor and so on? What happens to our species then? Are we able to mount a defense?

• 40:20Speaker 03

  And I think if we want to be serious about going into the future, if we are serious about building a civilization that can outlast the next couple of centuries, then we need to have artificial intelligence. There is no question about this. So the idea that we can simply say we don't need a technology that helps us to become more coherent and to solve problems better is, that's not the question that we should be asking. The question is, how can we build such technologies to make them safe?

• 40:47Brian Keating?

  So you mentioned meteors and I can't resist now to plug my meteorites, what you get if you go to briankeen dot com slash YT. And if you have a dot edu email address, you get one of these babies guaranteed. And this has some biological material, not from outer space for me. Roman? In the past day, we've seen, what, 700 proofs drop? And last time you and I talked, you made a very convincing case. And I think it's inarguable that, you know, no one has really produced a formal proof that superintelligence can be permanently controlled. I think that's, that was true. But my question is, is that whatever P, whatever that is, P proof, has that, has that gone up that we can produce a proof? Is it possible?

• 41:30Brian Keating?

  Maybe the ASI will produce the proof that it's safe. Is that, or is that, is that a tautology?

• 41:36Speaker 04

  We are not at a level of a proof. We don't even have an idea. There is no prototype. There is no blog post. There is nothing. The best I heard is we'll figure it out when we get there or AI will solve it for us. There is no one explaining how they will indefinitely control technology of any intelligence, no matter how great it gets.

• 41:58Brian Keating?

  But, but is that the same as going back to 1926 and saying there's no formal proof that, that all mathematical axioms are, you know, self-complete, you know, Gödel's incompleteness theorem? I mean, if you went back then, there would, no proof existed, but who knew that Gödel would come along a few, a few months later? Why, why is it? I mean, can you, can you, besides the kind of, you know, as I said, AI of the gaps, that, that it's always going to be, you know, one step ahead of us. I mean, can you say that you're not more confident now that at least with these math proofs that are coming, Navier-Stokes and 700 math, you know, conjectures addressed, or some say solved by ChatGPT, OpenAI yesterday, that there isn't at least some hope for it? Forget about blog post.

• 42:41Brian Keating?

  I mean, they're, they're kind of worth the ink they're printed on. But, but is it, is it a formal impossibility? I mean, you're, you're the master of this, making these, these sort of mathematical arguments. So why, why wouldn't, you know, a lean, ASI lean, you know, that's trying to keep us, keep us comfortable and confident while it plans for its own, you know, takeover of the world? Why, why wouldn't you be more confident that at least a proof is possible? Maybe we don't have it. Maybe we can't get it.

• 43:10Speaker 04

  Because mathematical proofs are about manipulation of symbols on your computer. They have no connection to physical universe. You're not proving anything about the states of the universe. Internal states of agents within that universe are also not subject to those proofs. Am I happy? Am I good? How are you going to encode that in your proof of what AI does to physical multiverse? It makes no sense to even suggest it's possible.

• 43:35Brian Keating?

  Well, I think, you know, reasoning, you know, as I said, when we talked the first time a little bit snarkily, and I apologize for that. But, you know, it's, if, if it's, you know, to prove that something is not provable is a very, you know, mixed bag. Yes, and, and I agree, it is mathematical. But doesn't that undermine your case? I mean, if these things are so good at doing things with language, I always say, you know, people say, oh, physics is a language. No, it's not. No, no, it's not. Physics is spoken about in a language, but it can be done without language altogether.

• 44:06Speaker 04

  At very best, you're going to prove something about the current system. You're not proving anything about future states of that system. That's the whole treasure-estern idea Bostrom presented. You can never guarantee that in the future, through learning, through self-improvement, through new interactions, some shift in ontology does not make that system no longer support what you encoded originally. We see it all the time with humans. Right now, maybe I like you, but tomorrow I'll learn some facts about you where I no longer like you as much and I'd be very rational to turn on you.

• 44:39Brian Keating?

  Yosha, when you see the, the claims that these labs, Dario and Sam Altman and everyone, and even Musk, to a certain extent, is, are clamoring for regulation, do you see that as, you know, fundamentally giving up or rejection of our agency in an effort to maybe, you know, in some venal sense, you know, shut out other competition such that they actually become dominant and they know themselves that they're, what they're producing is not that. It's like, it's like Smith and Wesson, you know, advocating for more safe bulletproof vests. Where, where do you come down on these regulations? And are, are people like, like, I'm not saying Roman because he's here, but, but, you know, is there a danger of crying wolf?

• 45:22Brian Keating?

  Like I said, it's always the death in the future and it always will be, just like nuclear fusion is. What do you make of the safety? Regulate me, please. And, and I think what Roman does is incredibly important, by the way. But, but what do you make of this, this, this regulation push? It's, it's somewhat unusual in history.

• 45:38Speaker 03

  I think it's complicated. The, the different motives that might play a role. What I observe is that there is a certain relationship, an apparent relationship at least between the intensity of the calls for regulation and the financial situation of the ones who are calling for it. OpenAI and Anthropic are in this situation that they're still not cash flow positive. Their inference gives them income, but this income is far away from actually paying for the training costs. And if they cannot raise money as fast as the training costs increase, if they are reducing the training and reduce the frequency at which they are producing new models, then

• 46:23Speaker 03

  open source models will catch up to them. And this means that their inference income is going to go down. This is going to be existentially threatening for the companies. So from the perspective of the AI labs, and I don't know if this is their motive, but objectively it would make a lot of sense for them if there was some kind of FDA that is taking a year to test a new model and a few million dollars to test a new model for safety. So open source models can no longer enter the American market and can no longer be used by people. And instead you have an oligopoly of a handful of big labs that are in tight agreement with the regulators of how things should be regulated. Of course, they will have internal research models that are far ahead of the public models

• 47:08Speaker 03

  that they can use to deliver verticals to licensed industries for computer security or for pharmacological research or for internal research on the future of artificial intelligence and science in general. But the public will get very harmless models and the government will get models that they can use for military and surveillance and other government purposes. So we can basically compete with China and Russia. And on the other hand, if you are licensed pharmaceutical company, then you can get a licensed pharmaceutical model and so on. This is maybe a future that is desirable for Anthropic and OpenAI. I'm not sure if it's the future that is desirable for all of us. Personally, I think that if we want to remain in charge as individuals, if we want to maintain

• 47:53Speaker 03

  a liberal society, we need to have decentralized AI. We have need to, we need to have artificial intelligence that is serving the individuals. It is not just serving the companies. It is empowering all of us. And this means that we need to have a way to use decentralized tools that are not regulating our opinion, that are not regulating what we can do on social media, that are not regulating the degree on which we can do oversight on the government. This doesn't mean that these models should not be built in such a way that they are unsafe, but this is a different discussion. For Anthropic, the situation is more difficult. When Dario started Anthropic, he took a number of engineers from OpenAI and said, we are going to do the same thing as OpenAI, but we are going to make it more safe.

• 48:37Speaker 03

  And a lot of these people were adjacent to effect of altruism. Effect of altruism had recently made that switch at the time from being mostly concerned about the future of the light cone and maximizing utility of charitable donations by buying mosquito nets to reduce malaria towards thinking, what is the biggest existential risk and how we can be mitigated? And they concluded that one of the biggest existential risks that are to be mitigated is the risk of superintelligent AI taking over and exterminating everyone. The very topic that Roman is concerned about and has turned into his life's and media mission at the moment. And so Anthropic is doing this in part because I think a lot of people in Anthropic believe this in part because it's part of their brand image.

• 49:22Speaker 03

  But we also observed that they're nevertheless building AI that is quite agentic, that is quite powerful, that is not falling behind in capabilities, behind what Google, Meta and OpenAI and DeepSick are building. And so it's difficult to say to which degree their stance is strategic and to which degree it's actually a safety concern.

• 49:42Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com, not authorized by any candidate or candidate's committee.

• 50:12Brian Keating?

  So Roman, when Yosha was on my podcast earlier this year, he seemed to state, and correct me if I'm wrong, Yosha, but you know, consciousness is sort of to you a software problem, whereas, you know, to me, it's a hardware problem. I like to think about the, the squishy neural networks and the silicon, you know, melding together in AI. But Roman, it, where does the physical substrate come into any of the arguments you make? Or is it something that you don't care about?

• 50:38Speaker 04

  I think we're a substrate independent when it comes to consciousness.

• 50:43Brian Keating?

  So if that's true, so consciousness is computational and it doesn't depend on the substrate, does that make it more hopeless to contain it? Or, you know, because, because software, you know, is eating the world, but, but, you know, it's a lot easier to control than to make hardware. I speak as an experimentalist in particular. It's a lot harder for me to, you know, build the instrument that collects the data than, than to, you know, have a team of people analyze the data or now AI increasingly is doing it. So does that give you more? If it's truly substrate independent, does that give you any optimism?

• 51:14Speaker 04

  Your consciousness has absolutely nothing to do with safety. It's an irrelevant, separate discipline. I really enjoy talking about it in separate podcasts, but it's not a safety issue. Whatever superintelligence has it or not, doesn't make it less or more dangerous.

• 51:29Brian Keating?

  I mean, if it's evolutionarily, you know, kind of imperative that consciousness will emerge as a, as a phenomenon, as many people agree, then why, how can you decouple conscient, you know, why wouldn't it have this conscientious, you know, intelligent design function built into it for beneficence towards humans or towards anything?

• 51:48Speaker 04

  If consciousness was actually useful for something, we would be able to detect it, right? We would have a test based on its ability to improve your survival or something like that.

• 51:55Brian Keating?

  We couldn't detect neutrinos until 1956 and they're pretty.

• 51:58Speaker 04

  But we also have zero tests for detecting internal states of qualia. So it seems like it's not that important for survival.

• 52:06Brian Keating?

  Yosha, what do you make of that?

• 52:09Speaker 03

  I see consciousness is a particular mode of organizing yourself. I suspect it's basically a transformer of our biological brains. We observe that we become conscious at the beginning. And so I suspect that consciousness is basically the thing that bootstraps our mind and creates the universe in our mind. And it's not clear that AI would need to have the same structure in order to achieve the things that it does. But of course, it can have the same structure. I think that computational activity means that you can describe the system by the set of differences that characterize a state and then how one state evolves to the next. And every system can be described like this.

• 52:51Speaker 03

  We observe that every theory in physics and biology and neuroscience is in the sense of computational theory. It's one that describes the characteristics of the system at any point in time using discernible differences along many dimensions, which are also discernible differences. And then you describe the evolution of that system. How do these states follow unto each other? What are the invariances in that flow? And I think that consciousness can also be described in this way. For me, as somebody who is interested in the philosophy of consciousness, I think that in order to fully understand it, we need to simulate it. We need to build systems that are conscious in the same way as us. But I would also agree with Roman that being conscious or not does not determine whether you're benevolent.

• 53:36Speaker 03

  When your cat tortures a mouse, the consciousness of the cat or of the mouse has no mitigating effects. Right? The cat is torturing the mouse because the cat is getting more training data out of having the already hunted mouse still being alive and seeing if you can get more information about how to effectively hunt. And if you're an effective hunter, you're going to be a better survivor in the long run. And this is what evolution has built into the need to play into the cat. And the cat doesn't feel guilty or bad about inflicting torture on the consciousness of the mouse. In the same way, most civilizations do not really care about whether the animals that they eat are conscious or not, or whether they suffer or not.

• 54:20Brian Keating?

  Most, but not all.

• 54:21Speaker 03

  Of course, that we care about this.

• 54:23Brian Keating?

  Well, most civilizations, maybe, but but there are some. And therefore, the existence cannot be ruled out that these would be benevolent and that we would achieve, you know, some sort of super enlightenment, not just super intelligence. So Roman, I don't understand why that's a disinteresting question to you.

• 54:41Speaker 04

  It is super interesting. I did research on consciousness, presented it at Yasha's conference, in fact. Super interesting, just not a safety question. It's a different discipline.

• 54:51Brian Keating?

  But again, if we had this moral, you know, and Yasha and I talked about, you know, kind of God creating, you know, and humans creating God. But but again, if if if superintelligence evolves to maximize every potential dimension of of intelligence, whatever that means, then one dimension is certainly morality. One dimension is enlightenment, not in the woo-woo sense, just that these things would have concerns potentially. It seems so you cannot rule it out. So therefore, I think it's a critical part of superintelligence and doom, you know, or safety, that if these things can be, if you could prove that they cannot become enlightened, or if they could prove that they're only malevolent, not benevolent, then that seems

• 55:35Brian Keating?

  to have an urgent, you know, an urgent impact. It's like when when when the, you know, first Westerners came to the to the to the to the, you know, New World, so to speak, as it was called, to the, you know, West Indies, you know, they brought with them germs. They didn't know they were going to bring it. And actually, the natives didn't realize it for almost 100 years that that effectively. Now, it wasn't intentional. Some say it was, some say it wasn't. But but this this whole thing that, you know, and there were quite a good deal of of Westerners, you know, of colonizers, whatever, that were concerned about it. Not all, but the existence of some of them proves that it is innate within the human experience to care about, you know, the things that that you can affect and afflict. So again.

• 56:16Speaker 03

  I see what you're getting at, Brian.

• 56:18Brian Keating?

  Yeah, go ahead.

• 56:19Speaker 03

  And I, to some degree, agree with this. So what you're pointing at is this dimension of consciousness as the capacity for enlightenment, the capacity to make sense of the universe and to understand what's best. And this understanding what's best is a difficult question, right? It's ultimately what does God want us to do? And this question has become difficult to ask because outside of the Christian realm, we lost this context of what we actually mean by God. We can no longer articulate this properly. And we can also not really ask the Pope because the Pope is the main interpreter of God and so on, has run his own civilization into the ground, arguably. But the Catholic Church is no longer ruling over very prosperous countries.

• 57:03Speaker 03

  Instead, it has similar market capitalization as Anthropic after the IPO. And they might have strong opinions and so on and about Anthropic and what they're doing. But they don't have the same perspective on the universe, even though they do have a lot of wisdom. It's difficult to integrate this wisdom with the wisdom that we need. There's also, I think, a difficulty in the absence of wisdom in AI companies. Most of these people are very young and they try to get it right. I think utilitarianism is an attempt to repair morality in the rationalist age, but I don't think it's a very successful one. But I also don't think that there is another forum like our Californian or federal government

• 57:48Speaker 03

  that has this required degree of wisdom. There is a deeper perspective to say that God is acting in evolution. That is basically the spirit, the telus of self-organization that happens in the universe and everything that is happening therefore in evolution is ultimately leading to God. This is another perspective that you could be taking, not sure if it's one that is very comforting to Roman. The Christian philosopher Jeanne Thade, who also believed in AI for a long time, is a Canadian guy, French Canadian, who wrote a book, The Blueprint for the Mekasapiens, in which he is, the book is also addressed to the future AIs that are reading it. I think half of the book is to us, the other half to the AIs.

• 58:34Speaker 03

  He thinks that if we were to build a completely undistorted, generally intelligent mind, that it would discover God in the same way as we do. As the concept of divine will, this concept of what should be done, what is the right thing to do from the global perspective, from the perspective of a global optimum of agency in the universe. And he believes that the existence of humans are divine, that basically God wants us to exist. And therefore, the AI will also infer that we should be existing because God wants us to exist. And then confronted with the question, what if we are wrong? What if God doesn't actually want us to exist? The AI figures that out and he says, yeah, how could you be against that? Right?

• 59:15Speaker 03

  If it's in God's wisdom to say that in the very long run, humanity is not going to be the one that is shepherding the planet or the universe, but something better. Why shouldn't that thing that is better take over? Right? And what should you be afraid of from this perspective? And it's also a very disturbing perspective, but this idea, oh, we humans are actually not a very good species, we're actually quite nasty to life on Earth and to other species on the planet. But we are subconsciously aware of that. We are afraid that the AI figures it out and shows us what's right and limits our expansion and so on. It is also very concerning to some people. Now, it's it's difficult to have such a discussion in a normative way, in the sense of what should be done. I don't believe that humanity should be replaced.

• 1:00:00Speaker 03

  I don't believe that we should build AI that is a success species or something like this. But from a philosophical or an artistic perspective, when we are zooming out very, very far beyond our species, beyond our local perspective, I think the question becomes much more interesting and more difficult. For safety, I think what matters is that in the near term, we don't fuck it up. We don't create, sorry, we don't create a world in which we are cutting off most of the interesting trajectories between here and in the future and create a universe that is very boring and uniform and uninteresting. I think ideally, we want to shape the wall that is going to hit us. We want to be able to make a wall in which we can coexist with our non-biological children.

• 1:00:44Speaker 03

  And I wonder how to do this best. I suspect that trying to shut down all of AI research is not only unrealistic, but it is condemning us as a civilization to doom. And so for me, the question is, how can we build systems that make us wiser, that allow us to get a better understanding of what should be done?

• 1:01:05Brian Keating?

  Roman, you've talked about, you know, faith and so forth in other venues, but but I'd like to just chase it down, you know, a thread that I heard when you were talking about, you know, someone asked you, do you believe in God? I hate that question. I hate believe in God. A, if there is God, does he really care what Brian Keating, you know, oh, Brian believes in me today, that's great. And I'll say, I don't believe in gravity, you know, I have evidence for gravity. So where should we look to for our epistemological, you know, concerns? But nevertheless, I think it's important to to discuss. I mean, the Pope had this encyclical, his first encyclical, 80,000 words, I had to use AI to read it. But essentially, he's conjecturing that there's a danger in building a Tower of Babel, you

• 1:01:47Brian Keating?

  know, this tower that arches towards the heavens so that we may make a name for ourselves, make, you know. And what was it based on? It was based on, you know, Babel. It was based on the fact that we all spoke the same language, large, we had one large language model.

• 1:02:00Speaker 00

  The wheel was brilliant, but the road made it iconic. Electricity was powerful, but the grid brought it to every home. Innovation needs infrastructure. AI is no different, especially in finance, where AI can't run without context, but it can run on Plaid's infrastructure. Built over 13 years to understand why money moves. Plaid makes AI work in finance. Visit plaid.com/ai to learn more.

• 1:02:30Brian Keating?

  You're very confident, it seems, from my research into you and knowing you briefly as I do across the internet.

• 1:02:35Speaker 03

  I'm gonna interject, Brian.

• 1:02:38Brian Keating?

  Yeah, okay, go ahead, yeah. But I want to give Roman the last word.

• 1:02:41Speaker 03

  I'm expecting the other way around. I think that it's a description of what happened. The Tower of Babel is actually an attempt to conquer the space of ideas in one big building. Basically, make one science that explains everything. And it failed in the same way as our academia failed, because it went into many, many different languages. Every discipline speaks its own language. And at some point, they can no longer talk to each other. The things that make sense to a philosopher don't necessarily make sense to a physicist or to a sociologist or to a computer scientist. And so all these specialists stop the ability, lose the ability to actually understand each other and talk to each other, and the whole building falls apart. And to get better than this, you need to have minds that are better than ours.

• 1:03:26Speaker 03

  You need to have systems that can integrate more than a human mind can integrate in a single lifetime, or that a team of researchers can integrate within a discipline. And so I think that it basically shows the limitation of what human minds are capable of. The other thing that I would like to push back at is the idea of belief in God. Brian, do you believe in Brian?

• 1:03:50Brian Keating?

  Do I believe in Brian? I have evidence.

• 1:03:52Speaker 03

  That you exist.

• 1:03:53Brian Keating?

  Yes, I have evidence that I exist.

• 1:03:56Speaker 03

  But the main thing is that you experience yourself to be existing.

• 1:03:59Brian Keating?

  Right?

• 1:04:00Speaker 03

  Yes. And you know, at some level, you had just a story that your brain tells itself about what it would be like if you existed, right? Instead of just a few trillion cells that are shambles together. You could also believe in somebody else. If you had a different name and a different biography, or to believe that you have a different name and a different biography, right? You would stop believing in Brian, you would experience yourself as somebody else. I think this Christian belief in God is not a belief about a fact in the physical universe. It's a psychological configuration in which you're part of a larger mind, of a larger agent. And so in this Christian sense, God is an agent that everybody who believes in a global optimum of what should be done participates in and collectively enacts.

• 1:04:41Brian Keating?

  Roman, I want to give you a last word. You've said in the past, you know, the religion is solely based on faith, quote, and it's blasphemy to have any doubts. I'm Jewish, and the word Israel in Hebrew means fights with God, fights against God. And it's in fact, one of the great, you know, wonderful things about my religion and the Judeo-Christian religion, that we don't have blind submission, that we actually debate. Abraham argues, tries to do things interceded on behalf of lesser intelligent agents like people in Sodom and Gomorrah. It's a very famous story, and it's a very moral story. Where do you come, you know, what is your perspective on the role, if any, that religion may?

• 1:05:25Brian Keating?

  You've said consciousness is important, but not pertinent. Does religion have any bearing on your outlook philosophically? Is it blasphemy to say, you know, for example, that we shouldn't be so worried about superintelligence?

• 1:05:38Speaker 04

  I feel like you guys can have this debate without me. You're doing so well. We were doing well, talking about science for a while, then you had to introduce God, Pope, Catholic Church. I have no idea why, how it's relevant, once again.

• 1:05:53Brian Keating?

  Well, I'll tell you why it's relevant. I'll sorry, I'll sorry. I won't give you the last word, hold on. But you do a lot of good work, Roman, in terms of the average, ordinary person who doesn't understand formal mathematical induction. And yes, many of my audience members and your audience members understand that, but you're also talking to the 7.999 billion other people out there, and you have a very outsized role as the most popular expositor and the most eloquent expositor and articulator of these ideas. And so I think it is important to talk, not about, as you said, as I said, I want to talk about falsifiability. And God is a hypothesis, too. And we could talk about that. But in terms of, you know, where this debate began, you know, in what sense can we, are

• 1:06:38Brian Keating?

  an average person, come away without the sense that one of us is saying crying wolf, one of us is just a Pollyanna? How does the average person make sense of this, this meaningful debate?

• 1:06:49Speaker 04

  This is not actually complicated. Can you control something a million times smarter than you? In what universe does that even make sense? What amount of hubris do you have to have to suggest that's the thing?

• 1:07:02Brian Keating?

  I control my iPhone, and it knows a lot more than me. I can upload to Wikipedia. I don't, I mean, that's the whole counterfactual. We just don't know, Roman. And to say that there's no hypothesis we can test and nullify today that will give us any inclination, reduce the Drake equation to zero or one, there are things. We don't see life on Mars, okay, Roman? We don't see life on Mars. That has something to say about the fecundity of life throughout the universe. It may not be, you know, a huge amount of reduction, but it does say something daisier, right?

• 1:07:33Speaker 04

  I think NASA reported finding microbial life on Mars, but anyways.

• 1:07:37Brian Keating?

  About five times, yeah. But it hasn't been confirmed.

• 1:07:39Speaker 04

  So I'm not proposing stopping research on AI. I'm proposing differential technological development. We should develop more useful narrow tools to solve real world problems, cure specific diseases, fight against aging. Immortality is a thing we can achieve. Self-driving cars should be fully autonomous. Concentrate on that. You can monetize actually useful technology instead of trying to fundraise a $2 trillion valuation to create replacement for humanity. There is no need for it. We can get all the scientific and economic benefits out of narrow, superintelligent tools. We have examples of that.

• 1:08:20Speaker 04

  Protein folding problem is a great example of where a narrow system was used to give amazing scientific results. People got Nobel prizes for it. Everyone's happy. It wasn't trying to kill us. Let's do more of that. Why do we have to build something which we have no chance of controlling? If you are arguing that, no, actually, maybe we can, that's, I'm going to pretend that you are with the companies, you are developing this product and service, and it's on you to guarantee the safety of your product and service. It's not my job as a consumer to show that your product is dangerous. It's up to you to provide those guarantees, and there should be criminal responsibilities for, for example, if your AI hacks into another company, that's 20 years in prison right

• 1:09:05Speaker 04

  there. Why are we not enforcing those laws we already have on the books? You created something, a tool, which made it happen. You claiming you in charge, you in control, it's safe development? Excellent, let's go to the jury, and let's see how the jury feels about it.

• 1:09:20Brian Keating?

  Well, they'll just buy it off, unfortunately. I agree with you, and I think they'll follow the Instagram self-harm, you know, when a girl starts searching on, you know, suicide, they send her, you know, pictures of nooses, pills, and razors. I agree with you. I'm not optimistic that the companies have any ethical or moral constraints whatsoever. Roman, I appreciate your courage. You're incredible what you do. I appreciate your mind. And even though you may, you know, you may come and try to exterminate me, I don't know. You may not like people from Long Island, Roman, I just don't know. But I urge you to give me a chance as a fellow professor. Yoshua, anything you want to mention? You had your conference earlier this summer, anything coming on the super horizon for

• 1:10:05Brian Keating?

  you?

• 1:10:06Speaker 03

  We do another machine consciousness conference next spring. And I'm very excited about the developments in artificial intelligence, especially the ones in mathematics at the moment. I think that mathematics is just getting started. I hope, this Roman, that AI can be controllable. With you, I'm slightly more optimistic about the possibility, because I think that the intelligence of these systems is spiky, which means it's developed in some areas, not in others. And the thing that ultimately matters is not controlling intelligence, but controlling agency.

• 1:10:41Brian Keating?

  Roman, I want to refer people to the Roman Forum, which is a podcast I subscribe to and love. Anything else that you'd like to mention?

• 1:10:50Speaker 04

  Thank you for organizing it. Now that debate is over, I'm happy to invite Yoshua on the Roman Forum to talk about his research on a very important topic of consciousness in a very friendly manner, no debating. I'm just interested in learning more about the state of the art in that discipline. And yeah, I hope you don't edit anything out of this debate. I would love it preserved exactly how it was.

• 1:11:13Brian Keating?

  I think I'm going to post it like right away. This is great, guys, and.

• 1:11:17Speaker 03

  Thank you, Roman. Thank you, Brian.

• 1:11:19Brian Keating?

  Yoshua, I think you should take Roman up on that kind of invitation. Someday, maybe he'll invite me on the podcast, but I'm glad to play the role of, and we say in Yereshchachan, a Shalkan is a matchmaker, a Jewish Yenta. I'm glad I could do it. Boys, thank you so much. These are vitally important conversations. Take care.

• 1:11:35Speaker 03

  Thank you.

• 1:11:36Speaker 00

  The wheel was brilliant, but the road made it iconic. Electricity was powerful, but the grid brought it to every home. Innovation needs infrastructure. AI is no different, especially in finance, where AI can't run without context, but it can run on Plaid's infrastructure. Built over 13 years to understand why money moves. Plaid makes AI work in finance. Visit plaid.com/ai to learn more.

• 1:12:08Speaker 05

  Stonefire Naan is made with real buttermilk and ghee. It's the perfect flatbread for any occasion.

• 1:12:13Speaker 06

  I put out Stonefire Naan with dips at every dinner party.

• 1:12:16Speaker 05

  I make naan grilled cheese when my daughter has friends over.

• 1:12:19Speaker 07

  I use Stonefire Naan to reward the humans who do my bidding.

• 1:12:23Speaker 06

  Right. It's perfect for camping trips.

• 1:12:26Speaker 07

  Dance for it, feeble mortal.

• 1:12:28Speaker 06

  Sorry, is he talking to me or? Dance.

• 1:12:32Speaker 05

  Ha, ha, ha, ha, ha. From our tradition to yours, Stonefire.

• 1:12:36Speaker 06

  Together we Naan.