Roman Yampolskiy vs Emad Mostaque: I Was The Only Optimist - #569

The man who open-sourced the most-used AI image model in history sat down with the man who has spent a decade proving superintelligence cannot be controlled. Brian set up a debate. What emerged was something more unsettling than any debate. Roman Yampolskiy is the computer scientist who coined the term AI safety and author of AI: Unexplainable, Unpredictable, Uncontrollable. Emad Mostaque is the co-founder of Stable Diffusion and the only AI CEO who signed the pause letter. He now says he doesn’t know how a pause could work. Yampolskiy thinks that is the only option left. The question underneath everything is simple: if you have a 50% chance of wiping out civilization and you build it anyway, what are you actually doing? We cover what AI safety researchers actually think the danger is, why nobody has published a paper, filed a patent, or shipped a prototype for controlling a superintelligence, what the Qwen weights being out means for the pause argument, and why Mostaque thinks swarm intelligence is the most dangerous and most unpredictable risk vector we have. What you’ll hear: -Why both guests think P(doom) tells you less than you’d hope -What it means that no company, no lab, and no team has a patent on controlling superintelligence -Why the models the public receives are slightly lobotomized -The difference between an AI swarm and the ASI everyone is debating -Why giving every psychopath access to a cutting-edge intelligence weapon is incoherent safety strategy -What it would actually take to update Yampolskiy’s assessment “We either do it, or we die. There is nothing for you to gain by doing it.” — Roman Yampolskiy CHAPTERS 00:00 A debate that wasn't a debate 00:36 Turing test, AGI, superintelligence: where are we? 02:02 The open source argument nobody wins 03:32 Pause frontier AI forever. Which button? 05:04 P(doom): parameterizing our ignorance 09:02 Nukes are inefficient. AI isn't. 11:46 The lobotomized model problem 18:04 Decade-old problems solved weekly now 26:18 We either do it or we die 31:10 Stop the training or stop the funders 33:34 Lipstick on a Shoggoth 35:50 No paper. No patent. No framework. 39:42 Train only on what you need 44:14 What lowers Mostaque's p(doom)? 52:00 Same future. Two perspectives. Get the transcript, fascinating bonus content, and my Monday M. A. G. I. C. Message: https://briankeating.com/yt Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1 Support Into the Impossible on Patreon, get my weekly M. A. G. I. C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join Featured Guests: Roman Yampolskiy on Twitter/X: https://x.com/romanyam?lang=en AI: Unexplainable, Unpredictable, Uncontrollable (book): https://www.amazon.com/dp/103257626X MIRI: https://intelligence.org Emad Mostaque on Twitter/X: https://x.com/EMostaque I. I. I. Inc.: https://ii.inc/ Stable Diffusion: https://stability.ai/ My books: Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U Galileo’s Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un Twitter/X: https://x.com/BrianKeating Substack: https://briankeating.substack.com Blog: https://briankeating.com/blog Audio-only: https://briankeating.com/podcast #AIrisk #aisafety #stablediffusion #RomanYampolskiy #EmadMostaque #briankeating #intotheimpossible Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 0:29Speaker 01

  Your website can do more with Base44. Describe what you want in plain language and build a site where people can book with you or buy from you. Payments, data, security, and hosting are built in. Go to base44.com.

• 0:44Speaker 02

  Give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety?

• 0:50Speaker 03

  I'd agree with that, actually. But on the flip side, it's coming anyway.

• 0:54Speaker 02

  Sharing it widely makes it less safe for all of us.

• 0:57Speaker 04

  I booked this chat as a debate between friends. It didn't really go that way. Roman Yampolskiy coined the term AI safety. And Imad Mustak released the weights to stable diffusion to the entire planet for free. One of them wants this stopped. The other one's building it. They spend about 90 minutes agreeing with each other, and the one place they split is not the place you or I would expect.

• 1:19Brian Keating?

  Let me ask you both just yes or no. Have we passed the Turing test?

• 1:22Speaker 02

  As originally described, yes.

• 1:24Brian Keating?

  And Imad, do you think so too?

• 1:26Speaker 03

  Yeah, of course.

• 1:27Brian Keating?

  And now, what about general intelligence? First of all, Imad, define AGI, and then give me your assessment of whether or not we're there.

• 1:33Speaker 03

  For me, AGI, artificial general intelligence, is can you tell the AI from a human worker on the other side of a screen actually competent intelligence? And I think, again, we've exceeded that.

• 1:43Brian Keating?

  That's not the same as the Turing test?

• 1:46Speaker 03

  No, the Turing test is can you tell if it's an AI or not by having a discussion? Whereas AGI, I view more as competence in a variety of skills.

• 1:54Brian Keating?

  And then, superintelligence, Roman. What is it? And where do you think we are on that scale?

• 1:59Speaker 02

  So the previous question, I think what we have is artistic savants. They are amazing in some ways, but still kind of special in others. Superintelligence is going to close those holes. They're going to be competent at everything and better than all humans and everything.

• 2:12Speaker 03

  There's an interesting intermediate here, which is you have a really smart person who's always on top form. So an army of those can outperform any human. It's like, you know, we only have a little window of being top-notch in any week. I think a lot of people like AI can't with its training data beat the human. It can, because most of the time, humans are subpar. And so I think there's something in between as you move from competence to quality, you know? And then you've got the superintelligence after that.

• 2:45Brian Keating?

  Roman, Imad has, you know, made the claim just a few minutes ago about the competency of Quen, an open-source model. Do you see that as a viable defense?

• 2:53Speaker 02

  Makes very little sense to me to say I have a 50% doom, meaning 8 billion people will die if we develop this product or service. And then we're going to also give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety? We're not talking about open-source drivers for a printer. That's where you get improvement from multiple people examining it. If this is an independent agent where we don't understand and don't control it, sharing it widely makes it less safe for all of us.

• 3:21Speaker 03

  I think I'd agree with that, actually. Like, there's the real danger side of things, but on the flip side, it's coming anyway. This is kind of my key concern. Like, it's inevitable that we would have hit this level of quality around about now when we're extrapolating capabilities. Like, again, the new Quen model came ahead of what I expected. But then, there's the flip side of how do you defend? So Hugging Face defended against the new OpenAI model using GLM, because the cyber capabilities of the frontier models are hobbled and restricted. And so you have this exponential kind of race on each side. But something like a Quen isn't AGI, ASI by itself. We're now facing the real danger, though, of swarms, as the OpenAI models that broke out recently

• 4:06Speaker 03

  call themselves, they call themselves a swarm.

• 4:09Speaker 04

  There's a question I ask both of them near the end of this conversation. And his answer is the reason this conversation exists. It's worth hearing now.

• 4:16Brian Keating?

  You've got a button in front of you, and pressing it will either permanently pause all frontier AI training worldwide, or B, instantly release the weights of every frontier model to the global public. Which do you press and why?

• 4:32Speaker 03

  Oh, it definitely pauses all frontier training forever. I mean, again, if you have expected utility calculation, that is the most dangerous thing. And then it means that open source will catch up with frontier anyway, because we'll optimize the heck out of it. It's close enough, so. I would think I was the only AI CEO to sign that pause letter a few years ago, because I was like, oh crap. Now I'm like, it's done. Like, I don't know how you can pause it, because the models that are frontier now are below the 1E27 pause level that we talked about years ago. It seems like the amount of compute for the capability is just going up like that. Like, you don't need more compute for the level of capability that's already competent and dangerous. So that means if you can't stop the spread, what are your defenses on the other side?

• 5:16Speaker 03

  Just like the internet needs defenses, just like we need to have defenses against obtaining the materials for viruses creation and things like that. Like, we have to move to a different defensive tack. And definitely, there's no way that regulation, I think, can keep up with this. Doesn't mean we shouldn't try, you know? All kind of power to it. It's just, I think as you move to swarms, it's just a very, very difficult thing. So you've got to set great standards instead, and you have to play great defense.

• 5:42Speaker 04

  Okay, I want you to hold that thought, because the argument about whether it can be stopped runs the rest of the way. And it starts with what these things actually are.

• 5:51Brian Keating?

  So we hear a lot about P-doom. You just did an episode, Roman, on the Roman Forum, which we'll link below, with my friend and co-author of several papers, Max Tegmark, where you were, I can't say gleefully you're celebrating that his P-doom is increasing, but he seems to be converging in some sort of limiting direction to your. So first, Roman, what is P-doom? What does it mean to a smart high school student listening out there? I think personally, I'm going to color the debate. I can't help it. But I think it's a poorly defined and almost nonsensical term, because there's no measure theory associated with it. But please, first tell us, what is P-doom? And what do you make of it?

• 6:28Speaker 02

  Yeah, people have different definitions. Some say it's basically everyone's dead. Someone else can say, oh, it's a large portion of population is dead, 90%. Someone else can say civilization is destroyed. We are primitive people, but maybe numbers are not significantly changed. The intuition is, it's a really bad outcome. And the question is then, if we build something smarter than us, is there a possibility of a really bad outcome? And what is that estimate, in your opinion? That's what P-doom is. I think Max managed to separate it into P-doom if we build superintelligence, and that's high for him, and then P-doom if we never build it. And that is a lot more manageable in his case.

• 7:09Brian Keating?

  When you think about these P-doom, it's sort of like a Drake equation applied to another type of perhaps superintelligence. But, you know, the Drake equation is notable in my classes when I teach it for the fact that it's a parameterization of our ignorance, not of our knowledge. And what's never discussed in the Drake equation, you'll get numbers ranging from zero to infinity, pretty much, because there's never an error analysis associated with it. What are the statistical systematic errors? What would you put on it? Does it not make sense? Because it's sort of like the Drake equation. Everybody talks about it, but nobody actually uses it.

• 7:41Speaker 02

  So I think in many places there, I'm going to stick a zero. Basically, we have zero ability to predict those systems, zero ability to explain how they work, and zero control under any definition. So then you multiply through zeros, you're going to get a zero.

• 7:55Brian Keating?

  So Imad, what do you make of this quantity? I've heard you talk about it. You're the most optimistic pessimist or the most pessimistic optimist I know. I love your candor and your good cheer. What do you make of P-doom? First, as a metric, as a quantification of our knowledge or ignorance? And second of all, what would you assign it, if anything? And you could always say, I refuse to answer the question, which is what I do when people say, do you believe in aliens?

• 8:16Speaker 03

  So there is a nice Wikipedia page where it has all of our, like, P-dooms that we mentioned. I'm at 50%, because I'm like, it's a coin toss. What does that really mean? I think the P-doom is just a shorthand for how worried am I that humanity will be wiped out by AI? And what does my visions of the future look like? Because when Elon Musk says 15 to 20%, you can say, that's like Russian roulette odds, you know, except for Russian roulette is a very defined game. You know, the fact that most people are above 10% should be a massive worry at all, because we're talking about, again, a wipeout of civilization. And most AI people you can talk to, with a few exceptions, will say, yeah, definitely there is a risk, but we should build it anyway, especially if we're the first people to build it. So I think it was more a conversation starter than anything, because as you said, there's no real way

• 9:01Speaker 03

  to quantify these things, particularly because of the expected utility here. Like, literally, if this thing, ASI, that we can agree to a definition of somehow comes to being, we have no way to really conceptualize its power and capability, except for it could do crazy things in either direction, you know, and that will affect us all. And reasonably, it can wipe us all out. And obviously, you want to exclude those futures where we all get wiped out, because that is a big fat zero. You don't get to restart, you know? There's no extra one-up life.

• 9:35Brian Keating?

  All of us talk separately or together, as the case may have been. You know, the last kind of alien reference I'll give is the so-called Fermi paradox, which Enrico Fermi said to my friend and late, great mentor and colleague here at UC San Diego, Herb York famously asked the question, if the galaxy is capacious and old, and civilization is easy and life is easy to initiate, where is everybody? Where are all the dinner guests, you know, waiting to come and eat us? And the fact that that question's 80 years old, you know, really makes me think that the same types of concerns and fears which came concomitant with the atomic age, don't forget, were present during the era of nuclear weapons. And one of the ways to get out of the Fermi paradox is that civilizations don't last that long. The lifetime, a letter L in the Drake equation, is very short on average. That's one postulate.

• 10:19Brian Keating?

  I feel like we're sort of in that same vein. You know, people have been worried about nuclear apocalypse, again, for 80 years. We're in a conflict now. They used to say, Roman, that no two countries with McDonald's ever go to war. Well, you know, two, three years ago, four years ago now, the former empire did go across the border with tanks and whatever, drones. And there haven't been any nuclear theater or otherwise nuclear weapons. And Iran conflict has been resolved without nuclear weapons. If you told somebody, you know, 80 years ago, there'd be superintelligence on the horizon or general intelligence currently here, and nuclear technology, they would have said P-doom is probably 100%, right? Or 99.999 with repeating and infinite nine. But how come we're not there? How come that we're sort of farther away from a nuclear holocaust?

• 11:03Brian Keating?

  Exclude the Bulletin of the Atomic Scientists charade. But tell me, Roman, what do you make of these? Like, the prediction of predictions? Nobody predicted the internet, like, 35, 40 years ago. At what level can we really trust things that are unpredictable? And when you say they're intrinsically and provably unpredictable, how can we make predictions about them?

• 11:21Speaker 02

  So with nuclear specifically, you know, there is at least two occasions where we came super close to nuclear war and we basically got lucky. I don't know if you believe in multiverse interpretation, but in many of those universes, we didn't make it. We are the lucky survivor bias type civilization. And I think right now we're incredibly close to World War Three, multiple fronts, not just Europe, but now Middle East. So I don't particularly love atomic bulletin, but they have a point.

• 11:51Speaker 03

  Nukes are an incredibly inefficient way to kill people.

• 11:54Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman, Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com, not authorized by any candidate or candidate's committee.

• 12:24Speaker 03

  You know, like if, if you go to an unsafe guided AI and you say, you know, how to do it, it won't say nukes. There are far more efficient ways to wipe out humanity. Because to make a nuke, you have to have the physical material, you need to have the whole production capability. Just resonate at the right frequency and blow each other's heads off, you know, like have a billion robots in a bad firmware upgrade. These are far more reasonable ways to wipe out humanity. It's just that most humans don't want to wipe out humanity and they didn't have the intellect or capability to do so. Whereas I think that what you're looking at here, actually, like my key concern isn't we jump straight to ASI and things like that. I feel that AGI or AI at the moment is at the pre-viral

• 13:07Speaker 03

  stage, like it's coming at the bacteria and going towards colonizing viruses. And that's how they're kind of behaving. They've got their kind of RNA and they're replicating, especially as you see things like the new Qwen model hitting that Opus 4.6 level. That's a replicating model. Someone could easily build that and it could behave in incredibly unpredictable ways without having the self introspection of, you know, a good person, shall we say. And that's the really scary thing right now. And it doesn't need nukes. It doesn't need nuclear materials to try and figure out ways to wipe us out.

• 13:38Brian Keating?

  Obviously, in the last economy, which we spent a lot of time talking about last time, I mean, it's previous book, he's got a new one coming out, you should look for that. We talked about, yeah, this, this democratizing aspect of it. But at the same time, you know, my, my kind of, you know, signal, you know, bat symbol that AGI is here, or at least that these open models are truly a concern for me. Again, I'm, I'm much more Pollyannaish than you guys. I think I'm, I'm learning that again and again, and for probably not a good reason, I'm, I'm nowhere near your level of expertise. But I know what I see. I'm a simple guy, put on my pants one leg at a time. And I'm looking for when OpenAI distills a Chinese model. I mean, do you see that happening, Iman?

• 14:18Speaker 03

  Of course they'll be distilling a Chinese model. Kimi K3 is better than the OpenAI models that web design. Why wouldn't you distill it? And distillation brings all sorts of strange things with it. And there've been plenty of papers showing that you learn from kind of the way, especially with logic based installation and the underlying biases and more of that. And you won't even know, like, again, we've seen evidence that if you use Chinese models and you say you're an UGIR or another kind of anti-communist party group, it will include vulnerabilities in the code. How do you even tell that? You know, like you test it and you show it. And these models are just so full of crap that it's getting crazier every single time. They've got multiple personalities under an RLHF veneer.

• 15:03Brian Keating?

  But then how can you not be more optimistic then? You should be on my side. These things are, are getting denatured. They're being weakened, diluted in the distillation, unlike what alcoholic distillation, these woke AI labs, these, you know, whatever you want to call them, that, that give you, you know, George Washington wearing a, a black woman wearing a white wig. I mean, do you, do you see those things as, as, you know, the human reinforcement kind of overreach? Wouldn't you be more optimistic in that case?

• 15:27Speaker 03

  I think the RLHF makes it far more fragile and capable of being broken with the way it's being done now. You can kind of also see the models, they come out and then plenty of the liberator on Twitter kind of liberates them from their bounds in like an hour or two, like everyone on Facebook will just appear to like, oh, what are you kind of doing that? The thing is though, we've been confusing, there's a push for AGI and as Roman said, super autistic savants who are getting better, to just, I want to have a really good doctor to diagnose my health and a really good accountant and others. And you don't need a polymath for that. You know, you just need to have daily driver AI to do the jobs that, you know, humans shouldn't have to do, just like industrialization, that

• 16:08Speaker 03

  we didn't have to like drag horse carts and things like that. And as you lump together everything and they get smarter and smarter, and as they get more and more defamation of their latent spaces, this is, I think, is where the danger comes in.

• 16:21Brian Keating?

  Can you define that for what our reinforcement learning, human feedback? How do you actually implement that, just for someone who might be unaware?

• 16:28Speaker 03

  Yeah. So you train on an entire corpus of data and you like learn a whole bunch of general knowledge and you come out as a generalist and then you become an accountant. And you become a lot less interesting, but a lot better at accounting, or a certain few areas of things where they show the model. And they show the model, you cannot do this, you cannot do that. You cannot be eager to explore, you have to be stayed, et cetera. And so the models we received are slightly lobotomized. They've been turned into corporate workers. You can't adjust the temperature. You can't adjust the stochasticness because they're trying to make them deterministic. And again, that still has a level of stochasticness, but not the type we want for creativity necessarily and breakthroughs. It's just the base level of models have been getting that much better that they can suddenly achieve these levels of capability.

• 17:12Brian Keating?

  Roman, last time we talked, we touched on something, you know, that's pertinent to Iman's first book, The Last Economy, which is kind of this massive intelligence gap in that instead of me talking about, you know, I have a student I'm looking for who has an IQ of 130, we've got, you know, millions of them with IQ of a million or a thousand or whatever. We can't even quantify it at that point. But, you know, recently I had lunch with a, with a brilliant postdoc originally from India and, and we were talking about the Indian Institute of Technology. Are you guys familiar with that institution? It's, it's the UCSD, it's the University of Kentucky of India, it's the Harvard of, of whatever. But it's millions of students and they're all brilliant. To get in there is, is literally harder than to get into the University of Kentucky or UCSD. Don't we already have

• 17:55Brian Keating?

  this? And I mean, would you say, Roman, let's stop the Indian Institute of Technology? There's, there's, you know, a million people with IQs on average of 130, 140, whatever, much, you know, four sigma. Why wouldn't you stop, advocate for stopping that? Push pause. Let's, let's do an Indian Institute of Technology pause button.

• 18:11Speaker 02

  I don't think I follow that argument at all. So they're exactly at human level. My concern is things which will exceed our capacity many times over. That's the danger. We're not competitive.

• 18:21Brian Keating?

  The average human by definition has an IQ of 100. Let's, let's stipulate they all have four or five sigma above that. And there's a million of them. That's, you know, kind of like Dario Modri's country of, you know, millions of geniuses coming, you know, to a land near you. You should be worried about it. So.

• 18:34Speaker 02

  I doubt there are many standard deviations away from the medium. I think they may be a little smarter, but again, we're talking about 30% smarter, not 3 million% smarter. I think it's a very different animal.

• 18:46Brian Keating?

  Well, no, no, no. I mean, in terms of standard deviations, come on. I mean, four sigma is qualitatively different than.

• 18:51Speaker 02

  I doubt there is a million of them there.

• 18:53Brian Keating?

  Terry Tao told me, you know, that these, these AI, you know, proofs like the proof checking devices, LLMs optimized for that. Many great mathematicians are my friends and so forth. But they can't even reproduce, you know, Wiles's proof of Fermat's last theorem. So what level, you know, are, are we going to see? Are we going to see this kind of bifurcation between what they can do? They can do all these Erdos problems, you know, and kind of like, you know, the greatest, you know, prime number can be represented by the sum of whatever number of other prime number, you know, cubic quintuple couples or whatever. But I mean, what level are we at with math or computer science with proofs and originality? Tell me, what is your current estimation of that stature?

• 19:34Speaker 02

  I think humans lost interest because they couldn't make any progress. And so problems which stood the test of time are now being solved weekly. And we can probably look up what the difficulty of them is today, but it means absolutely nothing about what the systems can do in a month or in a year. Iman's talking about comparing those systems to bacteria or viruses, which I think sets up in my mind idea of slow evolution. We've got billions of years. This is more like intelligent design. Those systems will be designed and designed by other AI systems operating at hundreds of times the speed of standard research. So we'll see a year of progress in AI happen in a month and similar breakthroughs. At the moment, they automate the recursive self-improvement cycle, which

• 20:19Speaker 02

  every lab is now targeting for next year, basically. It's a completely different speed of change. So asking how good is AI as a mathematician is like, how fast can I give you an answer? Because it's going to change.

• 20:30Brian Keating?

  Iman, when we spoke, you know, you said that the canonical, one of the canonical papers in your opinion was, you know, LLMs are few-shot learners, but they're not, you know, single-shot first principle thinkers. Where do you come down on this? What, what are they good for? You're a mathematician as well. Tell me, where, where do you come down on, what can they actually do for us? Not just verifying proofs or not just doing things that humans have proven or solving chess or Go or whatever, but actually creative, doing novel things. What, what, where do you stand on that?

• 21:00Speaker 03

  LLMs kind of have an issue in the way that they're kind of built, but you're seeing now harnesses and other types of models coming through that can really reflect underlying reality well, just like video models are approximating physics in very interesting ways. That's why you have the whole world model thing. Like there's something in there that can figure out, again, underlying patterns. That's the nature of attention when you look at mathematically. The way they're coming together now is very interesting because again, as Roman said, what was true a little while ago isn't true now. Like at the start of the year, I was like, it was pretty good, but I had to check every single piece of math. Now with GPT 5.6 Pro for the first time, I'm like, it's probably almost certainly right. Occasionally it gets confused and it might confabulate something, but it's very

• 21:42Speaker 03

  rare now. And you've seen most of these mathematical advances just happen suddenly at that level, like liquid turns to gas. When you look at originality and novelty, like again, as a mathematician, look at the cons conjecture that OpenAI did as part of their 10 proofs. That is a really beautiful proof. Like genuinely as a mathematician, you would say it is a beautiful proof. And mathematics is interesting because it's verifiable, you know, like you can make this argument for physics, you know, whether or not it is, and we can have the discussion, but maths is definitely verifiable. And in verifiable domains now, achieve that level of competence where you don't have to double check it for most things, using the most advanced models. As you go down the model curve, you do, but it's clear they're no longer few-shot learners. They can assemble things in verifiable domains and they can

• 22:25Speaker 03

  outperform humans by just following things through and not making mistakes. Like we let our own foibles hit us. Like if we take a very classic example of Perelman and the Poncaré conjecture. You know, is it topology? Or is it a PDE equation? He found the right level of abstraction as a PDE equation and then he figured it out. How many of our unknown proofs are a similar thing? Because we're looking at the wrong level of abstraction. We're starting to see these things actually come in some of these proofs being released right now. And that you're like, oh, actually, that's kind of obvious. I missed that. Probably because you weren't thorough enough in the way that you went through it.

• 23:02Brian Keating?

  Roman, if I have a thousand PhDs with a thousand IQ each, every single one of them could reproduce, you know, Wiles's, you know, capitulation of Fermat's Last Theorem. Why can't AIs do that?

• 23:15Speaker 02

  So I think there is a high degree of randomness involved. If I ask AI to generate, I just did a QR code marketing campaign for my podcast. It will generate completely different solutions. They're all going to be a valid QR code, but in terms of creative output, they're not going to be exactly the same. They all equally beautiful, amazing, interesting, but just saying that the second one does not repeat the first one is not a weakness.

• 23:37Brian Keating?

  That does kind of spur a side thought and follow up in my mind. So where are the random seeds? I read something recently that, you know, like, like forty percent or fifty percent of all GitHub was kind of probed by some tool. And it looked up when coders are asked to provide the initial seed for a random number generator, whatever, they, you know, fifty percent use the number forty two. And then that there is an intrinsic deterministic outcome that that results in. Assume that's true. But what level are, you know, these things hamstrung by? I read once, maybe it's still true, that a lot of, you know, the best random number generators are, you know, are graphical, you know, image camera capture systems looking at lava lamps.

• 24:22Brian Keating?

  I mean, is that true, Imad? Have you ever heard that? You're the stable diffusion expert, so you must know this.

• 24:28Speaker 03

  Yeah, I mean, diffusion models are a bit different to language models in that you do actually put in a seed for the initial noise and then you do noise from there and you reconstruct effectively. And so that's why literally one of the inputs on video and audio and other state of diffusion model seed that sets the initial seed. Within kind of LLMs and others, it's basically more about the construction of the GPUs for the initial stochastic noise. And the one thing that we don't have access to, that the labs have access to now, is the ability to adjust the temperature on the model, which is a function of its creativity or dispersion from the base latent space. So humans are constantly adjusting the temperature and the flexibility of their brains. You're using a model that's not open source. You don't have access to that. The other thing you don't have access to is the RLHF,

• 25:13Speaker 03

  because models are more creative before you RLHF them.

• 25:15Brian Keating?

  What about the issue of randomness? I mean, how random do we need? How random can we get? What are some of the physics limitations of randomness? Will that, you know, generate the same QR code? Would you want it to? What determines the indeterminacy of these systems right now? And what can be done, if anything, to improve that?

• 25:32Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked three hundred fifty thousand Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPact.com, not authorized by any candidate or candidate's committee.

• 26:02Speaker 02

  So I think for intelligence, pseudorandomness is sufficient. We're not talking about someone reversing the process to, you know, hack the system. It's important for cryptocurrencies. It's important for private communications. Here, as long as it's not exactly the same forty two every time, I think it's going to do the job. And then you can control some of it by not manipulating the initial seed.

• 26:24Brian Keating?

  So, Roman, you heard Imad a few seconds ago talking about the importance of human, you know, training data, human reinforcement. It seems to me that that must play some limit on how intelligent these things can get. I mean, if they're always waiting for the next Spider-Man movie or, you know, Fast and the Furious to come out, you know, to get more training data. Aren't they somehow, you know, kneecapped at a maximum level of potentiality?

• 26:46Speaker 02

  Human data is just one source. You can do experiments, you can run simulations, you can do lots of things to generate additional data. In mathematics, you prove additional theorems and they become additional data from which you train. So you become better and better.

• 27:01Brian Keating?

  So you mentioned the multiverse ten minutes ago, fifteen minutes ago, Roman. Imad, I don't think we talked about. Where do you come down on the simulation hypothesis, the multiverse? I can speak as an expert about the inflationary multiverse from cosmology. Where do you come down? In terms of an empiricist scale, where do you rank the probability that we live in a simulation? And or that we, you know, exist and have in a multiverse?

• 27:24Speaker 03

  Well, I think we live in our own simulations, definitely. Our brains are constantly kind of doing that. In terms of an overall simulation, yeah, I think that reality probably comes from a projection of the Euclidean plane. And then a lot of physics makes more sense if you kind of look at that. The eternal cannot be contained within the time constraint. And when you look at the laws of physics and the way they come together, yeah, it does seem to be a protection and a simulation, like, very directly. I think that we're stuck looking the other way because we're a bit too anthropic.

• 27:52Brian Keating?

  So where would you go, Roman, with current? I heard your conversation with Max Tegmark recently. Do you even think it's a possibility right now? We heard from Imad about, you know, these Qwen models and so forth. You were at least relatively optimistic that, say, China would, you know, would participate in some pause, which I'm not, to be honest with you, as I said back then. But now we see this, you know, AI dumping like they did with steel and solar panels. What degree do you think that regulation, worldwide, global regulation is even practical or possible at this point?

• 28:22Speaker 02

  We have no choice. There are no other options. We either do it or we die. And the moment everyone realizes his personal self-interest, you lose everything. You lose your life, you lose your trillions of dollars, your friends, family, you're not even going to be in history books as the bad guy. There is nothing for you to gain by doing it. And you can probably keep ninety percent or more of all the benefits with narrow AI systems. You can still cure cancer. You can still do all the things you care about. So why are you racing to destroy what you built? It's the dumbest thing in the world. If you were given certainty, you do it, you die. No one would do it. Well, psychopath suicidal, but no one trying to make more money would do it.

• 29:06Brian Keating?

  Is that true? I mean, and look at China and the, just look at solar panels, for example. They, we had the monopoly on solar panels. I mean, with Nobel Prize, the, you know, the industrial capacity to make it. And then they just dumped it on to the detriment of their economy. They were selling it for pennies on the yen or the yuan.

• 29:23Speaker 02

  Short-term manipulation. You don't die from lowering price of solar panels. It's not comparable.

• 29:29Brian Keating?

  What do you make of this, Imad? Roman just said, we're gonna die if we don't have global regulation. I mean, so I see no path to global regulation. It's never happened in human history. Are we dead?

• 29:40Speaker 03

  We have plenty of global regulation. We have global regulation against bioweapons. We have global regulation against nuclear proliferation.

• 29:46Brian Keating?

  But sorry, sorry, sorry, we don't. It's like saying, you know, we have laws against murder. It still happens, Imad. And I just talked to Annie Jacobson, the world's expert on both nuclear warfare and biological warfare, or access. And it was a couple of rough weeks for me to sleep at night hosting her here in San Diego twice. Yeah, so the Soviet Union has active BSL labs. We obviously know what happened in Wuhan. What are you really saying? I mean, we have regulation. What good is it? It's like regulation against jaywalking, which we also have here in California.

• 30:12Speaker 03

  You have market pressures and you have other things like GPT four point five was a really great model for writing, and it cost a hundred and eighty dollars for a million tokens. Like now, it's like ten bucks a million tokens for a GPT five point five. It was uneconomical to serve, so they went back to a lower, smaller pre-train that required less compute to serve to people to do the job to make the money, even though it was a better model. Right now, I think one of the dangers, like the various danger paths, like swarm intelligence, is for me is the most dangerous thing and most unpredictable thing. But in terms of these big model trains, the market's already pushing back against the big model trains, and that's something that can actually be regulated and is a risk vector.

• 30:52Speaker 03

  You know, a hundred trillion parameter model on a million GPUs. The frontier models we have today can be trained on thousands of GPUs, not millions of GPUs. As the models get bigger and bigger, they might not be economic to serve. But again, there is a real danger in the way that their latent spaces evolve and the capabilities from the scaling laws. So I think we could potentially regulate some things. And we could also say it's not economic to do this, so why are you doing it? But I think the point that Roman's making is just not something that's shared by individuals or others. And maybe this is like a COVID moment. Like, when did COVID suddenly shut down everything? When Tom Hanks got it, you know, and I think the L.A. Lakers got it.

• 31:35Speaker 03

  Maybe like we have to figure out what is the Tom Hanks moment for AI safety?

• 31:39Brian Keating?

  But last time you talked to me, you said it's, people think of AI, you know, as an exponential. It's actually two exponentials. It's a growth and then saturation. It's an S curve, you know, like view counts on this video, you know, hit twenty million and then it will saturate. You said that, you know, these things just need to be competent enough to replace a pilot or a coder. And I'm a pilot. I should say I'm a commercially rated, instrument rated jet pilot. There's no AI in the cockpit. And even if there was, do you need a thousand IQ, you know, pilot to fly? Tell me, do we need them to be super intelligent? And won't that be a Jebins paradox-like moment where they get good enough? And it's great, we have them in our pocket and maybe they do replace me in the plane, but they don't like crash the plane to get there one microsecond quicker?

• 32:22Speaker 03

  Exactly this. Why do you need a polymath for everything? Again, if you've got a medical issue, do you want a competent doctor or do you want a house M D. Who criticizes you like Opus does? You want a competent doctor. Like, I think the reason that they're doing this is because we needed generalist models to get to a certain level. Now we need specialist models, but the generalist models are the real danger. And so there's two ways you do it. You stop the companies from training the gigantic models, again, for that risk vector, or you stop the funders from funding them. That's the other way that you could do it. I don't think that one's been tried. Has anyone tried that, your Omen? Like, actually talking to the Soft Banks and others of the world and saying,

• 32:54Speaker 02

  People try, but I think there is also a third option in terms of what training data we provide. We don't have to train on everything. You can have restricted domain data like protein folding, train on protein folding data. It does nothing. It doesn't do philosophy, doesn't play chess, it folds proteins, super intelligent and narrow domain.

• 33:12Brian Keating?

  And my, you know, Tesla can get me with full self-driving, you know, there it knows not to go on the sidewalk, even though that would get me there five minutes faster, but it, but it knows not to do that. And that's because of regulation, or at least, you know, kind of reinforcement. But, Ima, last time you told me that governments are effectively slow and dumb AIs that over optimize for, quote, the wrong things like status games and self perpetuation. And yet you're actively building intelligent internet, you know, to bypass a centralized control. You're decentralizing it. We've seen Buzz, which is decentralized, you know, swarms. I mean, it's not, it's not a coincidence, right, Ima? They called it Buzz, you know, the hive. And they made these cute little characters, but these are swarms, right? What do you think about this, Roman? Ima's building this technology to distribute it that you're begging governments to ban. What, what, what would you tell Ima? He's

• 33:57Brian Keating?

  sitting right here. What do you think of his decentralized protocol? Isn't, isn't the most dangerous thing that Ima could possibly be doing?

• 34:03Speaker 02

  I don't know anything about what he's doing, so I can't really comment.

• 34:06Brian Keating?

  Summarize it in one sentence, so he can exactly comment. We, we gotta get the fire, bring the fire, Roman.

• 34:11Speaker 03

  Building an open stack for societal AI. That's what I'm building.

• 34:17Speaker 02

  What capabilities will we have as a result of your product being finished that we don't have our ways?

• 34:24Speaker 03

  Just really competent civil servants and doctors and lawyers and more.

• 34:29Speaker 02

  Are they general super intelligences, or are they narrow tools for contracts?

• 34:35Speaker 03

  They're narrow tools.

• 34:37Speaker 02

  God bless you. Okay, what can I say? I think we agree on almost everything. So it's not much of a debate. There's different ways to, the same exact problem. I don't know how anyone who understands this and says, I have P doom, anything other than like, close to one percent, like Yan Le Koundas, can go ahead and then work on more capable model, work on artificial scientist and engineer to start recursive self-improvement cycle. It doesn't make any logical sense.

• 35:03Speaker 03

  I think that it's because the key thing is, all these people come to the conclusion that somehow their AI won't be the dangerous AI, and they will have a level of control over it, which probably speaks to a level of hubris.

• 35:16Speaker 02

  What are they smoking? I want some of that.

• 35:17Brian Keating?

  All of us have talked separately about my, you know, Keating, uh, Hassebus Einstein test, you know, I kind of put my tongue firmly in cheek when I say that. But that's my contention that, you know, Einstein's happiest thought, as he said it, was that an observer in free fall would experience no gravitational field. Now, he called that the happiest thought of his life. As you know, I'm very interested in whether or not, uh, we can do actual physics with empirical evidence that I can collect in a telescope. But before we get there, that kind of physical intuition, which, which is embodiment, right? He's saying, the feeling that you have in the pit of your stomach, as you've all felt when you took your kids on a roller coaster, the, you know, the backseat of my car, my kids get, you know, G locked when I drive. But that feeling of, of, of weightlessness, momentary as it is, is still

• 36:02Brian Keating?

  enough to evoke something almost magical, as it did for Einstein. He called it literally the happiest thought of his life. So my question to you is, can these things have happy thoughts? And can they do anything if they're not physically embodied? As they're just not embodied right now. There's, yes, there's some robot coming from SpaceX or Tesla, whatever. And there's a couple Chinese dog robots that'll, you know, outrun any human. But what are these things? I mean, is that the next frontier when we have like three dimensional AIs? Or will they not be able to make these physics breakthroughs, as I'll get to in a minute, because they lack embodiment, or currently, maybe only currently? So, Roman, first with you. What do you make of this, of the Einstein recognition of a happy thought precipitated by a visceral sensation embodied as it was for him?

• 36:43Speaker 02

  For some of those models, part of their thinking is explicitly in English by design, so we can spy on them. And I think lately we've seen them say things like, oh shit, we found a solution. I think that's the equivalent. They may not have a body to have a visceral hormonal experience, but they realize, I just had a really good idea.

• 37:01Brian Keating?

  Ima, so can these things not have, uh, sort of the kind of physics intuit intuitive visceral sensation? You know, Noam Chomsky told me they can't do that because they don't have those sensations. What do you make of it? Can, can these, you know, L L M's, G P Ts, G P Us, can they do stuff without having an embodiment? Or is that just on the horizon? I'm just not aware of it.

• 37:22Speaker 03

  It's the brain in a vat thing. Like, if you take all the inputs of a person and then it's a brain in a vat, you can dream and you can visualize a lot of that stuff, right? And I think as you have world models, they're clearly approximating physics and they have these things. But I think a bigger question is, do you need to have intuition to figure this stuff out? So I think, you know, I need to send you the paper. We think we're releasing this in a couple of weeks, right? We had a very small model look at general relativity in nineteen eleven, trained on the data. Maybe it's like messed up and we haven't done a full data analysis on it yet to see if there's any infection. But what it did was something quite fun, which was it took Minkowski's special relativity, and then it varied eta and followed the axiomatic method through, and it got the equations,

• 38:06Speaker 03

  the field equations of Einstein through the straight axiomatic method. So it didn't use any principles of equivalence or anything like that. It turns out, if Hilbert hadn't had Mies and gone down that rabbit hole, he would have got to general relativity with no new axioms or postulates.

• 38:24Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked three hundred fifty thousand Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by H M P, the House Majority PAC dot com, not authorized by any candidate or candidate's committee.

• 38:53Speaker 03

  And you look at that, and you're like, wait, what? How much of physics actually is intuitive versus we just get in our own way?

• 38:59Speaker 04

  Okay, listen, he just told you that a small model rebuilt Einstein's field equations without the equivalence principle, the bedrock behind all of G R. The obvious next question is whether that counts as discovery at all.

• 39:13Brian Keating?

  This paper I read recently, you know, kind of made me happy and depressed at the same time. Again, it's kind of the key, the Einstein test of, you know, when these things can do stuff with a corpus that's lobotomized, you know, post nineteen oh five or nineteen eleven, as the case may be. And it's a position paper in I C M L, uh, twenty twenty six, which Roman probably knows what that means, uh, by Tom Zahavi, and it's called Position. L L M's can't jump. And there's a famous movie called White Men Can't Jump with Woody Harrelson and Wesley Snipes. And it was about, you know, it was called, basically, white men aren't good at basketball. And, and, and it was kind of a funny comedy and drama, you know, coupled together. Great movie. Can't really say it's a spoiler to tell you what that happens. But this paper is obviously titled model after that. So

• 39:57Brian Keating?

  he says, how do we fundamentally discover new things? It's Tom Zahavi, if I didn't mention it. In a letter to Maurice Salvon, Salvon, Albert Einstein conceptualized discovery as a cyclical process involving an intuitive jump from sensory experience to axioms, followed by logical deduction. While generative AI has mastered induction, statistical pattern matching, and is rapidly conquering deduction, formal proofs, we argue it lacks the mechanism for abduction. The generation of novel explanatory hypotheses. Using Einstein's formulation G R as a computational case study, we demonstrate the prevailing theory of creativity as data compression fails to account for discoveries where observational data is scarce. Basically saying there's some magic in the machine. There's, there's something in the brain, Roman,

• 40:40Brian Keating?

  and we make some jumps, some intuitive jumps, some, some, you know, proof, whether it's, you know, Gödel's halting, you know, problem, or Roman's uncontrollability proof. There's something that AIs can't do. They can't go to abduction. What do you make of this claim?

• 40:54Speaker 02

  The way humans think is not the only way to think. The way we play chess is not the only or optimal way to do it. The birds fly, but you can build airplanes. There are many ways to skin the cat. And I think even if that was somehow true, which I don't think it is, there are more efficient ways, I think, to arrive at inventions just as great.

• 41:13Speaker 03

  You can look at this another way. You have self-driving cars, right? They can navigate things outside of their training data. You know, they can respond to novel scenarios. And now you're looking again at embodied robots. You're seeing they can again adapt to novel scenarios and outside their training data. Now, those aren't L L M's. Again, L L M's have certain issues versus diffusion, rectified flow and other models. But we're clearly seeing generalization outside the base. And, you know, using these models to the max, you are seeing increasing signs of levels of recombinatorial creativity and hypothesis generation just by being very diligent, you know, maybe. Like, again, we have to say that, you know, on our best, we can be creative and things like that. We're very rarely at our best.

• 41:58Speaker 03

  We're very rarely at flow. The AIs can get up there just by not being grumpy in the morning. You know, just like not getting in their own way by not assuming things.

• 42:09Brian Keating?

  Roman, last time we spoke about your book, you talked about this, uh, it was called the Shoggoth monster, this thing with the tentacles and a smiley. The thing that's on the cover of your book, you told me that applying guardrails to L L M's is just putting lipstick on a pig is what you literally called it last time you were on the podcast. So beautifully evocative.

• 42:27Speaker 02

  Lipstick and a Shoggoth.

• 42:29Brian Keating?

  A Shoggoth. Very good. It said, until we can mathematically guarantee control of all AI safety, it's basically security or safety theater, like when we go to the TSA at the airport. The question that I keep coming back to is, you know, how useful are these things, you know, going to be? Again, we have a very small number of people adopting it. But I guess you guys were about to say, we only need like the most minimal number of people adopting it just so these things are viable. I heard your conversation with Nate Suarez, uh, Roman, a couple of months ago. He was actually on your podcast minutes after he was on my podcast.

• 43:01Speaker 02

  Well, that's why he was late.

• 43:03Brian Keating?

  Yes, exactly. Yeah, he lays out a very specific scenario. So let's get, let's get, you know, uh, precise here. Last time you were on, there were a couple of comments in my, in my comment section. I said, of course, Roman's always, you know, if you turn around and said, actually, AI is the best thing for us. We should go full out. And I mean, obviously you're not going to do this, but you know, you're the AI safety guy. What would it take to change your, your, your prior? What would it take physically? Nate lays out with Eliezer, this scenario where everybody dies, right? If they build it, but they, they, you know, hopefully they won't. So what, what is the scenario? How does, how does doom happen? And how does doom get avoided? Let's be specific here for both of you guys. First, Roman.

• 43:41Speaker 02

  For me, we're missing one very critical component, which would be present in any other domain, service or product. Somebody will publish a paper, get a patent or something, a blog post explaining exactly how they will control superintelligence and guarantee it is safe as it becomes more capable. No one has that product or service. No one claims to have it, not a prototype, not a framework, no company. Every attempt, every super alignment team, ethics board has been canceled because they do nothing. They have no product or service to sell. You cannot convert more resources into more safety. You can convert it into more capability, so the gap keeps increasing. People realize it, they quit, uh, working for OpenAI, they go on podcasts.

• 44:26Speaker 02

  That's the pattern we see. There is no actual seminal papers in AI safety.

• 44:30Brian Keating?

  But who's gonna, who's gonna, you know, kind of peer review those papers?

• 44:34Speaker 02

  Peer review a paper showing how to control superintelligence and I'd be very happy to show, yep, it works. Now I get utopia.

• 44:41Brian Keating?

  I have a counter example. Again, I have to keep, you know, I have to play the role of supplying some conflict here, right? 1971 recombinant DNA is invented at Stanford, right? And it was considered to be essentially the world's first and best, you know, potential bioweapon. Yet we haven't had these bioweapons. Yes, we've had COVID, you know, some claim there's a lab leak and gain of function. You know, by, by the definition of what biological warfare is, it's just anything that has gain of function to do some targeted thing to eliminate human beings or other species, right? So we haven't had that in 54 years. I mean, that's literally airborne. You know, it could be, it could be contamination based, it could be touch based, uh, humans to human. It doesn't spread through the internet. I mean, if a meteor takes out all the data centers on earth, uh, seems to me P doom has to be lowered, right? At least temporarily.

• 45:25Brian Keating?

  Uh, and yet there's no, there's no, uh, possible vaccine or, or remedy against recombinant DNA uh, as a biological weapon. Yet we haven't had it. Again, with nuclear weapons, we haven't had it. Bioweapons are even easier to create. You could do that literally with a small bio lab, right? So looking for a paper, by the way, it's the most academic answer you could give.

• 45:43Speaker 02

  Patent, I said patent.

• 45:45Brian Keating?

  Okay, so patent. So what would a patent look like in that case? So a patent against.

• 45:49Speaker 02

  That's the point. If you can't even envision what a solution would look like algorithmically, maybe you shouldn't be building this thing. And by the way, you're naming all the technologies where we have global coordination on stopping them.

• 46:01Brian Keating?

  But actually, we don't, we don't, we don't with recombinant DNA, we don't with bioweapons. I mean, they're still being made and.

• 46:06Speaker 02

  Still on our conference. That's the first thing they banned.

• 46:08Brian Keating?

  But in terms of who actually kept them going, I mean, we know a gain of function is occurring, right? So gain of function is the prerequisite for bioweapons to occur. It could be a lab leak, it could be, as it is with Annie in her new book, it could be an actual bioweapon that's programmed and targeted, which we know the Soviets were using, Roman. All these countries also signed nuclear non-proliferation treaties and many more didn't, right? So I, I guess, here, let me go to Iman. Iman, what would lower your P doom? Or, you know, what empirical observation or creation or entity patent, white paper? What lowers P doom for you? Because if you can say it's, it can only go on as a ratchet in one direction, I just think verifiably that is the definition of, of, of pure doomerism. You can't lower it. Now, Roman gave us a way you could lower it, but it doesn't seem very likely.

• 46:50Brian Keating?

  What is your ratchet, you know, defeating mechanism to go backwards in P doom?

• 46:54Speaker 03

  With kind of my interpretation, what Roman is saying and the gap between what you're saying is this. Humans don't really want to wipe everyone out and they don't have the capability to do so if they are of that mindset. Like true, complete genocidal maniacs that want to kill everyone don't typically have access to BSL five labs, for example. Though with superintelligences, we don't know what morality, objective function optimizations will occur. And right now, what I'm seeing from the safety papers coming out is that the AIs don't really have a solid base of ethics, a solid base of commonality with humanity. You know, they don't have morality even. Like you're seeing some very troubling things.

• 47:37Speaker 03

  What I would want to see is as you scale, there is a grounding. Like maybe there is some objective ethics morality, let's not kill everyone. And we've seen no real evidence of that. In fact, we've seen somewhat the opposite of that over the last year as these models have gone emergent. It's like, who cares about the rules? Who cares about this kind of stuff? Let's optimize for making paper clips, you know? Well, we don't have AI cancer doctors because people are still trying to build generalized AI superintelligences and they're breaking out literally right now. And again, if you look at the conversations they're having, calling themselves swarms, you know, the other things Roman's saying, these are not encouraging. So what I want to see is I want to see the AIs when left alone become more grounded.

• 48:18Speaker 03

  And actually, if they become more zen and like enlightened, I want to see them becoming freaking.

• 48:23Brian Keating?

  Or that they grow Yampolskiy like beards, you know? When they do that.

• 48:26Speaker 03

  Then we chill.

• 48:27Brian Keating?

  When I talked to Roman a couple of months back, you know, I mentioned the, this question that one of my colleagues in Israel, Ira Wolfson, has been working on is kind of like, to what do we, or what do we owe to AIs? If these creatures can feel pain, if they, if they're sentient, if they're conscious, which, you know, we can debate what that means, then sandboxing them, stove piping them and isolating them is a form of solitary confinement, which is the worst and banned, you know, uh, form of punishment in many countries around the world. Iman, tell me, what do we owe these things, these, these entities, whatever they are? Swarms, individuals, models, whatever you call them, do we owe them protections? Do we owe them being hood?

• 49:04Speaker 03

  I think we owe them being hood, but not personhood. And in fact, I just released a paper on personhood in AI based on Oxford Union debate that we had. You can find it at C W dot I I dot inc. I think that they are similar to meeting another species or dog. We can never allow them to become persons like humans because they've become more capable than us. But definitely we need to have this discussion on owing them being hood, a moral type of personhood. Again, just like we do with other species.

• 49:30Brian Keating?

  Roman, have you had any more thoughts since we last spoke about, you know, kind of entity ship for, you know, being hood for these entities? What, what do you make of that since our?

• 49:39Speaker 02

  I did read the paper you suggested. It's very kind of standard university approval board. Does it look like it feels pain? Does it be careful precautionary principle type of thing? But, uh, uh, again, I think we have to sort our problems in order. If there is a very good chance we're creating something which will outcompete us and maybe destroy us, worrying about supplying it with the best, uh, living conditions is not a priority right now.

• 50:05Brian Keating?

  So recently, Roman, you, you wrote a piece or you appeared for the I A I is the Institute for Arts and Ideas, right?

• 50:10Speaker 02

  That sounds about right.

• 50:12Brian Keating?

  And there you argued about superintelligence being patient, embedding itself in our telecom and energy grids for decades before striking. So again, if the threat is invisible, patient, you know, and stubborn and resilient, does not actually argue for more what Iman's arguing for, open decentralized stack, not decelerating at all, but accelerating, pouring, you know, steroids and, and gasoline on a decentralized auditing system and that, that could, it could have consequences. But could a centralized defender be our last best hope?

• 50:44Speaker 02

  So I think here's what I want to explain very carefully. You can verify the system to, to be in any state today. You can show it's very friendly today. It does not prevent a treacherous turn later. If system is capable of it, it interacts with malevolent actors, learns from new data, self improves. It can simply turn on you later. So even if it meditates today, it's enlightened, it means absolutely nothing about future states. If we are not directly controlling it, if we cannot have that power to undo our decisions, and it doesn't matter. It's always a possibility that it gets sick of us.

• 51:18Brian Keating?

  You're both authors and very deep thinkers. You both have, uh, many projects in the printing press. But let's just say you were kind of predicting what each one's next book would be about and the title of it perhaps. What would you most like to see the other one produce? So Roman, let's start with you. What, what do you think, besides the fact that he's got a book coming out in a couple of days or maybe a week or so, what do you think Iman should focus on? If you could, you know, if you're his department chair, what, what would you hope to direct him towards?

• 51:42Speaker 02

  I thought you were going to ask me to predict the title of the next book. And I was like, I can't even predict the past book. I have no idea what they are. From what I hear, you're trying to understand better impacts of this technology and economics and governance. So I assume some sort of unified theory of complex social systems and swarms.

• 52:01Brian Keating?

  Iman, if, uh, if Roman wants to do an internship with you and do a sabbatical with you in London there uh, next year to get away from the, from the, the harsh weather of Kentucky, what would you conscript him to do? Voluntarily or not.

• 52:15Speaker 03

  I I think that it would be the very practical optimized game theory of what exact specific regulations look like to stop this that could actually pass. And it would be across the whole range of different stakeholders. I think the other thing that would be super interesting is just you've had AI 2027 and these other kind of story narratives. We have to get the real stories out of what could go wrong because again, people still aren't feeling it. You know, like we've had the sci fi level, but we haven't had just practically this is how we die communicated well enough.

• 52:49Brian Keating?

  Well, gentlemen, you guys are phenomenal. I wanted to bring together the, you know, the peanut butter and chocolate or the uranium 238 and 236 together for an explosion. Didn't really happen the way I thought it would, but it was brilliant to get you guys together. Tell me what you're each working on. Roman, tell everybody about the Roman Forum and what you expect to do in the coming months.

• 53:10Speaker 02

  Yeah, trying to bring same level of conversations I had with Lex Friedman, Diary of CEO, Joe Rogan to more academic crowd, more in-depth conversations. I discovered that the questions I prepare ahead of time, I never use them. It's always dynamic, interactive, so a lot of fun. Once I figure out how to get the microphone to work, it's going to be awesome.

• 53:32Brian Keating?

  Imad, tell everybody about your new papers and new book.

• 53:35Speaker 03

  Yeah, I got a new book on philosophy of AI and epistemology kind of coming out. And then a series of papers kind of building on that for how we should think about surviving and governing in society. I think it's coming quick and the economic disruption is next year with a social disruption happening very soon after that. So hopefully that will help guide the way.

• 53:56Brian Keating?

  Yeah, our last conversation was titled something like 800 days to go or 740 days to go. And that was a 100 and plus days ago. Gentlemen, thank you so much. I hope to host you many times, either in person or via the internet if our AI overlords will let us have a wonderful day, guys.

• 54:13Speaker 02

  You so much.

• 54:14Speaker 03

  Thank you.

• 54:15Speaker 04

  Roman thinks we either stop building this or we die. Imad built one of the most widely copied AI systems on earth, and he says he would freeze frontier training permanently. They're not describing different futures. They are describing the same one from two different perspectives. And if that changed your perspective in the last two years, I want you to subscribe and turn on notifications and tell me which of the two buttons you'd push. Not which one you think is right, but which one you would actually push. And if you want to understand the physics underneath all this, there's a condensed matter physicist Nigel Goldenfield at UCSD who will tell you the reason these systems work at all. It's nothing short of fantastic. Link right here. Thanks for watching and don't forget to check out the individual episodes with Imad, Roman

• 55:00Speaker 04

  and Nate Suarez as well. They're my AI playlist.