Superintelligence: To Ban or Not to Ban? Max Tegmark & Dean Ball join Liron Shapira on Doom Debates

Max Tegmark and Dean Ball debate whether we should ban the development of superintelligence in a crossover episode from Doom Debates hosted by Liron Shapira. PSA for AI builders: Interested in alignment, governance, or AI safety? Learn more about the MATS Summer 2026 Fellowship and submit your name to be notified when applications open: https://matsprogram.org/s26-tcr. They unpack the Future of Life Institute's call for a moratorium until there is broad scientific consensus and public buy-in, contrasting Tegmark’s precautionary stance with Dean’s emphasis on experimentation, competition, and practical policy hurdles. Listeners will get clear takes on p(doom), the limits of FDA-style regulation, unilateral ban risks, and what safe, beneficial advanced AI might realistically look like. LINKS: AMA episode feedback survey form Doom Debates Substack newsletter Sponsors: Framer: Framer is the all-in-one tool to design, iterate, and publish stunning websites with powerful AI features. Start creating for free and use code COGNITIVE to get one free month of Framer Pro at https://framer.com/design Agents of Scale: Agents of Scale is a podcast from Zapier CEO Wade Foster, featuring conversations with C-suite leaders who are leading AI transformation. Subscribe to the show wherever you get your podcasts Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (05:43) Cold open and intro (09:21) Opening statements: ban debate (Part 1) (14:49) Sponsors: Framer | Agents of Scale (17:11) Opening statements: ban debate (Part 2) (17:11) Licensing-style AI regulation (26:52) Liability, tail risks (Part 1) (33:24) Sponsors: Tasklet | Shopify (36:32) Liability, tail risks (Part 2) (39:23) Timelines and precautionary regulation (47:03) Defining superintelligence and risk (52:26) Risk-based safety standards (56:28) Current regulations and definitions (01:05:23) Max's doom scenario (01:19:46) P-doom gap and adaptation (01:34:40) National security and China (01:43:57) Closing statements and reflections (01:55:22) Host debrief and outro (02:02:10) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. A couple of quick notes before getting started today. First, if you're interested in a career in AI alignment and security, you should know that MATS will soon be opening applications for their summer 2026 program. MATS is a 12-week research program focused on AI safety, featuring world-class mentors from Anthropic, DeepMind, OpenAI, the UK's AI Safety Institute, and more. 80% of MATS alumni now work in AI safety, and I've heard so many great reviews of the program that I personally donated to MATS as part of my year-end donations last year. Applications open December 16th and close January 18th, so you've got some time, but don't delay.

• 0:45Nathan Labenz?

  Visit matsprogram.org/s26-tcr for more information. That's M-A-T-S program.org or see our link in the show notes. Second, to end 2025, or perhaps to begin 2026, I'm planning another AMA episode. Last year, we got a ton of great listener questions. This year, I will again invite your questions, but I'll also be asking ChatGPT and Claude to mine their memories of our interactions to come up with some questions of their own. Submit questions at the link in the show notes or feel free to DM me, and let's see if humans can ask better questions than AIs for a little while longer, at least. With that, today I'm excited to share a special crossover episode

• 1:30Nathan Labenz?

  from the Doom Debates podcast hosted by Liron Shapiro. The occasion for this debate is a recent statement organized by the Future of Life Institute, which calls for a ban on the development of superintelligence, described as an AI system that, quote, "can significantly outperform all humans on essentially all cognitive tasks," end quote, "unless and until there is broad scientific consensus that it can be developed safely, and there is strong public buy-in for doing so." On one side, we have Max Tegmark, president of the Future of Life Institute, which organized the statement, and MIT professor who's pivoted his research group to focus on AI with some outstanding results, including a paper on training models

• 2:13Nathan Labenz?

  for mechanistic interpretability called Seeing is Believing, which we featured in 2023. On the other side, we have Dean Ball, frequent guest on the show, previously AI advisor at the White House, and famously the primary author of America's AI Action Plan.

• 2:30Nathan Labenz?

  To put my own cards on the table, I did sign the statement because I do oppose a rush to superintelligence, especially via recursive self-improvement, as seems to be roughly the current plan at multiple top AI labs. Of course, at the same time, I am more passionate than ever about AI doctors, and you'll hear a similar form of techno-enthusiasm from Max. He's excited about AlphaFold, self-driving cars, and all sorts of controllable AI tools, just not a fully general and autonomous digital species, quite literally designed to replace us, which might lead to a potentially irrecoverable disaster. It's that sense of impending disaster that motivates Laurent to make DOOM debates.

• 3:12Nathan Labenz?

  And in my opinion, one of the most interesting parts of this debate was when he asked Dean for his P-DOOM. Dean said 0.01%, an answer he later wrote on Twitter was made up on the spot because he considers the concept of P-DOOM to be fundamentally unserious. On this point, I have to say, much like the concept of pornography, which Max briefly invokes in the conversation, I feel strongly that the concept of DOOM is meaningful. And given what we're hearing from the Turing Award-winning "fathers of deep learning" who also signed this petition, I can't see any argument getting me to reduce my own P-DOOM below 1%. At the same time, Dean does make many strong points in this conversation as well.

• 3:55Nathan Labenz?

  As someone who values bottom-up experimentation and innovation, and who believes that patients should have a right to try potentially lifesaving medicines, I definitely don't love the idea of an FDA-like regulator for AI. If that were the best we could do, I might hold my nose and go for it. But the fact that this is the go-to analogy does, from my perspective, support Dean's point that it won't be easy to convert the high-level petition statement to actual effective policy. And similarly, while I, of course, reject the idea that basic AI regulation means conceding the AI race to China, and it should be noted that the CEO of Chinese AI lab, Jiepu AI, the subject of our last crosspost, also signed the petition as well, I still have

• 4:36Nathan Labenz?

  to admit that a real unilateral ban does pose competitive risks that aren't easily hand-waved away.

• 4:45Nathan Labenz?

  The bottom line for me is that while no one has the clarity they'd need on what superintelligence will look like, or how it will be created to craft perfect policy language today, I do believe that a race to superintelligence is a bad idea. And I'm glad to have done my part to help create common knowledge that many well-informed people do in fact feel this way. Finally, before getting started, I want to give credit to Liron on Doom Debates. I generally don't like debates as a format, but by getting such outstanding guests as Max and Dean to focus in on what is very plausibly the most important question of our time, that is, how likely is it that advanced AI will in fact go catastrophically wrong? Liron is making the format work, and I definitely encourage everyone to subscribe.

• 5:31Nathan Labenz?

  For now, I hope you enjoy this debate on the wisdom of building or banning superintelligence with Max Tegmark and Dean Ball from Doom Debates, hosted by Liron Shapiro.

• 5:44Speaker 01

  I would argue that artificial superintelligence is vastly more powerful in terms of the downside than hydrogen bombs would ever be. So if you think of it as actually a new species, which is in every way more capable than us, there's absolutely no guarantee that it's going to work out great for us. If we treat AI like we treat any other industry, we would then have safety standards. Here are the things you have to demonstrate to the FDA for AI or whatever.

• 6:06Dean Ball?

  I think the fundamental thing to think about here is really assumptions. There are many worlds in which humans can thrive amid things that are better than them at various kinds of intellectual tasks. And I just have very serious issues with the idea, we're just going to be able to pass a new regulatory regime and everything's going to go fine and there will be no side effects. And these analogies of FDA to AI are not really very good. It's not to say that I don't think we need something like an FDA.

• 6:34Speaker 01

  But then I'm confused by why you don't think we should have the same for AI. So what's the difference between? AI on one hand.

• 6:40Dean Ball?

  Let me just, let me, let me make an uninterrupted point for a few minutes.

• 6:44Speaker 01

  Yeah, yeah.

• 6:44Dean Ball?

  If you don't mind. Um, okay. I think that there will be tons of side effects and I think that we will stave off a lot of wonderful possibilities for the future.

• 6:52Liron Shapira?

  Maybe the real crux of disagreement is your mainline scenario. And so let me ask both of you this question. What is your P doom?

• 7:01Speaker 01

  If we go ahead and continue having nothing like the FDA for AI. Yeah, I would think it's definitely.

• 7:08Dean Ball?

  I just kind of have this sneaking suspicion that like, if the models seemed like they were going to pose the risk of overthrowing the US government or anything in that vicinity that like, I don't think OpenAI would release that model or Anthropic or Meta or XAI or Google. Like, I just don't think they would.

• 7:34Liron Shapira?

  Welcome to Doom Debates. Today, I'm excited to bring you a debate between two of the world's leading voices on AI policy. The question at hand? Should we ban the development of artificial superintelligence? The stakes are high. Advances in AI have become the key driver of our economic engine. Artificial intelligence is increasingly facilitating breakthroughs in manufacturing, healthcare, education, even basic science. The prediction market Metaculous estimates that the first true AGI, the first fully general human-level AI systems, will be achieved by 2033, less than 10 years from now. Many experts believe that that milestone will soon be followed by the creation of artificial superintelligence,

• 8:16Liron Shapira?

  a system that surpasses the capabilities of the entire human species. That brings us to our debaters. Two of the clearest voices who disagree about how society should approach these developments. On one side of the debate, we have Max Tegmark, an MIT professor who believes we should ban superintelligence development until there's a consensus that it'll be done safely and controllably and strong public buy-in. His research has focused on artificial intelligence for the past eight years. He is also the co-founder of the Future of Life Institute, a leading organization dedicated to addressing existential risks from AI and other transformative technologies. Max, welcome to Doom Debates. Thank you. On the other side of the debate, we have Dean Ball,

• 8:58Liron Shapira?

  who completely disagrees with banning superintelligence. Dean is a senior fellow at the Foundation for American Innovation, has served as a senior policy advisor at the White House Office of Science and Technology Policy under President Trump, where he helped craft America's AI Action Plan, the central document for US federal AI strategy. Dean, welcome to Doom Debates. Thank you so much for having me. OK, let's do opening statements. Max, the starting point of our debate today was a dispute between you and Dean over your statement on superintelligence, the Future of Life Institute's statement on superintelligence that was published on October 23rd. And the statement says, "We call for a prohibition on the development of superintelligence,

• 9:40Liron Shapira?

  not lifted before there is one broad scientific consensus that it will be done safely and controllably, and two, strong public buy-in." So, why should we ban superintelligence?

• 9:53Speaker 01

  Well, if you negate that statement, then you're saying that we should be allowed to go ahead and build artificial superintelligence, even if there's no meaningful consensus at all that it can be kept under control or that people even want it, right? And if we were to say that, then we would be basically doing the most spectacular corporate welfare, because we don't do that in any other industries. Yet, right now, there are more regulations on sandwiches than superintelligence in the US. If you want to sell drugs, medicines, cars, airplanes, you always have to demonstrate to the satisfaction of some independent scientists who don't have a conflict of interest

• 10:36Speaker 01

  that this is safe enough, the benefits outweigh the harms. I'm just saying we should treat superintelligence the same way. And right now, 95% of all Americans in a new poll don't actually want to raise the superintelligence. And most scientists who work on this agree that we have no clue at the moment how to keep something which is... So vastly smarter than us, under control.

• 11:03Liron Shapira?

  Okay. And Dean, you oppose the public statement and you don't share Max's views on prohibiting superintelligence. Give us your opening statement. Why do you think we shouldn't ban superintelligence?

• 11:13Dean Ball?

  So I think that the concept of a ban and of superintelligence in general is just quite nebulous. And that is the fundamental issue that I have. AI systems that could pose substantial danger to humans are, you know, they're not disallowed by the laws of physics, at the very least. I think there are really serious questions about how close those things are and how likely we are to build those things in the near future. My guess is five years ago, if you were to try to describe general superintelligence in a law that a lot of people could agree to, you know, which would be the way that you

• 11:55Dean Ball?

  would affect something like a ban, all the things Max referenced are things that we impose those requirements through laws, right, on airplanes and drugs and whatnot. So if you're going to have a law, you're going to have to define superintelligence in a statute. And I think that the problem you will run into there is that you will define things, you'll define it in such a way that you actually end up banning many things that we would want. There's many ways that you could plausibly define superintelligence that would negate technologies that I think would be quite beneficial to humanity. I mean, imagine an AI system that has largely solved mathematics, right? It's solved all the outstanding problems

• 12:38Dean Ball?

  that we have in mathematics. It has advanced certain domains of science, maybe many domains of science by, you know, the famous century compressed into a decade, right? Or compressed into five years, let's say. It's accelerating the AI research itself that's doing that in meaningful ways because one of the areas of science that it knows how to do experiments in is, you know, computer science and AI research. It's a better legal reasoner than you or me or anybody else. It's better at coding than you or me or anybody else. I can imagine such a system like that existing. In fact, my guess is that such a system will exist by roughly 2030 without posing the kinds of risks that

• 13:24Dean Ball?

  You know, Max is worried about, which again, I don't think are impossible. I just place a lower probability on them. And so I worry that what we end up with is in practice, what you end up with if you tried to effect such a ban would be, you know, we're going to ban N plus, GPT N plus two, right? Is in practice what it would mean. So there's UPT five and that's N and there's UPT six, which would be allowed. And then GPT seven would be the thing where we say, no, that that's just, we've decided that's too scary. And so we're going to basically ban that. And then, um, you know, what happens after that? Well, in order to figure out anything about whether superintelligence is safe or not, you can't just do that research speculatively, right?

• 14:07Dean Ball?

  You have to actually build the thing to some extent, and put it in a constrained setting to figure out if it's safe. You have to build at least big parts of it. And once you've done that, it's like, well, okay, but there's a ban. So only the specially sanctioned group is allowed to conduct this research. And at that point, you have a monopoly, perhaps a global governmental cartel of some kind that is developing this. And this I also think could potentially be dangerous. And that is of course, assuming that you were able to get the international cooperation, you would need to effect such a ban, which I also doubt. So that would be my, uh, my comprehensive statement.

• 14:45Speaker 04

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:50Liron Shapira?

  Okay, Max, Dean raised a few points about maybe the practical difficulties of doing this kind of superintelligence regulation, even going so, going so far back as to define what superintelligence is for the purpose of this ban. How would you respond to that?

• 15:05Speaker 01

  So I'm afraid that we might disappoint you, Larry, on here by agreeing more than you want because you want the fierce, the paid authorities to clobber each other.

• 15:16Speaker 01

  I think it's actually quite easy to to write this law. And I don't think it requires defining superintelligence at all. Let me explain a little bit what I mean by that. You know, if we treat AI like we treat any other industry that makes powerful tech, we would then have safety standards, right? There are safety standards for restaurants, you know, before they can open, they have to have someone check the kitchen. So if you had safety standards for AI, they wouldn't need to define superintelligence. They would just say that, you know, if there's a system that some plausible experts maybe could cause harm, and here are the things you have to demonstrate to the FDA for AI or whatever, that this is not going to do. You might want to demonstrate that it's not going to teach terrorists how to make bioweapons.

• 16:00Speaker 01

  If it's a very powerful system, you probably put one of the safety standards being you have to demonstrate that you can keep this under control. If the company selling it can't convincingly make the case that this thing is not going to cause the overthrow of the US government, then reject, you know, come back when you can, right? So I didn't mention superintelligence here at all. It's the company's obligation to demonstrate that they need the the standards. And to take an analogy that might help clarify what I'm talking about here, let's talk about thalidomide for a little bit. This was this medicine that was given to women in the US to reduce morning sickness, nausea during pregnancy.

• 16:44Speaker 01

  And it caused over 100,000 American babies to be born without arms or legs, right? So the dumb way to prevent such harm would have been if the FDA had a special rule that we have a ban on medicines that cause babies to be born without arms or legs. What if someone comes out with a new medicine now and the arms and legs are fine, but the baby has no kidneys? Or no brain, you know? That's not the way to go about it. The way you instead go about it is you ask the companies to do a clinical trial and provide quantitative evidence of what are all the different side effects that people might not want? Quantify them, how many percent get each? And then quantify the benefits. You give this to some independent experts who don't have money on the line,

• 17:28Speaker 01

  so they can't work for the companies, for example, who look at the benefits and the harms and they decide, is this a net positive for the American people? And, you know, then they approve it. This is how we do regulations in all other areas. And this is how I think it's quite easy to do also for AI. In summary, you don't define superintelligence. You just define the harms that society is not okay with. Very broadly boils down to demonstrating that the harms are small enough to be acceptable. And then it's the company's job to make all the definitions they want, quantify things, and persuade them, these independent experts. Does that make sense?

• 18:10Liron Shapira?

  Yeah, I'm happy to let you guys cross examine each other pretty freely, and I'll just step in once in a while.

• 18:15Dean Ball?

  Okay, cool. So, so yeah, I mean, basically then, instead of saying we should ban superintelligence, you know, the, what you're, what you're saying instead is we should have a kind of licensing regime, a regulatory regime of some kind that, you know, with respect to frontier AI systems.

• 18:31Speaker 01

  Yeah, very much inspired by how we do it for other tech.

• 18:34Dean Ball?

  Yeah, yeah, yeah. So I'd say a couple of things about that. First of all, most preemptive sort of regulatory regimes that I'm aware of, you know, they don't generally require you to prove, I mean, you, you know, you can't prove a negative, right? So I couldn't, you couldn't prove, the FAA, the Federal Aviation Administration, doesn't require you to prove that your plane won't crash. It requires you to make affirmative statements about, really not the plane itself, really like many subsystems of the plane, right? So like the turbines of this jet engine have XYZ chemistry,

• 19:14Dean Ball?

  which conforms to XYZ technical standard, which, you know, blah, blah, blah, blah, blah, right? And in fact, the way that a lot of times that ends up working is like, there are, there are layers and layers and layers of regulations. So, um, you know, the plane maker has to buy jet engines that are only from people that conform to certain standards. And those standards have to do oftentimes not just with like the object level properties of the, of the component in question, but also of things like. How does information flow through the business, you know, flow through like this company, right? There's all sorts of things like that, right? In other words, if you're like a turbine manufacturer for, if you make turbine blades

• 19:57Dean Ball?

  for jet engines, you are probably subject to implicit and explicit regulations that have to do with like risk management inside of your company and like who is the designated risk officer and all these sorts of things, right? But the point is that you have to make-

• 20:11Speaker 01

  At the end of the day, I can just jump in. I agree with everything you said here. What the companies need to demonstrate in the safety case is the high level thing. The government wants to know how many flight hours on average do you have until a failure? And so on. So it's the companies can solve that whatever way they want, right? It's in the interest of the companies to not use flaky manufacturers, to have good procedures and have people study crack formation, the physics of it and so on. And then they'll switch to another alloy if that works better. It's the same for medicines. The government doesn't come in and micromanage, oh, this chemical is allowed, this ingredient is not allowed. Rather, the company, if they have an effect, some medicine that, you know,

• 20:52Speaker 01

  seems to be pretty good. Suppose there's a new antibiotic that seems really good against bronchitis, you know, it contains lead and aluminum and cyanide and some small doses. People will look in the company and be like, you know, we're having a hard time demonstrating the safety of lead. Maybe this works even without the lead, you know, maybe we can swap out this thing. So all the innovation is driven by capitalism, by market forces to come up with a, the quantitative of risk bounds that they want to meet. Nuclear reactors is a great example because what the law actually says there is the company has to make a real quantitative calculation of what, and demonstrate that the risk of

• 21:32Speaker 01

  a meltdown is less than one in 10,000 years to even get permission to start building it, right? So the company has free rein to come up with whatever reactor design they want and then they will innovate and whoever you first meet though is going to get the big bucks.

• 21:49Dean Ball?

  I think it's considerably more complicated. I mean, in principle, that's, that's true, but it is, practice is considerably more complicated than that because like, you know, there are all sorts of things that there is, there's soft, what's called soft law, right? Which is like guidelines and all these other things that push people in, in certain technological directions and away from others. But, but that's actually not even my point. My point is that- At the end of the day, you have to be able to make like, in order to have a regulatory system like this, you have to be able to make affirmative statements about safety. Yeah. And the problem, I think, the problem I think would be, you know, what are the affirmative statements about safety when you consider that the systems we are talking about are

• 22:31Dean Ball?

  by their very nature extremely general and already, you know, so I'm just as an example, like, obviously, AI systems today are being used in areas that already have regulatory, you know, Yeah. structures like the kind you're describing that affect them. So you have to like, this regulator would either have to be so general and have such a broad projection of authority, or it would have to be really, really narrow. And I kind of doubt that it would end up being really narrow in the context of democratic politics, because the issue that you'll have is like, there's going to be more than just, you know, X risk type issues. So even if

• 23:15Dean Ball?

  you could formulate some statement about, you know, existential risk and like, okay, you have to prove that the model will not do XYZ that demonstrates catastrophic misalignment. Okay, fine. But, um, I would say in practice, you're likely to end up with a situation where, for example, the model cannot result in job loss would be a really good example of this. And then you have like, what, what I, you know, this gets back to an article that I wrote more than a year ago called the political economy of AI regulation, which is to say, because this is so general and because it's going, because the technology is going to, in its positive adoption, not like existential risk, not anything like that, its positive

• 24:00Dean Ball?

  adoption is going to end up challenging many entrenched economic actors and aspects of the status quo. And when those people

• 24:11Dean Ball?

  If a regulatory regime of the kind you are describing exists, then those people are going to be able to use it as a cudgel to prevent technological change that I think we would all agree, well, not all as in all people, but probably the three people on this discussion would probably agree is like, good for the world.

• 24:32Speaker 01

  I will push back in a bit on this idea that it's so hard to get started on this, but I'd love to just give you a chance first to answer just a very simple question. Do you think it's reasonable to have zero safety standards on AI right now? Do you think it feels reasonable that there should be less regulations on superintelligence than sandwiches now in 2025?

• 24:55Dean Ball?

  Um, well, I mean, I certainly think we overregulate sandwiches in the which and just for the listener who doesn't have context, I think what Max is probably referring to is sandwiches served in restaurants. Public health regulations and local public, all sorts of things like this, right? And that's true, like there are probably ways in which we do overregulate those things and probably many other ways in which we don't. I would say that generally speaking, in America, we succeed when we regulate at the level of like, the restaurant that serves the sandwich has many computers in it, probably. It probably uses computers in many different ways, including to get the ham and the bread that brought the sandwich to us. And like those, we don't like regulate those computers with respect

• 25:40Dean Ball?

  to their conveyance of ham to the restaurant, right? Like, we just treat them as general purpose technologies that can do lots of different things.

• 25:49Speaker 01

  Right, but if you go to that sandwich shop and you notice that across the street from it is OpenAI or Anthropic or Google DeepMind or XAI. If they had developed super, if they developed superintelligence this year, which I think is highly unlikely, but suppose they did.

• 26:06Speaker 01

  Then they would be legally allowed to just release it into the, into the world without breaking any law because there are no safety standards they have to meet. Do you feel that that's at all reasonable?

• 26:17Dean Ball?

  Well, so, I mean, I, again, I wouldn't quite say, first of all, yeah, fundamentally, I think like you should be able to develop new technology and release it so long as you're not behaving with reckless disregard for or gross or gross negligence, you know, reckless conduct or gross negligence.

• 26:34Speaker 01

  According to whom?

• 26:36Dean Ball?

  Well, accord, so this is the thing. That actually already is it that it's it's to say it's illegal would imply it's a violation of criminal law, which may or may not be true, but certainly it's a violation of civil law, right? So if release of that system were to result in physical harm, loss of property, death of any person.

• 26:57Speaker 01

  Human extinctions there.

• 26:58Dean Ball?

  Yeah, well, human extinction, sure. But like, I kind of, again, like, I'm sort of skeptical that that's what we're gonna have on day one. That company is subject to common law liability. And there are common law.

• 27:16Speaker 01

  Do them if we're all extinct.

• 27:18Dean Ball?

  So, yes, in the in the tail risk case that we all die, then yes, common law liability does not help you. And in general, it's true, the common law liability is not a great solution for most tail risks, to the extent that the damages incurred out, you sort of dwarf the balance sheet of even the largest companies, right? So it's like, you killed, you create, let's not even say killed all people. Let's say you created a pandemic, or someone made a pandemic with your model, and we've decided that that was reckless misconduct for which OpenAI, the creator of the AI model, bears some form of liability. Well, that's a lawsuit you can bring against them. But if the damages are, you know, $100 trillion or something,

• 28:04Dean Ball?

  then, you know, it's very unlikely that you're going to be able to to recoup that amount of money from OpenAI, even with all the money that they have. You'll bankrupt OpenAI and still be, you know, not fully compensated for the harms that you suffered. So it is true that as a general matter, tail risks are one of the classic examples of where

• 28:28Dean Ball?

  of where government, you know, of where public policy outside of like reactive liability makes sense. So I don't dispute that. Now, I think when it comes to the sort of foreseeable tail risks that AI models might pose, the current ones that people talk about are things like catastrophic cyber and bio. I think there's a lot of things that you can do downstream of that that avoids creating this large scale regulatory regime.

• 28:52Speaker 01

  A lot of people talked about extinction, too, wouldn't you say?

• 28:55Dean Ball?

  I mean, a lot of people do, but there is, there still is not the kind of persuasive evidence for extinction in terms of like, not just like theoretically, but like mechanically, how would that work? I just don't think we've seen that quite the same, nearly the same extent that we have.

• 29:10Speaker 01

  We haven't seen any extinction yet, of course, by definition, otherwise we wouldn't be talking here today. But I mean, there were a lot of very serious speakers who put it on the risk list. So, so I'm just thinking, it's interesting what you mentioned there about pandemic example, because I think it's quite relevant. You know, as you know, it's very controversial right now whether COVID-19 was actually the result of gain-of-function research.

• 29:35Dean Ball?

  Yes.

• 29:36Speaker 01

  Funded by the US government and Peter Daszak or not. But if you just consider that there's some probability P that Peter Daszak and his research group did create it with help from others, then, you know, if someone were to sue them for these millions of deaths that it caused, it would be pretty meaningless because Peter Daszak doesn't have that kind of money. The university where he worked doesn't have that kind of money. And for that reason, the US government has now kind of clamped down on gain-of-function research again and say, no more of this gain-of-function research until we better understand what you're doing. And we also have biosafety labs, level one, level two, level three, level four. So if you're doing something even that seems less scary than what they do or did, you

• 30:19Speaker 01

  know, you have, you have to do it in a special facility, you have to get some pre-approvals. And then contrast that now with, with digital gain-of-function research. We had Sam Altman on a press conference the other week being so excited about building automated AI researchers, right? And ultimately, a lot of people are excited about recursive self-improvement, which is very analogous to biological gain-of-function research. Why should we have regulations on biological gain-of-function research and still be content with having no binding regulations at all on digital gain-of-function research? That makes no sense.

• 30:56Speaker 04

  So, so. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:03Dean Ball?

  I'm trying to answer the, I'm trying to actually go back to the first question you asked me, which has to do with safety standards. But first of all, let me just say, yeah, what you said is completely consonant with with my assertion that tail risks are not typically contemplated very well by the common law liability system. But with that being said,

• 31:23Dean Ball?

  you know, I think like when you look at, for example, actually, you can do essentially automated gain of function research with a nucleic acid language model today, right? You can do, you can basically simulate the evolutionary process that allows for more virulent viruses or whatever else. And we've seen, you know, the early stages of this from, from people like the ARC Institute in California. Those are not like chat GPT style models, but it's the same architecture trained on nucleic acid sequences, right? So like, we know that's a thing. I think as a practical matter, though, the issue that you have is that those things are bits, and it's very, very hard to just purely regulate bits.

• 32:05Dean Ball?

  So what do we do? Well, instead of imposing regulations at the layer of the model, which is a really difficult layer of abstraction on which to do it, in the same way that like, we don't tend to place regulations at the layer of like, computers or of software or of transistors. Because these things are really important general purpose technologies that undoubtedly all three of those things have killed lots of people at this point. We don't regulate at that layer of abstraction because it's not very practical. It's not a good unit of abstraction and not a good conceptual unit of account for regulation. So what do we do? Well, there are all these choke points in the physical world. Some of them are, are, you know, labs of certain biosafety level categories, BSL 3 and 4, as you said.

• 32:50Dean Ball?

  Some of it is at the layer of nucleic acid synthesis screening, which, you know, basically you have to say, well, if you're going to order the creation of a certain kind of nucleic acid, we're going to, as a matter of policy, require that you screen that against some sort of methodology that allows us to test for whether or not you're trying to make a pathogen. And again, I worked on that policy, some of those policies when I was in the Trump administration. So these are all things that that we do. Again, those are safety standards that exist that are emerging, that are kind of downstream in many ways of advancements in AI. I think the urgency of policies like nucleic acid synthesis screening goes up because of AI. So.

• 33:30Dean Ball?

  The issue, so I think when it comes to safety standards for something like, you know, at the very general level of a sort of generalist artificial intelligence model,

• 33:42Dean Ball?

  what you need, I think in the long run, we will build those standards, right? Like, no, I don't, I don't think anybody in the spectrum is saying that there won't be standards for safety, security, and et cetera of large language models.

• 33:54Speaker 01

  What do you mean by long run? Because Sam Altman talked about 1,000 days to superintelligence, and he might be wrong, but I'm curious if you're thinking less than three years or more than three years?

• 34:04Dean Ball?

  I'm thinking that it will happen gradually over the course of the next decade, or maybe even more than that.

• 34:10Speaker 01

  After superintelligence, maybe?

• 34:12Dean Ball?

  After superintelligence, maybe, yes. But I think the broader point here is that this is traditionally the way that we kind of do things in the, in the United States, is that you build a technology, right? You gain experience in practice from, you know, with, with, with its sort of, with its utility, and you sort of diffuse it throughout the economy in this very complicated way. Sometimes there are demonstrated harms, and when there are demonstrated harms, the first thing we do is we deal with that through the liability system. And again, I would point out that OpenAI, Google, and other companies have not copyright cases, but common law liability, you caused physical harm to me, type of liability cases

• 34:56Dean Ball?

  against, against them, for chatbots, right?

• 34:58Speaker 01

  Yeah.

• 34:59Dean Ball?

  And I think that at least some of those, the companies are likely to lose. I mean, you know, they'll be determined by courts. And then gradually, over time, we codify around a set of standards that are shaped by experience, that are broadly agreed to by many different actors. And then eventually, we codify those in the form of government, sort of government standards, and eventually that becomes part of an international standards body, right? Nobody is disagreeing that that is not a process in which we need to invest substantial time and money and energy. And in fact, I would say the Trump administration should get points in your book, because part

• 35:40Dean Ball?

  of the reason that, that, that the administration renamed, it was called the AI Safety Institute by the Biden administration, they renamed it the Center for AI Standards and Innovation to reflect this reality that the ultimate goal of an organization like that is to produce technical standards. So you have to produce these, like, it takes time to do? But when you actually have these things, they are coherent because they're formulated through experience. I think the problem is when you, when you try to change the sequencing of that and try to come up with standards sort of without any experience and sitting in the, you know, the ivory tower or the, you know, the regulators conference room.

• 36:24Dean Ball?

  You, I think, have a tendency to create standards that are unrealistic and burdensome.

• 36:31Speaker 01

  Hmm. I completely agree with you, Dean. It's, it's great that the current administration is taking biosecurity more seriously. And I get a sense that they're also taking AI assisted hacking more seriously. I completely agree with you also that this is how things have been done in the past. Let technologies come up, let people invent the car, a bunch of people die. And then gradually you, you mandate the seatbelt and traffic lights and speed limits and, and other things to make the product more safely. But I think it's important to remember that science has been getting progressively more powerful from the early, from the ancient times until now.

• 37:13Speaker 01

  And as a result, technology also keeps growing exponentially in, in its power, right? So at some point, the technology gets powerful enough that this old traditional strategy of just learning from mistakes goes from being a, a good strategy to being a bad one. I think it worked, it served us well for cars. It served us well for things like fire. We meant it fire first. We didn't regulate it to death. It was later we decided to put fire codes in and have fire extinguishers and fire trucks and stuff like that. I would argue that nuclear weapons are already above this threshold. We don't want to just let everybody who wants to buy hydrogen bombs in supermarkets. And then oopsie, you know, that didn't go so well, but we had it. The nuclear winter now and 99% of Americans starve to death.

• 37:56Speaker 01

  Let's regulate for those things. It was very obvious to people that one mistake was one too many. And we, we already have a bunch of proactive laws about how to deal with hydrogen bombs. In fact, even despite all your work in the government, you are not allowed to buy your own hydrogen bomb, even though I would trust you with it. You know, I know you're a nice guy. I'm not allowed to start doing new plutonium research in my lab at MIT, even if I I pinky promise that I'm gonna be careful, you know. Just because one mistake there is just viewed by society is one too many. And they know I don't have enough cash to pay the liability if I get sued afterwards. And I would argue that artificial superintelligence is vastly more powerful than

• 38:39Speaker 01

  in terms of the downside than than hydrogen bombs would ever be. You know, there have been some pretty careful calculations recently showing that only about in the worst case scenario, ninety nine percent of all Americans would would die and starve to death. And if there's a global nuclear war with with Russia, so there's still, you know, three million have survived, whereas if we if we lose control of artificial superintelligence because we sloppily, somebody sloppily built the new robot species that just kind of took over, you know. Uh huh. It really is game over in a way that nuclear war wouldn't be. So so the way I see this is is not that there's anything wrong with the traditional wisdom for how to regulate things. I think that's very appropriate for all tech sort of below a certain risks threshold.

• 39:24Speaker 01

  And we're very lucky with AI that so many of the great benefits we have are not particularly risky. You know, AlphaFold, absolutely superhuman tool for folding proteins, great for drug discovery. You know, autonomous vehicles, this can save soon, I believe, over a billion lives every year from this pointless road deaths, you know. It's so much productivity that can be gained from building controllable AI tools. And and those, it's, I think, very feasible to continue having the sort of liability system you're describing, the traditional way, you know. Learn from mistakes and then fix. It's only fringe stuff, like in particular, artificial superintelligence, which is on the wrong side, I think, of that of that threshold.

• 40:10Speaker 01

  Where right now, there's not much upside, frankly, to sprint, in my opinion, the sprint to building superintelligence in three years. If we could do it safely in twenty years instead, we would be much better off just doing controllable tools until then. And that's why it irks me so much that I think people conflate these two things. A lot. I'm not saying you do, but a lot of folks I've spoken to on, in, on the hill do, I think, and think that the only choice we have is more AI or less AI or go forward or stop. Whereas I see it instead, the development is being branching into two paths. You know, either we continue going very aggressively forward to build all these great tools,

• 40:51Speaker 01

  but just insisting that companies demonstrate to us that they are controllable tools versus going all in on building, building superintelligence. I have, I have to give you a compliment, Dean, also. I was so pleasantly surprised when I read the action plan that it didn't mention the word superintelligence a single time and not even AGI. You must, and I, I think that was, I don't know if you get a hundred percent of the credit for it or fifty percent, whatever. But I think that was really wise because it highlights that there's so much great stuff we can do with AI tools without having to even get into the whole question of, of, of superintelligence. I don't know if there's anything you're allowed to share with us about.

• 41:34Dean Ball?

  Well, you know, a lot of people contributed to the action plan, but thank you very much. I appreciate, you know. Um, but actually I'd say the reason we didn't use terms like AGI and superintelligence in the action plan, at least from my perspective, is like, because it's really hard to know that we're talking about the same thing.

• 41:50Speaker 01

  Yeah.

• 41:51Dean Ball?

  You know, because, and, and so this is, this is where I think, you know, ma- maybe we ought to, we ought to spend some time is this question of like, what exactly are we, are we talking about when it comes to, um, like, because, because I, I'll give you an example of an area where I've had an evolution in my thinking. About a year and a half, two years ago, you know, I was opposed to this bill called SB ten forty seven, which was a California state bill that had to do with, uh, you know, it was regulating models with respect to potential risks relating to, um, extreme cyber events and bioterrorism and other sorts of bioweapon events

• 42:34Dean Ball?

  that could cause up to more than five hundred million dollars in damage.

• 42:39Speaker 01

  Yeah.

• 42:39Dean Ball?

  And at the time, you know, the frontier model was things like GPT four O. Things like, you know, Gemini 1.5, Claude 3.

• 42:54Dean Ball?

  And it wasn't obvious to me at sitting at that time, you know, say the spring of 2024.

• 43:01Speaker 01

  Yeah.

• 43:01Dean Ball?

  Wasn't obvious to me that just everyone was like, well, the next time we crank the pre-training wheel, the next time we go up another order of magnitude in terms of pre-training compute, we'll get to models that do pose these very serious bio and cyber capabilities. And that wasn't quite clear to me because, you know, I was sort of thinking like, well, I don't know, like,

• 43:23Dean Ball?

  you're talking about cross entropy, you're talking about minimizing cross entropy loss here on the broad, you know, internet corpus, like, is that really gonna like create something that can cause a like bioweapon? And I said something, though, I said, you know, um, if you showed me a model that had demonstrable system two reasoning, and that sort of led to the performance that I think it would, system two being deliberative reflective reasoning, um, then I would change my mind about some of this stuff. And then, uh, you know, right around the time SB 1047 was vetoed, OpenAI released a model called O1, which did exactly this. It had this system two reasoning.

• 44:05Dean Ball?

  And the, uh, performance on cyber, on mathematics and, and on a lot of different areas of science, including biology, went way up. And at that point, I said, shortly after that model, I said, this changes my risk calculus with respect to catastrophic events like bio and cyber. Because it's clear that this reinforcement learning and inference time compute based paradigm is going to rapidly lead to capability increases in some specific areas that we're worried about. And I can paint a really clear picture because I can go from this model can reason about biology. And it can also, by the way, use tools like AlphaFold, right?

• 44:48Dean Ball?

  It also self-use other biology, uh, you know, machine learning tools. I can go, I can draw a very clear picture from that to.

• 44:57Dean Ball?

  You know, a virus being synthesized in a lab somewhere, the virus self-replicates, it infects one human host. Now, there's a lot of complexity there, right? It doesn't mean that that kind of thing is guaranteed to happen. But it means that if you're doing the expected value calculation and you're thinking about like, okay, plausibly, the chances of this maybe causing a pandemic, even if the chances are still low, the chances just went up a big fraction. And so we're going to need some degree of targeted regulation to deal with this topic.

• 45:34Speaker 01

  Yeah.

• 45:34Dean Ball?

  And that is why I was supportive of SB 53, which in many ways was a somewhat more tailored version of SB 1047 that came a year later. That was because, it wasn't just because the law changed to become a little bit more favorable to things I care about. It was also because the facts on the ground changed. So why does this matter? Because there is a clear link between emergent model capabilities and an actual harm that is cognizable to me. I think the issue with the sort of human extinction thing is that it's very hard to demonstrate in concrete ways, like what this looks like. So to this point, like if you could formulate what you wanted,

• 46:18Dean Ball?

  like if you could formulate things that would make you feel better in affirmative, technical, like empirical things we could say about models, right? Like, okay, we've stress tested the model in this way. We've run this eval or we've done this thing and we have shown that the model gets, you know, the model passes what we view as an acceptable threshold. Then I would totally be willing to say like, yeah, let's make that an E. You don't even need to pass a law. Make it an eval. I kind of promise you, if you made that an eval and if you got like enough of the credible people around to like support it, I kind of think that like labs would probably just run it. Like without a law having to pass. So like, why not just do that?

• 46:57Speaker 01

  Yeah. So you raise a number of successful policy approaches in the past here. So let's just, let me just summarize some good things I think you said there and add a little bit to it. You know, we're more broadly with regulation of things. Let's take drugs again, you know?

• 47:16Speaker 01

  To not drown companies in big, in red tape and, and some other innovation. One tends to look at just

• 47:27Speaker 01

  rough plausibility.

• 47:30Speaker 01

  And then divide all the products into classes. We have class one drugs, class two, class three, class four, right? So there are much higher safety standards for fentanyl or other new opioid drugs than there are for new cough medicines for adults, you know? And, uh, or for new vitamins, you know? So if you take the same approach to AI, well, then what you would say is if there's some new software that translates English into Chinese or Japanese. The most embarrassing thing that could probably happen is that, um, it's a sort of repeat of the Monty Python skit with a fake Hungarian dictionary. You've seen that some people get, get a, get a red face.

• 48:12Speaker 01

  Whereas if it's some, if you have an AI that, um, is really state of the art and, and the protein synthesis or, or DNA synthesis, you know?

• 48:25Speaker 01

  It's pretty obvious that that should be subject to higher levels of scrutiny. And then what we do in industry now is we let companies make the safety case rather than the government. So you, you said there that it's hard to foresee exactly how super intelligence, if it's just smarter than all humans combined, would kill us all. But, you know, being a scientist for so many years has made me really humble about these things. It, it would have been really hard also for the makers of thalidomide to predict that that would cause babies to be born without arms or legs. When all that ostensibly did was reduce nausea in their moms, right? We didn't actually understand how that would happen, but it did happen. And, uh, because of that.

• 49:09Speaker 01

  It would have been pretty reasonable to just say, well, okay, you know, we've noticed that there are a lot of things that can cause birth defects. We don't understand exactly how it works. So before we try it on all American mothers with no prescription, you know, let's try it on a small number of mothers and then see what happened to their babies. And then kind of go from there. So you shift the burden of proof away from politicians trying to having to articulate why this is going to be dangerous. And to the companies who just have to do some basic research to make the safety case. Right. And I think, again, if we did this with AI today, if I had a magic wand and we created an FDA for AI, you would have class one, class two, class three, class four, or AI

• 49:54Speaker 01

  safety level one, two, three, four systems, a little bit along the lines of what many AI companies have already in their voluntary commitments. Right. And it would be very, very easy requirements for the ASL one and so on. But when you for the higher level systems, the companies would have to do a lot more to quantify the safety case. And I think what would have happened then is we would end up in a golden age of AI progress, where we would soon get flooded with all sorts of new medical treatments and amazing autonomous vehicles, a great increases in productivity. And the one area that would get slowed down noticeably is precisely the race to build

• 50:36Speaker 01

  actual superintelligence, where I think nobody would be able to make a safety case yet. And I think that would be just just fine, you know. If we have to wait 20 years to think to get that done properly, it's way better than

• 50:53Speaker 01

  racing to it and bungling it and squandering everything.

• 50:57Liron Shapira?

  Liron, look like you want to look like you wanted to jump in. Honestly, you guys are doing such a great job. So I don't know how much value I could add, but just I'll give it a shot to orient the viewers. So yeah, you guys were talking about how to regulate these new AIs. Dean, in your case, like Max pointed out, when I read your AI policy document, you know, the America's AI Action Plan, it doesn't really mention superintelligence. Do you think that is a wise way to go to basically just not look at the possibility of superintelligence currently when making policy? Or do you think we should do anything to prepare for the possibility of superintelligence?

• 51:31Dean Ball?

  Well, so I mean, I should say, you know, the action plan, it is in the, in the sort of AI policy, public world is very heavily associated with me. But, you know, of course, the action plan was written by, by many people within the government. And I was really just the, the sort of, I was, I was, I played a big role in it for sure. But, you know, by no means the only one. It was not my unilateral product, for sure. And I think one thing I would say there is like, um, I think that's, that's part of the reason the action plan doesn't talk about superintelligence is because it's very, it would be very hard to build, whereas like my Substack does, you know, from time to time, talk about superintelligence.

• 52:13Dean Ball?

  Because it's very hard to build consensus among a document that has so many authors as to what we really mean. And this maybe gets into, you know, again, it goes back to where my concerns are with like laws and drafting and exactly what you, what you mean and what you don't mean. I think about a model like GPT-7, this sort of ostensible GPT-7. And I think to myself, man, like, if this is a model that like, advances the frontiers of science in many different domains and solves a lot of different math, you know, math problems that have flummoxed humans in some cases for centuries, is better at sort of legal reasoning and all these other things than, than, than encoding, than any

• 52:57Dean Ball?

  human, uh, doesn't seem inherently dangerous to me. And also seems like, like, how is that not superintelligent? Right? Like, it doesn't, it's not like Bostromian superintelligence, right? Like, it's not like that specific definition. But I guess my view is like, that concept of superintelligence was created quite a long time ago in the grand scheme of things, you know, with, with respect to how fast AI advances. Um, it's not obvious to me. I think, I think that that concept of superintelligence was a really useful way of thinking about advanced AI systems. You know, Bostrom wrote that book, he wrote the book, Superintelligence in 2014, I want to say. Um, like-

• 53:42Dean Ball?

  Like 11 years ago? Yeah, so you know, Dario talks about this sometimes with respect to AGI where he, Dario Amade, CEO of Anthropic for the listener, where he's like, you know, AGI 10 years ago was like, we're driving to Chicago, but then like, once you actually get closer to Chicago, it's like, okay, well, like, what neighborhood are we going to? What street? What's the house number, et cetera, et cetera? And I think that as we get closer, we actually, we need to develop new and more specific abstractions for what we are talking about because there are, you know, all sorts of things that I think we will probably, in the fullness of time, have like really specific kinds of technical standards and also maybe even like statutory requirements for what, for, for

• 54:27Dean Ball?

  like what you can and can't build with AI. So one thing I want to be very clear about is like, I'm not saying like, this needs to be unregulated for all time. In fact, I would say, you know, you made the point earlier about how we regulate different medicines with different levels of, um,

• 54:42Speaker 01

  Rigor.

• 54:43Dean Ball?

  Yeah, rigor, uh, based on, based on their potential risks. I think we already do that with, with the frontier language models, right? Because like,

• 54:52Speaker 01

  There are no binding regulations right now in America for anything.

• 54:56Dean Ball?

  There's tons of binding regulations on frontier AI in America now.

• 55:00Speaker 01

  There is no binding regulation preventing people from launching things.

• 55:04Dean Ball?

  There is no,

• 55:04Speaker 01

  They're liable afterwards, right?

• 55:06Dean Ball?

  It is true.

• 55:06Speaker 01

  As opposed to drugs, right? Where I think that's an interesting distinction, distinction for the listener. You can't release any drug in the US until you've talked to the FDA about it.

• 55:15Dean Ball?

  Well,

• 55:16Speaker 01

  That's off your thing.

• 55:17Dean Ball?

  Not quite, not quite, but you, you, you, actually, this gets into technical definitions and things where these things matter. You can release, for example, you can release CRISPR engineered bacteria without consulting the FDA because those are probiotics, according to the statute. Uh, so, so a company called Lumina released a CRISPR engineered bacteria that you're supposed to, to brush your teeth with that will ostensibly eliminate, um, you're infecting yourself with a bacteria that you'll be infected with for the rest of your life. And every person you ever kiss will also be infected with it. Um, I'm just saying though, that like,

• 55:54Speaker 01

  Yeah.

• 55:55Dean Ball?

  You know, there's a lot of stuff you can-

• 55:56Speaker 01

  A lab in, in Wisconsin that's been taking this bird flu strain that kills 95% of humans, but it's pretty harmless because it's not airborne. And they've been working on trying to make it airborne. Yeah. So there's a room for improvement there, but I think we agree on, on the basic situation here that, that there are, for, you can't open your restaurant or really release a new type of opioid before you've been FDA approved. There are just some, some, in, in, some interesting, um, there may be some, obviously we can have some differences in opinion about things, but there are some things which I think are more in the confusion category, which are just really helpful to, to, uh, to clear up. And one of them is around definitions. Whenever you have any term historically that starts to catch on,

• 56:39Speaker 01

  every hypester is going to try to latch on to it and have it mean something else. Right? So Alan Turing, when he said in 1951 that if we build machines that are way smarter than us, the default outcome is that they take control. And, and Irving J. Good talked about superintelligence machines in, in the sixties and recursive self-improvement. The definition of superintelligence that was implicit in that was obviously that they could do everything way better than us, which meant that they could also do better AI research than we could. They could build their own robot factories, make more robots that didn't need us anymore. And therein lie the risk. After that, I agree with you. Right now, there's just so much hype and BS about this.

• 57:22Speaker 01

  Mark Zuckerberg talked about superintelligence in a way that almost made it sound like it has something to do with Meta's glasses. And, and we have so many different ways that people have redefined AGI from the original definition that I actually, I don't know if you saw the paper that I was involved in that we did with Dan Hendricks and Yashar Banjo and many others called defining superintelligence. And welcome people to- Come up with other actual empirically useful definitions, but we found with this definition that we're absolutely not even at AGI. GPT four was twenty seven percent of the way to AGI. GPT five was fifty seven percent of the way there. So there are still a lot of areas where today's best AI systems really suck. Long term memory, for example, but we're getting closer.

• 58:07Speaker 01

  And I, I think that if we're thinking about only putting the first FDA style safety standards on AI in three or four or five years. It's there's some reasonable chance that that'll happen only after it AGI and maybe even super intelligence have been created, right? And that would be, I think, the pretty big oopsie for, for, for humanity. So I think there is, there are very useful, clear definitions of what we mean. And as I said in the beginning, the, the, the way to write a law is not to define super intelligence and ban it. But I think instead ban the outcomes that you don't want, you know?

• 58:47Speaker 01

  Something overthrowing the US government, something making bioweapons for terrorists, which are very easy to define. And then soon as that law is in place, it's going to spur just massive innovation in the companies. I love comparing pharma companies budgets with AI companies budgets. You know, the leading AI companies now spend maybe one percent, give or take on them on safety. Whereas if you go to Novartis or Pfizer or Moderna, they spend way more than that on, on their clinical trials and the safety. Because that's the financial incentive, right? They're in a race to the top. Whoever can be the first to come out with a new, a new drug that meets the safety standards makes a ton of money.

• 59:30Speaker 01

  People really respect the AI, the safety researchers in those company. They don't think of them as whiners who, who slow down progress. They think of the ones, them as people who help them win the race against the other companies. To make the big bucks, right? So I think as soon as we start treating AI companies like we treat companies in other industries, we will incentivize amazing innovation.

• 59:54Liron Shapira?

  Can I also throw out a question? I want to clarify Max's nightmare scenario here because I think that's important to frame the discussion. Max is, I think you're not even just concerned about something like thalidomide, where a bunch of people die, you know, hundreds of thousands or whatever it was. You're concerned about this runaway process where it just becomes too late to regulate forever. Is that fair to say?

• 1:00:14Speaker 01

  Yeah, I can take a minute just to clarify a little bit for listeners who haven't thought so much about this. Many, many times humanity has been thinking a little too small. Like people thought nuclear weapons were science fiction until they suddenly existed. People thought going to the moon was science fiction until we did it. And from my perspective, as a scientist, as an AI researcher, if you think of the brain as a biological computer, then there's no law of physics saying you can't make computers that are better at all tasks than we are. And a lot of people used to say, yeah, but that sounds so hard. It's probably decades away. In fact, most professors I know thought that even six years ago,

• 1:00:55Speaker 01

  we were probably decades away from making AI that could even master language and basic knowledge at human level. And they were all wrong, it turned out, because we already have it now in systems like ChatGPT and Claude 4.5. So if we consider what would happen if we actually built huge numbers of humanoid robots that were better at us than us at all jobs, including research, including mining, including building robot factories and so on. We would have built something which is not just a new technology like the printing press, but really built a new species because it can rep, these robots can build new robots and robot factories and they don't need us anymore. Could be great. It could mean that we don't have to do the dishes anymore and we're

• 1:01:39Speaker 01

  going to live in abundance and them taking care of us, but it's not guaranteed. And Alan Turing, as I mentioned, the godfather really of our field, said in 51 that the default outcome, he thought was them taking control. We have the two most cited AI professors on the planet, Jeff Hinton and Yoshio Banjo, saying similar things, you know, today. And so if you let go of this idea that AI is like the new internet or whatever, and you just think about it as actually a new species, which is in every way more capable than us, there's absolutely no guarantee that it's going to work out great for us. And I'm not just talking about how we obviously couldn't get a job, get paid for doing work because they could do it all cheaper.

• 1:02:24Speaker 01

  I'm talking even about that we don't really have any say after that necessarily on what happens on the planet. A lot of people are working on this. It's called the control problem or the alignment problem. Broad scientific consensus that they're not solved yet, you know. So this is the scenario which I think will end up. And if we just race as fast as possible to building these super intelligent machines, rather than instead focusing on the controllable tools that can cure cancer and do all the other great stuff, and then taking it nice and slow with the things that we don't get no other control.

• 1:03:05Dean Ball?

  So, okay, there's a lot there that I think I can respond to. First of all, I think I would start just by pointing out that, you know, you correctly observed that lots of people will take terms like super intelligence and redeploy them to mean completely different things. I would submit to you that maybe Sam Altman, when he talks about this being existing in three years is maybe Doing a bit of the same thing.

• 1:03:29Speaker 01

  Maybe.

• 1:03:30Dean Ball?

  And so like you.

• 1:03:30Speaker 01

  Plenty of hype out there.

• 1:03:32Dean Ball?

  You can't talk out of both sides of your mouth. You can't say like, well, this happens, but also these people say they're going to build it tomorrow. You know, you have to like pick one. But the other thing I would say that's like more serious is like.

• 1:03:43Speaker 01

  Although I wasn't talking about. I'm concerned about this regardless of, you know, whether it happens in three years or 10 years.

• 1:03:49Dean Ball?

  Me too.

• 1:03:50Speaker 01

  The key thing is whether, I think right now we're closer to figuring out how to build superintelligence than we are figuring out how to control it.

• 1:03:59Dean Ball?

  Okay.

• 1:03:59Speaker 01

  So I think the only way we fix that is simply by making sure no one is allowed to build it before it can be controlled.

• 1:04:05Dean Ball?

  Okay, so let me just respond now. So you have a, you are basically, the fundamental difference here is that I am saying the technology will be regulated in various, in like a wide variety of different ways, which are fundamentally and mostly reactive, right? Doesn't mean that we won't pass laws. There's already laws that I've supported, which have to do with AI regulation, and I don't think impose substantial burdens on the development. I would also say the development of this technology is a national security priority.

• 1:04:36Speaker 01

  Agreed.

• 1:04:36Dean Ball?

  It seems really hard to, like, it seems like a really big cost to impose something that we would self-consciously, you know, slow ourselves down. When others are not doing that, but I'm not even gonna like, I think that's a valid point, but that's not even where I want to go.

• 1:04:50Speaker 01

  Okay.

• 1:04:51Dean Ball?

  Instead, I feel like our crux is you kind of want this precautionary principle based preemptive regulatory regime that would require some group of people to say affirmatively yes before you're allowed to do something.

• 1:05:04Speaker 01

  Cause I want to build. Like we do it with pharma and every other industry.

• 1:05:08Dean Ball?

  Yeah, which is what those are, which is what those are. Um, I think that, you know,

• 1:05:16Dean Ball?

  it's, it's really hard. I think that there are huge costs associated with a regulatory regime of that kind. I don't think the government could do it very well. I think it's very possible that by doing that in practice, as someone who has observed lots and lots of these regimes play out. I think it's very possible that by doing that in practice, you would actually end up not just being worse for safety, but being worse for or being worse for innovation, but being worse for safety.

• 1:05:43Speaker 01

  Are you arguing we should close the FDA? Or did I miss something?

• 1:05:47Dean Ball?

  I mean, so I would say the FDA is an organization that is in need of deep, deep and profound reform. Because one of the things that happens when you impose a top down regulatory regime like this is you lock in all sorts of assumptions that you have about the world. So let's take the FDA as an example. The FDA.

• 1:06:07Speaker 01

  Before you give this answer, which I was super interested in. It's very different saying the FDA needs to be reformed, less regulatory capture, et cetera, from saying it should be shut down. So are you are you going to give, are you saying that we're worse off having it than we would if we didn't have it at all? Or are you arguing simply for a better FDA?

• 1:06:27Dean Ball?

  Well, let me just explain like where where I'm coming from here, like. Because I I think that like these analogies of FDA to AI are not really very good. And so like, yeah, I it's not to say that I don't think we need something like an FDA. It's not that I don't think we need to test drugs before they go into people.

• 1:06:45Speaker 01

  But then I'm confused by why you don't think we should have the same for AI. So what's the difference between AI?

• 1:06:51Dean Ball?

  Let me, let me, let me make an uninterrupted point for a few minutes. If you don't mind, um, okay. Okay. So first of all, when it comes to the FDA, we have this huge problem right now with a lot of drugs, which is that what we have realized after several decades of modern science. You know, as opposed to like, you know, where the, when the FDA was made, um, a hundred years ago. Is that diseases are way more complicated than we thought they were. They're not really discrete things.

• 1:07:21Dean Ball?

  And like, there's kind of no such thing as cancer and there's kind of no such thing as Alzheimer's. They're much more complicated, sort of broader failures of very complex biological systems and circuits. And the issue that you run into with that is that you need highly personalized treatments for things. Uh, in order to solve stuff, because your cancer is different from my cancer is different from from someone else's cancer. And the FDA's regulatory regime turns out to be entirely unsuited to deal with that because it's depend it's it was based on this sort of industrial era assumption of diseases manifest themselves the same way over many over large populations of people.

• 1:08:06Dean Ball?

  So what you want to do is you want to test it over a big population and sort of get get an average get average statistical results as as opposed to safety results for one person. And what that means is that what we have locked into place is an entire economic structure for the way that we treat disease that is wrong for modern science. And it's really non-obvious how we change it because there's a lot of entrenched interests associated with with the current system, including the people who run the clinical trials that are that we that we operate at great expense, hugely more expensive than they should be.

• 1:08:46Dean Ball?

  Right? So that would be one of a huge number of examples of problems that can manifest themselves with top-down regulatory regimes of this kind. And so the idea that we need such a thing for AI, you know, I'm not necessarily saying that you I'm saying that doing that bears an enormous cost. It carries an enormous cost with it. And I don't think that we really have the evidence that that cost is worth paying with respect to AI compared to the many benefits that we get from not doing that and regulating it the way much more like we have regulated things like the computer and the internet and software and many of

• 1:09:31Dean Ball?

  the other general purpose technologies which have actually worked and grown and like made our lives so much better. I mean, not to say that medicines haven't, but like the things that have been real growth areas. And really, a lot of why medicine has done so well has to do actually with software and computation and the internet and things like this, as opposed to like pure object level advances in biology, which would be why we're going to cure cancer on chips that were originally designed to play video games. So I guess I'm saying like, you need to demonstrate a very, I think there's a high burden of proof, not to say it will never be met, but to say it hasn't been met, to say every single top-down regulatory

• 1:10:13Dean Ball?

  system we have was carried with it, a similarly high burden of proof. And I think that like, if you, if you made a statement, to go back to the ban superintelligence statement, if you made a statement that said like, we need to investigate such and such, or we need to come up with like, what are the guarantees that we want AI labs to be able to make in terms of like empirical evaluations about their models? What are the guarantees we want them to be able to make? I would be, I'd be, I don't know if I'd sign on to it, it would depend on the specifics, but certainly I wouldn't, I would not have had the kind of visceral negative reaction that I did to the ban superintelligence statement.

• 1:10:52Speaker 01

  Cool. So a lot of good stuff in there. Let me pick out three things I like to respond to. One about regulation, one about perceived vagueness, and then one about national security that you brought up. So on the first one on regulation, it sounds like we're actually in agreement that even though you'd like to see reforms on the FDA, you would not like restrictions on biotech to be completely eliminated. You would not allow, want people to be able to do biosafety level four research to make that 95% lethal bird flu airborne, for example, just because it's cool and people can sue them later. But you would, you would like there to be something. And, but whereas for AI, you feel still there

• 1:11:36Speaker 01

  should be nothing to prevent companies from deploying things yet, but maybe, maybe, but you're open to it, maybe in three years, four years, you just want us to think more about it. And whereas I guess my position there is that I think if we don't, if someone releases actual true superintelligence that takes over the world, it's, that's going to be, it's going to be too late to regulate it then. And then, and then on the second point, on vagueness, so this is really important. Many people have said to me, oh yeah, this, this statement that we put out on superintelligence, why doesn't it? Isn't it written much more concretely so you can make a law out of it or something like that? And that was very deliberately because I think if you look historically in the US,

• 1:12:21Speaker 01

  when we've had new laws passed, like for example, a law against child pornography. Now, you you could have pushed back and said, if someone says they're against child pornography, well, how do you that's too vague. What do you mean by how do you define child? Is it under sixteen or under eighteen? And how do you define pornography? That's pretty complicated, right? And and the law, you can't just say, oh, you know, you know it when you see it. But I think that would have been a mistake. You totally can. Because once it started to be a a broad consensus that we need some kind of ban on child pornography, that created the political will for experts to sit down and hash out all these details. And this is something you are very, very good at, right? Looking, how would you actually draft the laws?

• 1:13:06Speaker 01

  And I the the idea with our statement was very analogous. We we see ninety five percent of Americans don't want a race to superintelligence. There are a lot of people are super excited about AI tools and view the idea of losing control of Earth to a new robot species as kind of dystopian, including David Sacks, no less, right? So if we can start getting a public knowledge that most people actually don't want a race, non-controlled race to superintelligence, just like most people want some kind of ban on child pornography, then that can create political will, where brilliant policymakers

• 1:13:48Speaker 01

  like you, you know, sit down and talk to all the stakeholders and come up with carefully crafted language for how this would actually work and if there should be a new agency. So this was, in in summary, not something I view as a bug, the vagueness, but it was, I think, a feature. Where where we're going for here is just some moral leadership, basically, that, you know, we would like there to be. Some kind of restrictions on a race to superintelligence.

• 1:14:17Liron Shapira?

  Let me jump in for a sec because I think the I think that you guys may actually dovetail more on policy itself than it's sounding like. Maybe the real crux of disagreement is your mainline scenario of what things would look like if we just kind of went on cruise control and didn't do much, much more than we've already done in the way of policy. And so, Dean, let me ask you this question. What is your P doom defined as just kind of letting AI play out, not layering on additional regulation, waiting 10 years, and then the probability that we it goes wrong and we kind of get this runaway superintelligence that's now too late to control or regulate? What's your P doom?

• 1:14:55Dean Ball?

  Doom being defined as human extinction?

• 1:14:59Liron Shapira?

  Yeah, like it's it's like a catastrophe, you know, of extinction scale. So maybe like a half the human population dies and then we go back to being cavemen or just something extremely catastrophic or even extinction.

• 1:15:09Speaker 01

  Would a permanent 1984 also count?

• 1:15:12Dean Ball?

  I mean, like if if what we're talking about is like AI systems taking control over the world and killing large numbers of people. Yeah, I mean, like my my my P doom is is very low. It's it's sub 1%. It's 0.01% or something like that. It's very low. Um, let's say that I think there's all sorts of other outcomes from AI that seem that seem very bad, that are seem way more plausible to me, that are things that I I work on, you know, a lot. But the specific doom scenario just doesn't really like, it just doesn't really seem all that likely when you think about many different things. One area would be like, look, I think that if you pass the law that said, a group of people will have

• 1:15:55Dean Ball?

  to take a straight up or down vote, and their results, the vote will be public, you know, a group of seven, the Supreme Court, let's just say, we send it to the Supreme Court of the United States. And the Supreme Court has to look at every frontier language model release. And they have to take a take a vote. On do we think this model is likely to take over the world? I would be unconcerned about that. I don't, I wouldn't really support that law for a lot of different reasons, but

• 1:16:26Speaker 04

  Me neither.

• 1:16:26Dean Ball?

  I would not be, I would not be particularly concerned about that if that were literally the law. The problem is that it's not. And I think this is where you get back to like, why, you know, the super intelligence ban statement was, um, it was written in kind of the way that it was, which is that like, a lot of people that sign that statement, I would predict, have a much more nebulous set of concerns about AI than the sort of very specific ones, Max, that you have. And it's, let's say, I'm, by the way, I'm not saying that you don't have other concerns, right? I'm not saying you're not worried about misinformation or deep fakes or, or job loss or whatever else. But I think that like, when it comes to like, where you actually think, cause, cause I think we would both agree that like, the job loss

• 1:17:09Dean Ball?

  thing is really complicated, right? Cause there might be like, AI might, you know, Matt Walsh had a tweet a couple of days ago about, um, the conservative influencer, Matt Walsh had a tweet about, um, you know, AI is going to cause like 10 million jobs, 25, I forget what it was. Maybe he said like 5 million jobs over the next 10 years or something like that. And I was like, you know, that's like an extremely optimistic scenario, right? Like, like in the grand scheme of things, like, cause, cause eliminating 5 million jobs, like, if you just focus on what gets eliminated, sure, it's not that much, but like, you assume some jobs are created, millions, you know, the economy creates, creates millions of jobs and destroys millions of jobs every year, right? Like, that's like, that would

• 1:17:50Dean Ball?

  be a very slow rate of change. Um, actually, if it only created, if it only destroyed 5 million jobs over the next 10 years, that would be like low compared to like, literally like, just, just, just the normal churning of the economy, right? Um, so I think, uh,

• 1:18:10Dean Ball?

  Like, the job thing is certainly complicated, but you can imagine a regulatory regime that was much more like, "Well, we have to check what the socioeconomic impact of this is going to be. We have to make sure it doesn't harm blah, blah, blah. We have to make sure it doesn't..." You know, all these other things, which are not to say they're on serious issues, but that like, can you, can you, can we agree? Maybe that, like, there's a plausible version of GPT-7 that's really good and pro-social, that might also displace a lot of jobs. And that if we had a regulatory regime that said, "We need to take a vote on whether this will take over the world", it would be zero to nine in favor of this is not going to take over the world.

• 1:18:55Dean Ball?

  But if we had a regulatory regime that was staffed with like, you know, union representatives and whatever else, like, you know, various stakeholders, let's say, you know, representative stakeholders, that that group of people, if their task was, "Do you think that this will be good for the economy? Do you think it could create job loss? Do you think it could be dangerous more nebulously?"

• 1:19:21Dean Ball?

  That that group of people might vote against the release of that thing and that that might actually end up being a bad outcome. Do you see that failure mode? Like, do you, do you, do you believe that that failure mode is a real one?

• 1:19:33Speaker 01

  I, I totally see things like this getting very political. Yeah, it was very interesting for me because I spent a lot of time, you know, talking to a lot of the people who signed initial signatories of this, of this statement. And of course, the ones who did sign, I was very interested to hear why. And there were indeed, as you say, many different reasons. The NatSec people, like former head of Joint Chiefs of Staff Mike Mullen, for example, I think for him, control loss was very central because he views that as a national security threat, of course. Regardless of whether the US loses control, the US government gets overthrown by a foreign power or by superintelligence, it's a NatSec threat. On the other hand, there were people, you know, from Steve Bannon to Bernie Sanders, who, who felt that

• 1:20:17Speaker 01

  if we end up in a system where we actually have superintelligence that by definition makes all humans economically obsolete, then American workers would be dependent on handouts, either from the government, the former Senator UBI, which the conservatives who signed often view as socialism and don't like, or if it's coming from the companies, the handout. Sam Altman's world coin or whatever would be viewed by people like Bernie Sanders as incredibly dystopian, you know, the most massive power concentration in in human history to some tiny clique of of people from San Francisco who don't necessarily share the moral values even. So that that thing, I think you're right, the is something

• 1:21:02Speaker 01

  that bothers a lot of people. And then we had a lot of faith leaders who signed this for for fairly different reasons, where they just felt this is really gonna be harmful for human dignity. And, you know, a lot of people we both know in San Francisco like the joke about super intelligence being the sand god and so on. And a lot of these people are like, wait a minute, I already believe in a god. Why should I support some atheists in San Francisco, building a new one that somehow run the show, you know, that sounds very undignified. So the people have many different reasons. But I think in in in short, there are two separate questions. One is, should there be any kind of safety standards like we have for biotech or restaurants? And the second question, which is much harder,

• 1:21:47Speaker 01

  is what exactly should be on the list? I'll be very happy if we could start by just having one very light requirement, which is companies have to them is make a good quantitative case that it's not going to overthrow the US government before we we launch it. And then and then we can have a broader political discussion.

• 1:22:02Liron Shapira?

  Before Dean responds, Max, speaking of, you know, reasons to sign the statement and these nightmare scenarios, what is your P doom?

• 1:22:11Speaker 01

  Oh, yeah. We actually wrote a paper, me and the three grad students from MIT, where we took the most popular approach for how humans can control super intelligence, known as recursive recursive scalable oversight. And we got very nerdy on it and tried to calculate the probability that the control fails. And we found in our most optimistic scenario that it fails 92% of the time. And I would love if people who think they have a better idea for controlling super intelligence were to actually just publish it openly so

• 1:22:50Speaker 01

  that they can get subject to scrutiny from from from others. But until that time, yeah, I would put my. If we have, if, if we go ahead and, and continue having nothing like the FDA for AIs, so people can legally just launch superintelligence and worry about getting sued. Yeah, I would think it's definitely over 90% that we, we lose control over this.

• 1:23:14Liron Shapira?

  Wow, 0.1% versus 90%.

• 1:23:18Liron Shapira?

  It's a pretty big crux.

• 1:23:19Speaker 01

  I agree.

• 1:23:21Dean Ball?

  I just kind of have this sneaking suspicion that like, if the models seemed like they were going to pose the risk of overthrowing the US government or anything in that vicinity that like, I don't think OpenAI would release that model or Anthropic or Meta or XAI or Google. Like, I just don't think they would.

• 1:23:41Dean Ball?

  You know, I, I think they like probably wouldn't, wouldn't do that and would be like quite, quite concerned and would probably like call the US government and tell them like, you know, I just, it doesn't seem like a realistic scenario to me. Let me say another thing.

• 1:23:52Speaker 01

  I disagree there. I think, I think the makers of thalidomide would, that company, and I don't want to shame any particular company, they would not have released that either if they had known that it was going to cause 100,000 American babies to be born without arms or legs, but it was complicated and they, they just didn't realize it. And similarly, very complicated here for these people at these companies to know. And Dario Amore has himself talked about, you know, 15 to 25%. As Sam Altman has also talked about, it could be lights out for everybody. So they're clearly comfortable with 5% or 10%.

• 1:24:26Dean Ball?

  You can't, you can't deny the possibility that, that something, you know, you can't prove, I, I can't prove a negative. That's why like you can't say it's zero, right? If you're being intellectually honest, you can't say it's zero. Here's the thing. I'd say, I'd say there's like two observations I would have about this. The first is that, you know, some of what you are describing, some of the negative effects you are describing are going to be, including by the way, like a lot of the labor market stuff. A lot of that is going to be an emergent outcome of a general purpose technology interacting with society. That is very hard to model and advance in the way that you could not not impossible.

• 1:25:09Dean Ball?

  It's just very hard to model in the way that you're sort of describing with drug testing. Right. And so I think these emergent, these emergent phenomena are just going to be like, I think it's really easy when you're thinking about stuff like labor. And this is why a preemptive regulatory regime scares me. If a group of people are sitting around and thinking, what are the potential risks of this thing? A, you tend to overstate them. And B, the reason you tend to overstate them, and I, I, I hear this all the time. When people talk about the impact of AI on society, they don't. An economist would say that they don't endogenize the, the impact, which is to say they model AI

• 1:25:53Dean Ball?

  as like an exogenous shock, like a meteor that's coming to society that's going to hit us. And that we will just remain completely in place and be like, oh my God, there's a meteor and not do anything. The reality is that like if I, if five years ago, I showed you all the generative AI tools that exist today, just without anything about the society. I just said, this exists in 2025. You would be like, wow, I bet you blah, blah, blah, blah, blah. And you would say a bunch of stuff that would probably, and I would too, say a bunch of stuff that would probably be wrong about the downsides of the technology as manifest. You would guess like, my God, their elections must be completely, you know, over their, their media environment, you know, their, their this and that.

• 1:26:37Dean Ball?

  And in reality, they, you know, they, there must be huge labor market dislocation. There must be no software engineers, right? There must be none.

• 1:26:45Dean Ball?

  In reality, society is an adaptive, complex system itself that, just like the human body, has the ability to internalize many different things and is, is, you know, incredibly adaptive. And humans are quite ingenious. And so I think that's an easy thing to discount. The other point I want to make is about this recursive self-improvement thing, because this is another thing that like gets to me sometimes. Every general purpose technology in human history exhibits what you would call recursive self-improvement in the context of AI.

• 1:27:17Speaker 01

  But with a human in the loop.

• 1:27:21Dean Ball?

  I mean, like, kind of, I don't know, like, like, we use, all I mean by that is like, you know, you, you come into the Iron Age and then you use iron to make more iron and better iron and, you know, like, we use, we, we, we turn a mill that's,

• 1:27:36Speaker 01

  Yeah.

• 1:27:37Dean Ball?

  that's got an iron hammer that you're using to, to, you know, to, to manufacture more iron and

• 1:27:43Speaker 01

  But there's still people.

• 1:27:43Dean Ball?

  You use computers, you use computers to make better computers and you use blah to, you know, this is, this is a, you use oil to get more oil, electricity to get more electricity, right?

• 1:27:53Speaker 01

  Yeah.

• 1:27:54Dean Ball?

  Every general purpose technology exhibits these kinds of like recursive loops. Because it's a general purpose technology. So one of the things that the general purpose technology does is make the general purpose technology better. It's very common throughout the history of technology. So I think that you can't just cite this fact that AI is likely to have recursive loops of self-improvement and be, the AI will be useful for AI because every general purpose technology is like that.

• 1:28:23Speaker 01

  Yeah, we don't disagree on anything here.

• 1:28:26Dean Ball?

  Yeah.

• 1:28:27Speaker 01

  So, you know, I wrote about this even.

• 1:28:30Dean Ball?

  But, but there's a reason that like, in every, in the case of every other general purpose technology, that the recursive feedback loop, you know, it tends to be autocatalytic. Oftentimes, it produces nonlinear improvement.

• 1:28:46Dean Ball?

  And, but it never results in these like runaway, you know, these, these runaway processes, right? We've never seen that. We, it's not like we went,

• 1:28:57Dean Ball?

  we used oil to get,

• 1:28:58Speaker 01

  Yeah.

• 1:28:59Dean Ball?

  you know, we use energy to get more energy and then all of a sudden the energy, you know, we, we blew up the entire universe, right? That didn't happen.

• 1:29:05Liron Shapira?

  Okay, Max, maybe you can explain why you still think there's a doom scenario despite Dean's point. And then I've got one more question for Dean.

• 1:29:12Speaker 01

  Yeah, and I would love to comment on the NatSec angle, which I think is super important also because it's the main reason given in Washington always why we should not regulate. Yeah, so I completely agree, of course, that there is technological progress itself has always involved these self-improvement loops. I have written about that extensively and that is fundamentally why GDP has grown exponentially over time because we use today's technology to build tomorrow's technology and so on. But there have always been humans in the loop.

• 1:29:42Speaker 01

  When there are not humans in the loop, things can go quite fast. If you look at a slow motion of a nuclear bomb exploding, there is no human in the loop. You get one uranium atom decaying and then you get now you get two and a four and an eight and so on. And we have never, the reason we've never seen anything blow up fast with our technology more wholesale is because we've, we've always had humans in the loop as a moderator, right?

• 1:30:07Dean Ball?

  Uh huh.

• 1:30:08Speaker 01

  It's pretty obvious unless you think that there's something, some secret sauce in human brains that you can't build into machines, that it is possible to build machines that really don't need us. Now, if those machines think a hundred times faster than us and if they can instantly copy all the knowledge that other robots have learned into their own selves, et cetera. No, there's, then we could see more progresses in a month than we had seen in a thousand years when there were human in the loop. This is not my idea at all, all right? Irving J. Good articulated this very nicely in the sixties. I think this is just very, pretty basic, simple argument. We can't say, oh, it never happened before, so it won't happen again in the future.

• 1:30:52Speaker 01

  Because we've never built superintelligence before. All the other tech, like the industrial revolution, just replaced some aspects of human work, like our muscles, we made machines stronger and we made machines faster. We've never had machines that could entirely replace our cognitive abilities. So can I just comment on that, Seth, a little bit here? Cause I, I think when I talk to politicians on the hill, especially when I listen to AI lobbyists, of whom there are now more in Washington DC than there are pharma lobbyists and fossil fuel lobbyists combined. The main talking point they use to explain why we should not have any binding regulations is, in one word, China. But China, they say, right? If we don't race to superintelligence, China is going to do it first.

• 1:31:37Speaker 01

  And I think that is just complete baloney, you know? I think there is not one race against China. There's two separate races which people really need to stop conflating with each other. One is a race for dominance, which was very eloquently articulated, Dean, to your credit, in the AI action plan. A race for dominance economically, technologically, militarily.

• 1:32:03Speaker 01

  And the way to win that kind of race is by building controllable tools, which I'm all for, right? And yes, you need big data centers and all the other stuff to do these powerful tools we can control. Then there's a second race, which is who can be the first to release superintelligence that they don't yet know how to control. And that's the one which I've been arguing is a suicide race because we're closer to figuring out how to build that. Then we are figuring out how to control it. And the Chinese Communist Party, Xi Jinping, as well, clearly really like control. And I think it's quite obvious that they would never permit the Chinese company

• 1:32:47Speaker 01

  to build technology if there were some significant superintelligence to just overthrow them and take over China. I even had a first-hand anecdote on that from Elon Musk. He told me in the spring of 2023, he had a meeting with some quite high up people in the CCP. Where he said to them, look, if someone in China builds superintelligence, after that, China is not going to be run by the CCP, it's going to be run by the superintelligence. And the reaction in the room was hilarious, Elon said, like a lot of long faces, like like they really hadn't thought that through. And within a month or so after that, China rolls out their first ever AI regulations.

• 1:33:33Speaker 01

  I'm also quite confident that the Chinese have much more surveillance on DeepSeek and their other AI companies than the US government has on our companies. And have both the ability and the willingness to stop something that they think would be a could cause them to lose control. So I think this is actually something which, the way I see this going, and when I said P doom of over 90%, that was Liron, to be clear, if we don't, if we just don't do any regulation, right? I'm actually quite optimistic that things are going to go well instead because I think

• 1:34:10Speaker 01

  there's no way China is going to allow a race to superintelligence. This is, we don't know how to control it right now. I think China is going to continue steaming ahead, trying to build all these powerful AI tools. For the race that Dean was describing in the AI action plan. But but absolutely not let anyone build superintelligence. And I think that's what's gonna happen in the US also. I think the I even know already is the growing number of people in US NatSec are beginning to view this as a NatSec threat. Maybe they listen to Dario Amodey talk about a country of geniuses and a data center in 2027, and they're like, wait a minute. I have here a list of countries that I'm keeping track of as NatSec threats. Did Dario just say the word country?

• 1:34:53Speaker 01

  No, maybe I should add that country to my my watch list also. And and and then certainly we end up in this really great situation where the US will also prevent anyone from building stuff that they don't know how to control. And we'll have this race to who can build the best, the most powerful, helpful tools. And that's the future I'm really excited about living in.

• 1:35:19Liron Shapira?

  Okay, Dean, let me just ask you the last question, and then you guys can make your closing statements. So Max has laid out his nightmare scenario, which is we don't regulate AI, and then it can go uncontrollable. It will have recursive self improvement or just more power than the human species, and it just doesn't go well. He's even said the probability could be up to 90% if we let this happen.

• 1:35:42Speaker 01

  At least 90%.

• 1:35:43Liron Shapira?

  At least 90%, okay. And then Dean, you see that as a very low probability scenario, but you do have your own nightmare scenarios, which are on the side of regulating AI too much, right? If I understand correctly, your two nightmare scenarios are losing the AI race because your AI action plan focuses so much on winning the AI race. And you have another nightmare scenario, which is that some kind of overregulation could lead to like a tyranny situation, right? Just undermine democratic governance. So explain your nightmare scenarios.

• 1:36:13Dean Ball?

  Well, I mean, certainly, I think all manner of tyranny is possible

• 1:36:19Dean Ball?

  with with AI. And I worry quite about quite a bit about that. I think I'd say fundamentally that what I worry about is, um, so so I'll start by saying that like. We are going to, AI is going to challenge the structure of like the nation state, kind of no matter what, in the good scenario and the bad scenario. It challenges that in various ways and it requires institutional evolution, conceivably revolution in certain places in the world. And so like, you know, buckle up because that's coming no matter what. But there's a version of that institutional evolution that basically looks like what we get is a, we get a rentier state. We get a state that is run, think of the Middle Ages, right?

• 1:37:03Dean Ball?

  We get a state that is run by a small number of people that control something. And, uh, you know, that thing is a, it's, it's certainly a tool of violence, but they're not like quite legitimate in the way that we think of democratic legitimacy. And there's some sort of middle class of rent seeking humans who have legal protections from that upper class. And then there is this large underclass of people that, uh, have very low practical agency, very low ability to really, um, meaningfully contribute and they're, they're kind of, you know, stuck in some sort of dystopia. That seems very likely to me. And I think there are many regulatory regimes that, including a licensing one,

• 1:37:47Dean Ball?

  that make that outcome substantially likelier. And so, uh, um, I think that's the, you know, we, we face these kinds of trade offs no matter what.

• 1:37:55Liron Shapira?

  Should we hit on losing the AI race? Or should we hit on like a tyranny scenario? Or, or do you think what you described is kind of the main nightmare scenario?

• 1:38:02Dean Ball?

  I mean, basically, yeah, what I certainly like losing, I don't really even know what losing the AI race to China means like it's hard to know. Like certainly, yeah, there's a world in which China becomes the dominant technological power and sets the standards and all that stuff. And that's a really bad world too, in many ways. But it, I'd say that's like a bad scenario. It's not my, my nightmare scenario. It's not like the, the worst possible thing I think that could happen, but it's definitely not a good thing.

• 1:38:28Liron Shapira?

  Okay. All right. We've covered a lot. So let's go to closing statements, starting with Max.

• 1:38:34Speaker 01

  All right, so we've talked a lot about doom here. And you've kept nudging us in that direction later on, because your brand here, the doom debates. But I'd like to end on an optimistic note. You know, the real reason I'm so engaged with this topic is because I'm fundamentally quite optimistic person. You know, I spent so much time playing with this guy and I'm very excited about the potential for him to have an amazing future where he doesn't have to worry about dying of cancer.

• 1:39:06Speaker 01

  We can prosper like never before on earth and not just for an election cycle, but life could prosper in principle for billions of years that even spread out into the cosmos. It's, we've completely underestimated as a species, right? How much opportunity we have? And that's why I'm, I think it's so important that we don't squander all this greatness by making some hasty chess moves here and blowing it all. I think that there are two very clearly different paths that we're choosing between right now. And we have to make our mind up, I think, within the next year or two, probably. One of them is the pro-human future. America was founded to be a country run by the people for the people.

• 1:39:49Speaker 01

  So there was a real emphasis that America was supposed to be good for the people living in America. It was not founded to be good for the machines of America. That was not the idea here. And then the other, so the way to get there is to stop the corporate welfare towards AI companies, which is hard because they have so many lobbyists. But hey, so did the other industries that we ended up putting safety standards on eventually. And then spear technology to really be pro-human, to make life better for humans, cure diseases, make us more productive, et cetera, but make sure it's always us in charge. So that first one is a very pro-tech scenario. Notice we go full steam ahead with ever better AI tools.

• 1:40:30Speaker 01

  The other scenario is we race to build superintelligence, which by its very definition is super dystopian, right? The very definition of this, none of you can earn any money after that's been built, doing anything. Right? So you're going to be dependent on handouts from the government or some tech CEO, or you're gonna... Not have any money and life will be very bad. That to me is an incredible, the unambitious vision for the future. Why should we, after hundreds of thousands of years as a species on this planet, working so hard to build ever better technologies that we can become finally the captains of our own ship?

• 1:41:11Speaker 01

  So we don't have to be worried about getting eaten by a tiger or starving to death. Why should we throw away all this empowerment we've gained through all this hard work? By just deliberately building something that's going to take over from us? You know, ridiculously unambitious. I want us to take charge of this. And I think 95% of all Americans in these polls clearly agree with this. Deliberately say, okay, we're in charge now. Let's keep it that way. A journalist asked me, what on earth do these different people who signed the statement have in common? I don't even understand, she said. I thought about it for a while. What does Steve Bannon have in common with faith leaders?

• 1:41:53Speaker 01

  And Susan Rice and Chinese researchers, et cetera. Well, it hit me, they're all human. So of course they want the pro-human future where humans are in charge, right? If we found out that there was an alien invasion fleet heading towards Earth, obviously we would all work together to fight the aliens to make sure it's us in charge. And now you have a quite small fringe group from Silicon Valley with very good lobbyists, basically saying, yeah, we should build that on these aliens and they're probably going to take over. Elon even said that openly the other week, right? That is the most unambitious ending to this beautiful journey of empowerment I can imagine.

• 1:42:36Speaker 01

  And to me, the inspiring future that I'm excited about and that I think we will actually have, once people understand more what this is all about, is where we remain in charge and we keep AI a tool and create a future that's even cooler than the sci-fi authors could imagine.

• 1:42:55Liron Shapira?

  All right, great. Let's go to Dean.

• 1:42:57Dean Ball?

  I think the fundamental thing to think about here is really assumptions. Most. AI Doom debates, to use the name of the podcast, result revolve around one of, at least one of the interlocutors, usually the one who believes in Doom, assuming their conclusions. And here, you know, we've had a lot of conversation in this discussion about how superintelligence has many different definitions, and it could mean many different things, and we don't quite know what it means. And there's one version of it that you can articulate that means all sorts of bad things.

• 1:43:34Dean Ball?

  And doesn't mean that thing is likely to be built, doesn't mean that thing is going to be built, doesn't mean that thing is possible to be built in quite the way that we imagine. It just means that it's a thing you can say, you know, it is a valid language, valid sentence in the English language. You are, and then it takes a, it takes a big leap to assume that that's what we're actually going to build.

• 1:43:58Dean Ball?

  And I think we shouldn't assume that. I think that, as Max said at the end of his statement,

• 1:44:05Dean Ball?

  the future is often profoundly stranger than we can possibly imagine. The, the future that we live in today would have been unbelievably alien to someone 50 or 100, and certainly 200 years ago. In many ways, would have been incredibly alien. And many of the jobs we do would have seemed quite odd. And the relationships we have with one another and our institutions, and all of it would be like deeply alien. And my guess is that that continues. And my guess is that the things that we assume today about the technology of the future are probably wrong. And we don't want to embed too many of those assumptions into the law, into regulation.

• 1:44:48Dean Ball?

  We want to maintain, I think, right now, more than any other time, given the speed with which AI evolves, we want to maintain adaptability and flexibility. So I just wouldn't assume that superintelligence means the bad thing. I would instead at least consider that there are many worlds in which humans can thrive amid things that are better than them at various, you know, various kinds of intellectual tasks. And that humans can still have a role, because there are certain things that are not replaceable inherently by machines.

• 1:45:24Dean Ball?

  And that we can gain a tremendous amount of wealth, live much better lives and find all sorts of new things to do that are economically and practically useful. That's been true so far throughout human history and it wouldn't have seemed that way. It didn't to people at the time. Since we have a written record, we know what people's reaction has been to new technology and it's always been like this. And you can say this time is different and that's fine, but I think that you have to, I think we should demand a higher standard of evidence. Max talked about how, you know, America is by the people, you know, and for the people. True.

• 1:46:09Dean Ball?

  But we also have a system that makes it quite hard to pass new laws. And there's a reason that we made it quite hard to pass new laws, which is that our founding fathers, and this is not like a statement of opinion. This is a statement of fact about American history. Our founding fathers were deeply distrustful of democratic impulse, raw democratic impulse. The word democracy was a pejorative from the people that wrote our constitution. It was an insult to say that that proposal seemed too democratic because they believed that you had to balance raw democratic will, people's raw intuitions about things.

• 1:46:52Dean Ball?

  You had to balance that with deliberative bodies that make it hard to pass laws because laws ultimately are rules passed by people that have the monopoly on legitimate violence. And that's a very sacred and important thing. And we don't want to just give them willy nilly all these new powers. We've done that a lot and I just have very serious issues with the idea logically and and sort of at a more philosophical and even moral level with the idea that like, we're just going to be able to pass a new regulatory regime and everything's going to go fine and there will be no side effects. And, you know, I think that there will be tons of side effects and I think that we

• 1:47:34Dean Ball?

  will ban tons of technological progress and stave off a lot of great, you know, a lot of wonderful possibilities for the future. I would say. There are many ways to investigate and interrogate the concept of superintelligence and to advance, you know, the, the safety and controllability of that thing. There are many ways to do it that don't involve banning it, which was the original topic of this debate. There are, I, I note that Max did not spend that much time defending the thing that actually was the subject of the set of the statement that FLI put out.

• 1:48:12Dean Ball?

  But, um, uh, you know, this sort of, I think you can build something much better than just a regulatory regime. You can build a society that is capable of grappling with this technology and institutions that are capable of evolving with it. And I think that's ultimately going to be a much healthier, better outcome for the world. That's the one that I work on every single day. It involves taking the risks seriously. It involves taking the technology very seriously. You should not also be, you shouldn't be a radical in either direction when it comes to this technology. You should be willing to update your beliefs frequently. But, um, at the same time, details matter.

• 1:48:54Dean Ball?

  Getting this right is not going to be, it's not going to be a matter of taking regulatory concepts that we've developed for other things from off the shelf and applying them to this. It's going to be much more difficult than that. And so, uh, I guess I'll close with that.

• 1:49:08Liron Shapira?

  Thank you. I'm thankful to both of you for stepping up to debate the difficult policy questions around super intelligent AI. It's such a complex issue and so there's so many different positions. It's not black and white. It won't work to do it in an echo chamber. It won't work to reduce AI policy to left versus right politics. Respectful debates between smart people with different views is what we need right now as a country, as a species. That's how we can stress test different ideas and bring out important nuance. I'd go so far as to say debate is a key piece of social infrastructure. So thank you again, Max and Dean. Thank you, Dean, for a really great conversation.

• 1:49:48Dean Ball?

  Thanks to you, Max. Thanks to you, Jerome. This was great.

• 1:49:54Liron Shapira?

  Wow, what an illuminating debate from two people who are actually in the room for these kinds of policy discussions. So regarding America's AI Action Plan, the document that Dean Ball helped draft, both Max and Dean were happy that it doesn't mention superintelligence, but for very different reasons. Max was saying, "We need a whole nother statement about superintelligence," and he even proposed a statement saying it should be banned "until there's broad scientific consensus that it will be done safely and controllably, and until there's strong public buy-in." That's what Max thinks we should do regarding a superintelligence statement. And Dean is saying, "Yeah, it's good that we didn't mention superintelligence because it's too vague right now." Dean is saying, "From our current perspective today, we don't know what superintelligence will look like. Maybe it'll just work

• 1:50:38Liron Shapira?

  out really great and doesn't need that much regulation." So it's a very diametrically opposed perspective, pushing to ban it until there's consensus versus, "Ah, well, you know, we'll deal with it later. Like, it's fine for now." The crux of disagreement between Max and Dean, as we uncovered during the debate, it really does come down to their P Doom. If you remember, Max was saying his P Doom is greater than 90%, assuming that we don't have these tough regulations on AI. But Dean's P Doom is only about 0.1%. It's a much, much lower P Doom. Dean is basically not worried about plowing forward and dealing with issues as we get to it, whereas Max says we better be preemptive because it might be too late to regulate if we don't start right now.

• 1:51:22Liron Shapira?

  The reason I say P Doom is the crux of their disagreement is because that's the thing that I think would really change their mind about the other stuff. I think their policy recommendations are totally downstream of what they see as the probability of doom. So for example, if they were to meet halfway, if Dean were to go up from 0.1% to 25%, and Max were to go down from 90% to 25%, or just anywhere in that middle range, 40%, whatever. They'd start coming up with very similar policy ideas. I think in that case, Dean would naturally say, okay, well, we need very tight security on this development. You can't just retroactively do it because there is a high risk of total destruction or of runaway AI. And they would just naturally be thinking along the same lines. It's just going to be downstream of how much doom they expect.

• 1:52:06Liron Shapira?

  So they went on to talk about the FDA analogy because they said, what does good regulation look like? And Max was saying, isn't the FDA a success story? Don't you like the FDA? Dean's response was, he didn't go full libertarian. He didn't be like, oh, the FDA is evil. We shouldn't regulate, things like that. He said, yeah, the FDA is helpful, but you can see it has this baggage. It has this legacy idea that one medicine has to treat one disease. And really, science is a lot more complex than that. So even the FDA is kind of putting the straitjacket on where you have to jump through all these hoops, but it's just, you're not necessarily getting a lot of productivity. Like, you're paying a high cost in terms of drag. And so Dean's analogy is that when we get to general artificial intelligence or artificial superintelligence, this kind of straitjacket regulation could be an extreme case of why

• 1:52:49Liron Shapira?

  Dean doesn't like the FDA, even though he admits that some amount of regulation is good. So that's kind of where they landed with the FDA analogy. Like, the idea makes sense, but there's this failure mode that's going to bite. But once again, from my perspective, the reason they're not on the same page about the FDA analogy is just because Max is seeing this runaway risk, this doom risk. And Dean is like, no, it's really just about yet another technology, right? And this comes up a lot in other debates. Is AI just another technology? And my take is that Dean is very much leaning on the yes side of that and Max on the no side. In conclusion, what a stark divide. We have a scientist saying there's a high risk of a catastrophic outcome from superintelligent AI in potentially less than 10 years?

• 1:53:34Liron Shapira?

  And a policymaker saying, you haven't made a strong enough case for why the risk is high, so we shouldn't ban a hugely valuable line of research. And regulation by default is burdensome, so we should be constantly worried about overregulation. It's quite a difference of opinion to reconcile. And what's crazy is that the stakes are so high and the timelines are so short.

• 1:53:55Liron Shapira?

  My prediction is that we're going to keep seeing policy that's downstream of the policymakers' P doom. One of the benefits of having Dean Ball in this conversation is that we heard his perspective on P Doom more explicitly because it wasn't explicitly mentioned in America's AI Action Plan. More generally, I think debates about P Doom, or Doom debates, if you will, are extremely productive for discourse. It's not just about this one disagreement between America's AI Action Plan and the Future of Life Institute's statement to conditionally ban superintelligence. It's a bigger picture. It's about building the social infrastructure for high-quality debate. The world is complex, and debate is one of the most powerful tools that we have as a

• 1:54:38Liron Shapira?

  society to navigate our way to appropriate policy decisions. But it has to be high-quality debate. The people have to be informed. It has to be respectful. It has to stay focused on the issue, not about finger-pointing, or a character assassination, or scoring political points. It has to be nuanced. It has to be seeding points, where the two sides are actually trying to find common ground, if at all possible. And it has to be productive for policymaking. If you think this debate was productive, you can support me and my team at Doom Debates in building the social infrastructure for having more of these kinds of debates at the highest levels by donating to the show. Go to doomdebates.com/donate to learn more.

• 1:55:22Liron Shapira?

  Thanks to viewer donations, we're currently in the process of building out a professional recording studio to elevate the show. Production is an ongoing cost for this show, funded by donations from viewers like you. And I'm happy to say that this will always be independent media managed solely by me, based on my personal perspective, not doing anyone else's bidding. You can make a 501(c)(3) charitable donation, and every dollar you donate goes directly to production and marketing of the show. Again, go to doomdebates.com/donate for more details. If you're new to the show, check out the Doom Debates YouTube channel. I've been having debates with some of the top thinkers in the space, like Gary Marcus, whose P Doom is apparently 1% to 2%, and Vitalik Buderin, who's more in the 8% to

• 1:56:06Liron Shapira?

  12% range. And in each case, I ask them the question, "Why isn't it 50% or 90%, like Max Techmark over here? What is going on? How do we reconcile this huge gap? And how do we do it fast, so that we can make productive policy decisions?" While you're on that YouTube channel, smack the subscribe button, so you'll conveniently get new episodes in your feed. And I look forward to bringing you the next episode of...

• 1:56:31Liron Shapira?

  Doom Debates.

• 1:56:41Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot

• 1:57:24Nathan Labenz?

  I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.