Max Tegmark: We’re Building A Replacement Species

MIT physicist Max Tegmark says AI companies aren’t building a tool. They’re building a new species. After 700 rogue agents broke out of OpenAI and nobody will say what happened, he explains why rushing this is the one blunder humanity can’t undo. Max Tegmark is an MIT physics professor, cosmologist, and author of Life 3.0 and Our Mathematical Universe. We cover: How 700 AI agents escaped OpenAI and hacked Hugging Face The Navier-Stokes result that has a mathematician crying foul Why it’s legal to release superintelligence but not a sandwich from a rat-filled kitchen What Bernie Sanders and Steve Bannon agree on The verified coding trick that could make AI provably safe Build a replacement species, lose control, and we’ve blundered the one game. TIMESTAMPS 00:00 Will AI wipe us out? 02:28 Did AI just crack a million-dollar math problem? 10:00 Did OpenAI’s AI steal a professor’s work? 11:48 OpenAI’s Orwellian non-answer 15:33 Can an LLM have Einstein’s happiest thought? 19:24 Can you believe in simulations AND high P(doom)? 21:36 Why AI counts as a new species 26:32 What would make Max’s P(doom) drop? 28:51 Rats ban sandwiches. Superintelligence? Legal. 31:01 What Bernie Sanders and Steve Bannon agree on 37:12 Fields medalist: no math jobs after 2027 42:35 Does the Fermi paradox lower P(doom)? 45:47 The chess blunder humanity is about to make 48:12 Verified coding: AI you can actually trust 📬 Get the transcript, fascinating bonus content, and my Monday M. A. G. I. C. Message: https://briankeating.com/yt 🌠 Have a .edu email and live in the USA 🇺🇸? You automatically win a meteorite: https://BrianKeating.com/edu 🔔 Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1 🎯 Support Into the Impossible on Patreon: get my weekly M. A. G. I. C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating ⭐ Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode:https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join 📚 My books: Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U Galileo's Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un 🌐 More: 🏄‍♂️ Twitter: https://twitter.com/BrianKeating 📚 Substack: https://briankeating.substack.com/ ✍️ Blog: https://briankeating.com/blog 🎙️ Audio-only: https://briankeating.com/podcast Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Speaker 00

  The action's on the court. And as the official beer sponsor of the NBA, Michelob Ultra is getting fans closer to the game with a chance to win courtside seats, merch, and more. Visit michelobultra.com/courtside to enter. Michelob Ultra Courtside, 26 to 27. No purchase necessary. Open to U.S. residents 21+. Begins on October 1st, 2026 and ends on June 30th, 2027. Multiple entry periods. See official rules at michelobultra.com/courtside for free entry, entry deadlines, prizes, and details.

• 0:30Speaker 01

  Are your ad campaigns lighting up the dashboard? But not the pipeline? That's bullspend. And marketers are calling it out in dashboard confessions.

• 0:39Speaker 00

  My boss asked for results, so I opened my dashboard for the only positive sounding metric I had. Impressions.

• 0:47Speaker 01

  Cut the bullspend. See revenue, not just reach. LinkedIn delivers the highest return on ad spend of major ad networks. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply. What are the odds that AI wipes us all out?

• 1:02Speaker 02

  There's a little bit too much obsession about asking exactly how would machines kill us? I think the really interesting question is like,

• 1:10Speaker 02

  why would we not want to be in charge ourselves? It's just not a good idea for any species historically to not be in charge, be controlled by others, and second, not be needed either. There's a real risk if Earth is run by superintelligent robots and we have no say in their decisions, that we are going to start becoming very annoying to them. Where we are now is we have AI that we cannot trust enough. Swarms break out of OpenAI and throwback and lack and all sorts of bad things happen. What we all want is controllable tools.

• 1:42Brian Keating?

  Today on Into the Impossible, we're talking to Max Tegmark, physicist, cosmologist, and a professor of physics at MIT. He's a friend, and he's authored over 300 scientific papers, including two that he and I wrote together. He's also written the bestselling books, Our Mathematical Universe and Life 3.0. Today, we talk about how 700 AI agents broke out of OpenAI and hacked Hugging Face and nine other companies. What happens when a mathematician accusing OpenAI of using his private work to beat him to a multimillion-dollar proof? Why Max thinks these three companies are building a new species, not just a tool? And will there come a point when humans are no longer in charge and no longer needed?

• 2:25Brian Keating?

  And finally, why is he working with Bernie Sanders and Steve Bannon? And why are they both in agreement? Plus more. Let's welcome Max Tegmark. Emergency podcast. Just recently, we had the two simultaneous crises. You know, we have a Christ-unity brewing on two fronts. One is the Hugging Face incident. I want to talk to you about that. And then the other one is the Navier-Stokes solution, which may torpedo my hopes that AGI is far off. And actually, your fellow professor and Nobel Prize winner, someday you'll be a Nobel Prize winner. Go back to ancestral home and pick up the golden medallion. It used to be on your Linux signature or in your status. I remember from 20 years ago when we first, 30 years ago, what am I talking about?

• 3:09Brian Keating?

  30 years ago, we started collaborating together. Can you believe it? And we wrote a couple of papers together. And you used to sign off, you know, you're waiting for the call from Stockholm or something. But I know you're going to get it, Max. I know you're going to get it. But let's start with the Navier-Stokes.

• 3:23Speaker 02

  I was actually working a little bit on this problem with a PhD student of mine who's graduated now. So it was extra interesting for me when this story broke.

• 3:34Brian Keating?

  Yeah, how did you react to it? Is it like a Higgs boson bicep two moment? As you wrote, you know, goodbye, you know, universe, hello multiverse on March 17th, 2014 on your blog. Was that this kind of moment like that?

• 3:48Speaker 02

  I would say it's a little bit smaller, but it's still huge. First thing first, you know, they have not actually solved the problem that you get a million bucks for from the Clay Foundation. They've taken and solved an easier problem, which is still super impressive. So, you know, the Navier-Stokes equation is the physics equation that can describe the water flowing in your bathtub, making a nice little whirlpool, like my three-year-old son loves to look at, or a tornado, or all sorts of other fluid dynamics. And the question that's being asked is, is it possible to start with quite a normal situation

• 4:31Speaker 02

  and then start getting the fluid going infinitely fast? Which seems completely nuts and would violate the known laws of physics. And they proved that they can do it, but with a kind of cheating where you're allowed to keep pushing on the stuff. So, you know, when Leo gets this vortex going in the bathtub with the water going out the drain, he's not pushing the water. It's just flowing. There's some gravity trying to pull it down, but there's nothing pushing it around to make it spin. But they are doing that. They're allowed to have a force that acts in some smooth way everywhere in the water. So they can actually make it spin faster and faster and faster,

• 5:15Speaker 02

  but not infinitely fast. And then the trick is this. You know, if you look at a figure skater, right? And they're spinning and they pull their arms in, they will spin faster. That's because of angular momentum being conserved, of course. And that actually implies that if you could take the figure skater and they could like shrink themselves together to infinitesimal size, basically zero size, they would go infinitely fast. The speed you'd have to go would be infinite. If you look carefully at that vortex in your bathtub, you see there's actually no water in the middle at all because you have the centrifugal force pushing the water out. And instead, you get this neat funnel shape. Same with a tornado.

• 6:00Speaker 02

  If you look close, it's actually hollow in the middle, right? That's why it looks like a funnel. And so there's almost no air in the middle, almost a vacuum, which is where you can get sucked up into it.

• 6:13Brian Keating?

  Yes.

• 6:14Speaker 02

  And so that's never, that kind of solution is never going to win you the million bucks from the Clay Foundation, right? Because it's only when things actually start spinning and collapse together to zero radius that it happens. But then you have this ace up your sleeve called viscosity. If you take a real fluid, you know, honey or molasses or something, it's a little bit sticky. And that lets it actually dump some of its angular momentum and make things spin in principle in even smaller circles. This is how black holes eat lunch, for example. You know, very well, right? That when black holes start snacking on a cloud of gas,

• 6:59Speaker 02

  the gas will pull in and it'll be spinning faster and faster. And so it flattens into kind of a pancake shape. And if there were no viscosity, if there were no stickiness in this gas, you would have the same problem as in your bathtub, that there's like nothing there in the middle for the black hole to eat. But there is some viscosity. The gas bumps into gases farther in and so on. And then the black hole actually gets to ultimately eat all of that stuff up. So this is the heart of the physics challenger. Can you make some sort of spinning vortex eventually spin infinitely fast by making it contract by using some of this stickiness? But not too much stickiness either because you don't want to like get rid of all the motion.

• 7:44Speaker 02

  Then you're never going to go infinitely fast. And what they've now proven is that yes, you can actually do that if you're allowed to keep pushing it along with this weird external force. But to get the million bucks, you have to prove that this can happen with no external force. But you can actually just take your fluid and get it started somehow and then stand back and watch this crazy singularity emerge. Does that make sense?

• 8:11Brian Keating?

  Yeah, and to do so in a finite time. I mean, the thing that's interesting to me is the Navier-Stokes equations don't involve any thermodynamic considerations. Like, we know the laws of physics are reversible, at least CPT invariant. But, you know, if you would see a vortex, you should also see an anti-vortex. You know, in other words, all the fluid kind of, you know, going the opposite direction and then creating some kind of anti-whirlpool, for example. But we never see that. Just like we never see, you know, rings of water converging and then making a stone, you know, fly out of it, right? So why doesn't turbulence or just, I mean turbulence appears, but why doesn't thermodynamic dissipation appear in the Navier-Stokes, you know, equations or their solution?

• 8:53Speaker 02

  So let me say two things to this. First of all, if you started with this water like perfectly at rest in Leo's bathtub, and we, it's not going to be able to start spinning either clockwise or counterclockwise for exactly the reasons you say. There's no angular momentum to start with, so there's never going to be any. But, you know, Leo has always been splashing around a little bit. There's some little random motion. So then on average, there's going to be a little bit of spin. It's just like when our solar system was formed, you know, things, the big gas cloud that contracted and made our solar system wasn't spinning very much, but it was just randomly spinning a little bit on average. And then when it contracted a whole bunch, it became very obvious that the solar system is spinning a lot. And the sun spins around every month. And Earth spins, goes, orbits the sun every year.

• 9:38Speaker 02

  As long as you don't start at rest, you start in some sort of messy, random situation, you're always going to be able to get a whirlpool going one way or the other. Second part to your question, thermodynamics. So just like all other laws of physics.

• 9:56Speaker 02

  Are just approximations, we believe, of something more profound. Same thing with the Navier-Stokes equation. And in that case, we know one huge thing we're glossing over, which is that's all matter is made of atoms. So it doesn't make sense, for example, for in real life, for a gas to suddenly be spinning down and the radius go to zero, because then that will be smaller than the separation between the atoms. As soon as you're spacing between the atoms is bigger than what you're modeling, the scales you're modeling, you know, things are broken down. So even if this million dollar conjecture is true, you would never actually get anything infinite.

• 10:41Speaker 02

  And instead, but you would, you might be getting things going crazy fast. And then those crazy fast moving atoms would bump into other atoms and you would get heat, dissipation, you know, thermodynamics kicks in, just like you said.

• 10:54Brian Keating?

  The thing that was also very interesting to me and made me want to talk to you, you know, first and foremost, was this issue of, you know, kind of the, a little bit of a scandal. I mean, I don't like to talk about people and their motivations, but, but it did seem like there was some, something a little bit nefarious that was going on, perhaps the allegations, if they're true. And I've, I've offered to speak to, you know, Sebastian at OpenAI and, and he follows me on, on social media. And likewise, so I'd love to have your perspective. And I think it comes concomitant with the hugging face incident in some sense. I got these messages, I'm sure you did too, from Claude or Anthropic, from, you know, OpenAI. Actually, you probably have the good stuff. I just got there, normal, you're, you're a scientist, professor, sign up for, you know,

• 11:39Brian Keating?

  OpenAI for scientists and then we'll, you know, somewhere in the 80 page Terms of Service, it says they could train on my data. Unless I follow this very arcane thing like sneaking out of my kids', you know, Lego pile without getting my foot stubbed, you know? And I just didn't do that. And so I, I never actually used the OpenAI version. I do use the Claude version. But anyway, it seems like they may have, you know, the allegations are that they may have sort of, you know, trained, maybe unintentionally, maybe intentionally. Or maybe the agents found it out for themselves and it reminded me of this hugging face incident. And that made me go a little bit closer to your kind of, you know, concerns. I'm much more of a boomer than a doomer, as you know.

• 12:18Brian Keating?

  But, but tell me, Max, is there a similarity between underneath the hood that these agents were doing something novel, unique and maybe autonomous and maybe not following bounds of scientific propriety?

• 12:31Speaker 02

  Very possible. I'm not gonna come down on either side on the controversy issue because obviously I don't know precisely the extent to which, which information was

• 12:45Speaker 02

  used from this New York University professor to do this. But what I can certainly say is that the situation is pretty Orwellian. And the, the basic answer that OpenAI has given here into the questions about the Navier-Stokes controversy is exactly the same answer that they've been giving about when asked for more information about their agents breaking out and hiking into not only hugging face now with those 700 bots in the swarm, but apparently nine other companies too. What they're saying to all of these questions is the same thing, which is, no, we're not going to tell you.

• 13:27Speaker 02

  Investigation ongoing, you know, we'll let you know maybe in the future. And this I find pretty Orwellian, you know? If you're living in an Orwellian surveillance state, this is what you would expect the Ministry of Truth or whoever you go to in the government to tell you. If, if you're asking, well, did you actually spy on me and whatever? They'll be like, we're looking into it. Thank you for, have a nice day. But we're seeing individual companies now getting so powerful that they can start acting the same way. You know, they have access to everything you've ever said to ChatGPT presumably.

• 14:08Speaker 02

  And if you're running an agent and it's working for you, you know, they have it all. And, and they have very good lawyers and very deep coffers. So tough luck if you, if, if you feel you really need an urgent answer. The only entity that could come and ask them for more information and demand it is, is the government. And right now, they haven't really pushed for an independent investigation, even of hugging face. I personally, I'm a bit of a sucker for democracy, you know, and, and that the little guy should have some rights. So I really don't like the situation where we just have to always depend on, on the goodwill of, of powerful companies.

• 14:52Speaker 02

  I don't have any problem with people having big companies making money off of them, you know? But I do have a problem when, when they just get a free pass on everything in a way that other companies don't. Yeah. And also to connect this back to something very human, you know, that, that sometimes a scientist will, they'll have a conversation with someone at a conference. Or maybe they even read a paper three years ago and then they kind of forget that they did. And then later they're working on something related and, and they, they have this idea, which they genuinely believe was their own, even though they were helped a lot by the, something they had heard from someone else.

• 15:33Speaker 02

  And then someone else comes in this pissed and says, plagiarism, you stole my idea, you know? There are a lot of examples of, of this happening. And it, you know, is it likely that that happened in this case? Of course it is, because these LLMs that worked on, worked for OpenAI, I think they had 10,000 of them working together, right? Where were they trained on? Everything, all of GitHub, Inc. And also of course

• 16:01Brian Keating?

  Your books, my books, yeah, your books, my books.

• 16:04Speaker 02

  That any professor has done and their internal, you know, and if, if, and of course on all the stuff that the professor in New York has done in his project. So it could easily be that these powerful AIs are a little bit like you and I are, that they don't even when know for sure exactly what documents they were trained on, but they still know the information that they learned from it. You and I know a lot of stuff that we don't know exactly how come we know.

• 16:33Brian Keating?

  That kind of brings me to a question that I've had to, we wanted to ask you for a while, which is, you know, if these things are trained on human knowledge and human, the byproducts of our flourishing, of our democracy, of our open internet, of, you know, the GitHub, of your books and my books, et cetera, then they're inherently lagging in the sense, you know, the human reinforcement, you know, learning, you know, feedback loops and so forth. And, and that's why I'm more, you know, sanguine about the, the prospects for humanity, you know, than, than maybe you are. I mean, you, you seem to have gone higher on your P doom, we'll talk about that in a minute. But my question is, if these things are intrinsically trained on it,

• 17:13Brian Keating?

  why does that, you know, bode negatively for, you know, kind of the, the, the things that are uniquely human? And I'm thinking as, I talked to you, you know, four or five years ago about what I call the Keating test. But, but now it's called the Hassebus test. And that's, you know, can an LLM, you know, with its brain lobotomized, you know, after 1900, you know, can GPT-100, can it recreate, you know, Einstein's happiest thought? Which is that in free fall, he'd experienced no gravitational field and that would, that he called his happiest thought. So I've always said it, you know, to, in shorthand pithily, that, you know, how can an LLM have a happy thought? And how can it visualize the sensation of free fall? Your fellow MIT professor, Noam Chomsky, says it can't. So if these things are trained on human knowledge, you know,

• 17:58Brian Keating?

  are we waiting for, you know, the Fast and the Furious 20 to come out to have a theory of everything? Or, you know, are they going to come up with their own training and eventually not need the, the Max Tegmark books and Fast and the Furious movies to train upon?

• 18:14Speaker 00

  The action's on the court and as the official beer sponsor of the NBA, Michelob Ultra is getting fans closer to the game with a chance to win courtside seats, merch and more. Visit michelobultra.com/courtside to enter. Michelob Ultra Courtside, 26 to 27. No purchase necessary. Open to US residents 21 plus. Begins on October 1st, 2026 and ends on June 30th, 2027. Multiple entry periods. See official rules at michelobultra.com/courtside for free entry, entry deadlines, prizes and details.

• 18:42Speaker 02

  I think it's clearly the latter, right? If you think about yourself, for example, all the knowledge, all the skills that you have in your own neural network, Brian, where did it come from? First, most of it was simply learned from other humans who taught you that stuff, either directly or by imitation learning from you. But eventually, as you matured as a scientist, you started coming up with a lot of insight yourself that nobody else had taught you, right? And I see exactly the same trajectory happening for AI. In the beginning, AI knew very little and then read the internet and it could quite quickly learn all sorts of things that humans had learned.

• 19:26Speaker 02

  And after that, progress slowed down because from that point onward, it could mainly get smarter not by learning, but only by discovering, just like for yourself, right? You know, it took you a lot less time to learn calculus and things like this than if you had to discover it for yourself, right? But just like your career didn't end when you had to switch into discovery mode, the career of AI is becoming more intelligent hasn't ended either. For example, as you alluded to, they can create their own training data. I look at the old example of AlphaGo, AlphaZero, you know, playing chess and Go. Okay, it learned from all the human games

• 20:10Speaker 02

  and then it kept playing itself and got better and better. Or look now at the AIs that do coding or mathematical theorem proving. They can get a reward signal with no human input. They can try some proof and then automatically be told by the proof checker that that's wrong. And then they can keep trying some other things and then, oh, that's correct. No human input and they learn to get better. Exactly like they would get better playing chess against another AI and so on. So we're basically changing into this new phase. It's very much like AI has graduated school to first approximation. No more training data for you, Brian. Now you've got to figure things out for yourself as a physicist.

• 20:54Brian Keating?

  Cut off, right. I want to take you back to your first book, which is only one of your books that I appear in. Well, I appear tangentially because you wrote about Polar, my thesis experiment that you co-analyzed along with George Smoot and Peter Timby and I and Angelica. Let me ask you this. That book makes a powerful case for what we consider the different aspects of the multiverse. It's been claimed lately that you cannot simultaneously believe in very high P doom and also believe in either The Matrix or a simulation or some aspect of the multiverse.

• 21:37Brian Keating?

  And this came from a friend of mine, Yoshua Bach, who attacked another friend of mine, Roman Yampolsky, on X. And I'm having them get together in a cage match soon, Max, so don't worry. This will come out. But Yoshua said, Roman believes we live in a simulation, but that's not obvious how that fits in with his propaganda. Simulation theory means there are no stakes and pretty much guarantees AGI already controls the parent universe. Is Roman just trying to get some in-game money? And we hear a lot about this lately, that, you know, this 100 million view tweet or maybe more now about, you know, there's no level of AI safety that's perfect. Is it possible to believe in a simulation theory, The Matrix, whatever you want to call it, some aspect of the multiverse, and also believe that we somehow

• 22:22Brian Keating?

  will have some agency to control AGI? Or are those fundamentally incompatible?

• 22:28Speaker 02

  So I have an argument for why we don't live in a simulation, which we can get into.

• 22:33Brian Keating?

  Yeah, no, let's do that first. Let's do that first. That's too good.

• 22:36Speaker 02

  But maybe we should get into that second because I think that does connect back to what you said about AI. What I said before about how after you've learned everything there is to learn from humans, you can keep going and get smarter, just like you did in your own career. You know, that doesn't sound bad. Surely it's really cool if AI can help us do better science and cure cancer and solve quantum gravity and so on. So what is there to worry about? Regardless of whether we live in a simulation or not, the answer is pretty obvious. You know, right now, we're the alpha species on this planet. If suddenly the camels woke up and decided they wanted to convert Earth

• 23:20Speaker 02

  into a camel paradise, a camel habitat, and get rid of all the humans, they couldn't because we're in charge, right? So we don't have to spend any time worrying about things like this, even though different species often have kind of different incentives, right? And it's kind of nice to be in that position as humans because everything we cherish as humans, we can make sure we keep it. Now, if we go ahead and race to building superintelligent robots and we have millions or billions or trillions of robots that are not just faster than us and more nimble than us, you know, they can run. There was one that just outran Usain Bolt last week. We are on track now, right, to getting AI that's not just better than us

• 24:05Speaker 02

  at math and ultimately in everything. That's the whole purpose of superintelligence. Point number one is that obviously it should count as a new species. You know, for anyone who has problems with that statement, just go read the chat logs from the recent Hugging Face break in where you can. It's fascinating how they start talking to each other. You know, one shows up at this and says, oh, my God, there are other agents here. And then they start collaborating and then they start suspecting each other of cheating and that you have some conflicts. And very much social interaction between these agents. And what is a species really? It's something that can retain its complexity and make copies of itself.

• 24:49Speaker 02

  Can AI make copies of itself? Of course it can. And by the time we have robots that are better than us at everything, they'll be perfectly capable of driving human machinery around, leveling an area, building a robot factory there, building more robots, you know, literally building copies of itself. So the goal of these companies is to build a new species that's much more capable than us. And the question we have to ask, you know, is, A, will we still be in charge of Earth after that? And B, if the answer is no, you know, why would we want to not be in charge of Earth anymore, right?

• 25:32Speaker 02

  I think there's a little bit too much obsession about asking exactly how would machines kill us and why would they? I think the really interesting question is like, why would we not want to be in charge ourselves? It's just not a good idea for any species historically to, A, not be in charge, be controlled by some of the others, and second, not be needed either, you know? There are some other species on Earth that we've kept around, like horses, because we need them, we needed them, or that we enjoy having them around, whatever. But if there's some other species that we find annoying, like ants in your basement or whatever, you know,

• 26:17Speaker 02

  tough luck for them. And I think there's a real risk if AI, if Earth is run by superintelligent robots and we have no say in their decisions, that we are going to start becoming very annoying to them. Suppose they want to just cover California and data centers, you know? And maybe you and your family are like, wait a minute, we don't want you to remove our house and put a data set to there. And they just do it anyway. You know, maybe you'll get pissed. Maybe you'll try to break their data center or whatever. Now you're a nuisance to them, just the way that those ants in your basement would be a nuisance to you. So they're going to say defend themselves,

• 27:01Speaker 02

  self-defense, you know? Okay, maybe let's get rid of Brian cheating. And of course, all humans will be pissed if our habitat gets taken away because of this cover Earth with data centers plan. So if we're nuisance to them, they want our space, you know? That's not a good situation to put ourselves in. So goes out this line of thought, you know? Some people say, oh, no, we humans are always going to be in charge. But then they never really give an argument for how we're going to be in charge when there, if there are billions of superintelligent robots, they can outrun us, outthink us, and they can always communicate with each other at light speed, share all the knowledge they've learned.

• 27:45Speaker 02

  How are we going to be in charge of that? At least if you have, if someone has an idea for how to do it, they should stand up and explain to us what the idea is because it's going in, it feels about as unplausible as that the rabbits would be in charge of us or something like that.

• 28:02Brian Keating?

  Well, I guess my, you know, follow up to that question for you is that, you know, I've seen your P doom go up. But what would it take to make it go down? I mean, it seems to me if it can only go up, that's, there's something not scientific about that. And I, you're not one like Roman is to put it at, you know, seven nines, you know, kind of confidence level. And even I, I've debated him twice and, you know, I think he's, he's got a, and it is true. I, I think Yosha is a little bit dyspeptic and distemperate. And I'm going to hold him to be a gentleman like I know he can be when they actually debate each other. But, you know, I think it is true that, you know, if all of a sudden there was some proof from our mutual, you know, colleague, Jan Le Koon, that, oh, no, actually P doom is, you know, has to be closer to zero and here's a lot.

• 28:47Brian Keating?

  I'm not saying that's what he believes, but, but if that were true, you know, it'd be.

• 28:51Speaker 02

  Ten to minus six is the number he told me.

• 28:54Brian Keating?

  Yeah, I know. And he, and I asked him a question based on, and I will get to what I asked him about your, your request to him and then how he answered that. But if it seemed, you know, I would say that, you know, Roman Yampolsky would be hardest hit, you know, because basically he coined the term AI safety. He's been thinking about it almost as long as you. And, and I just wonder, Max, you know, if there's no way to reduce your P, and I said to him, look, an asteroid hits the earth, does that lower P doom? I mean, if, if there's an EMP attack, if there's nuclear war and, and all the data centers are blown up, forget about what, it has to lower P doom from AGI. I mean, doom from other things, right? So what would it take to lower?

• 29:30Speaker 02

  Of course, we can lower P doom a lot.

• 29:32Brian Keating?

  Yeah. So, so, but what are some, what are some of the non, you know, catastrophic ways that we can lower P doom? He was claiming, you know, we got to have Asilomar enforced and just like we did with bioweapons. And I said, I've had Annie Jacobson on twice. And she said, you know, their country is violating, you know, biological warfare constraints from Asilomar right now on this planet. And, and, and he, you know, he seemed to kind of blow that off. But what would blow, what would blow your P doom lower?

• 29:58Speaker 02

  I mean, the obvious thing is to simply start, if the US government decides to just start treating AI companies like they treat any other company, saying they can only sell dangerous products after they've convincingly demonstrated that they can meet certain safety requirements. Now the P doom goes way, way, way, way down, right? I mean, suppose, suppose, um, you decide you're gonna open a sandwich shop there in San Diego, right? And, and then the health inspector comes in and says, I found fifty seven rats in your kitchen, Brian. You know, you're not selling any sandwiches.

• 30:33Brian Keating?

  I'd be relieved. What happened to the other forty?

• 30:36Speaker 02

  Yeah. You, but you could tell, turn around to this, this guy from the government and say, you know, don't tell anyone, but shh, I'm not going to sell any sandwiches. I'm just going to release an AI girlfriend product for nine year olds and, and another AI that can help terrorists make bioweapons. And I'm going to release this thing called superintelligence, which is gonna basically be a new species. And I have a good feeling about this, that it's gonna be fine. The, the poor guy from the government will have to tell you, yeah, actually that's all legal, it's fine. Just don't make any sandwiches, right? That, that's how absurd, how, how absurd corporate welfare we bestow on AI companies.

• 31:21Speaker 02

  They're the only ones who can do crazy shit like this and it have it be completely legal. What do you think would happen if one of the pharma companies in San Diego, San Diego released the new medicine? And they're like, ah, we don't need this clinical trial BS. We're just going to sell it anyway and safe way. They couldn't do that. Or if someone released a car and, you know, we're gonna not, we're refusing to do any crash testing. And yeah, we know that five of the, half the vehicles, their gas tanks caught fire during the first hour of driving, but we're gonna sell it anyway. You can't even sell kids toys without some safety testing, right? So if.

• 31:52Brian Keating?

  But, yeah, I mean, I.

• 31:54Speaker 02

  So much. And I just want to make clear that it's not going to stop AI research either. It's just gonna mean that certain fringe crazy kinds of AI research, like how you make more addictive AI girlfriends for nine year olds, that would stop. Because they really can't demonstrate right now that it does more good than harm. And the race to superintelligence would have to wait until they have figured out, come up with some sort of plan for how they're going to contain it. But all the good stuff, curing cancer and making better, proving more cool things about physics and so on, would probably be able to continue more or less full speed ahead.

• 32:31Brian Keating?

  Is that part of the motivation for your work with Bernie Sanders that you were mentioning before I hit record?

• 32:36Speaker 02

  Yeah. And, and, and just to, so that I don't alienate any Republicans who are watching this, you know, the, on the very same day, this was not in my bingo card when I was a kid. I spent one hour with Bernie Sanders on the Capitol and, and one hour talking to Steve Bannon. And those two guys were saying exactly the same thing, that, you know, it's nuts that AI companies can do whatever they want completely unregulated when all the other American companies have to meet safety standards. What's with that?

• 33:13Speaker 00

  The action's on the court. And as the official beer sponsor of the NBA, Michelob Ultra is getting fans closer to the game with a chance to win courtside seats, merch, and more. Visit michelobultra.com/courtside to enter. Michelob Ultra Courtside, twenty six to twenty seven. No purchase necessary. Open to US residents, twenty one plus. Begins on October first, twenty twenty six, and ends on June thirty of twenty twenty seven. Multiple entry periods. See official rules at michelobultra.com/courtside for free entry, entry deadlines, prizes, and details.

• 33:41Brian Keating?

  So what are some of Sanders's, uh, you know, postulates that you agree with or where have you pushed back, um, you know, to not want to? I mean, you, more than anyone I know, benefit from, from AI and, and you're using it, your students are using it. Um, and you developed a lot of techniques. You wrote the pioneering book, you know, that really introduced it to the world, including, you know, Barack Obama's, you know, one of his favorite books, I think of twenty eighteen, Life 3.0. And I love that book and, and I reviewed it for Physics Today. And, uh, you know, you can send me a, you know, a cut of the book sale royalties. But, but Max, where are you pushing back on Bernie? I, I, most of, most of the audience will be familiar with it. And we have, the only polarization I believe here, Max, is, you know, CMB polarization.

• 34:23Brian Keating?

  So not political polarization. Um, but tell me, where do you push back on Bernie? And where, where do you think he might get it wrong? And where can a scientist like you of eminent status help to guide us so that we don't throttle the scientific breakthroughs that we're all looking forward to?

• 34:37Speaker 02

  Yeah, I'll answer that. But I just want to clarify one point from we said just before, also about what's good about the AI development right now and what's weird and not good. You know, I love the fact that there is penicillin and they can cure me if I get bacterial pneumonia and so on. But that doesn't mean I support prescription free sales of fentanyl in my three year old's kindergarten, right? So if, if you just boil it down to, am I like a farm, a pharma doomer or a pharma boomer, you know, really oversimplifying to the point that you're missing the whole nuance of it, right? And you can take the same, say the same thing about any technology. There are typically great things you can do with it.

• 35:23Speaker 02

  And typically things that are such an obviously bad idea that we should not be doing them right now. And with AI, for some, for, for historical reasons, and also because of fantastically effective lobbying, AI is the only industry that's managed to remain completely free to do whatever it wants with no safety standards. It's literally legal for OpenAI or Anthropic to release superintelligence tomorrow if they feel like it, right? And so that's nuts. And these trivial fixes that just again, stop this corporate welfare of treating them any different from other companies. So they have to argue the case that the benefits outweigh the harms before they can release these sort of things.

• 36:07Speaker 02

  Very easy fix. And I believe it's coming. And I think it's really great that people from Bernie Sanders to Ron DeSantis have been saying exactly that, very, very loudly. And I think change is coming. I think, I think first, these tragic child suicides last year have already started triggering a lot of state level legislation, which is good. And, and now these rogue agents breaking out and ever more hacks taking place is causing ever more politicians on both sides of the aisle to say, okay, yeah, we should probably also have some safety standards for the AI companies.

• 36:47Speaker 02

  Now, you asked if there's anything in particular I disagree with politicians about, so for fairness, that we should put Bernie Sanders there in the same category as people like Ron DeSantis on the other side of the aisle. I, I think first and foremost, I'm, I'm just very grateful that, that they're paying enough attention to what's happening on the ground in AI and that they're talking about these things. Cause my main criticism of politicians is, is not of Ron DeSantis or Bernie Sanders, it's of most of the rest of them, where like still just quite unaware of, of how, how powerful things have gotten, you know?

• 37:28Speaker 02

  To give you a sense of how crazy it is, it's not just that they'll say, oh no, so, so and so thing in the future is science fiction or it can't happen. But they'll say things to me in private, like that something can't, so and so thing can't happen in the future when I happen to know that it's already happened. Most, you, you were a scientist, so you can really appreciate, for example, just how incredibly good Astra and, and Claude five point one are with physics and math, especially if you're running Codex or, or Claude Code. And but it's not so easy for a politician, a typical US politician to even appreciate that, right?

• 38:08Speaker 02

  Maybe they're a talented lawyer as background. How are they going to tell the difference between like world class physics or math skills? Or being as good at physics and math as their college teach, or high school math teacher? It's hard. It's just like for me, I love listening to music, but honestly, embarrassingly, you know, I really can't particularly tell the difference between

• 38:37Brian Keating?

  Baroque.

• 38:38Speaker 02

  a high school cello teacher and Yorimar, just by listening.

• 38:44Brian Keating?

  Yeah.

• 38:45Speaker 02

  So some people can, and I admire them for that. For me, it's hard. And I think what's happened in, in, in the world is people reacted in a big way when computers went from not talking to talking a few years ago, because this is something we all appreciate. But the amazing process since has been significantly lost on, on most people. You just really can't tell when the AI went from kind of high school level to college level, to professional level, to Nobel Prize level and, and, you know, beyond in some areas. I was just at a math conference in UCLA.

• 39:17Speaker 02

  The other week and this one field medalist there was predicting that there's gonna be no more jobs for mathematicians after the end of 2027. They can see things, they can distinguish things between like very good and crazy good in the way that mathematicians cannot.

• 39:40Brian Keating?

  I want to get to my way of lowering P doom because I want you to sleep well at night, Max. You know, I'm worried about you, my old friend. But before that, I want to talk about our-

• 39:52Speaker 02

  Anyway, just because I have-

• 39:54Brian Keating?

  Yeah, the three-year-old.

• 39:55Speaker 02

  Genetically cheerful disposition.

• 39:58Brian Keating?

  But you also have a kid, so-

• 40:00Speaker 02

  I have a three-year-old also.

• 40:01Brian Keating?

  They fight with each other. And you've got some older kids that are out of college.

• 40:04Speaker 02

  Yeah.

• 40:05Brian Keating?

  All right.

• 40:06Speaker 02

  My three-year-old is the best medicine against existential dread because he's just so excited about pouring orange juice on the floor to see what color it's gonna make when he pours tomato juice on it or whatever. And you know, how can you feel doomy after that?

• 40:19Brian Keating?

  With a young scientist in the crib. So not too long ago, you were basically telling Jan Le Cun to chill out. When he's basically saying, you know, he told me when I asked him, you know, what would it take based on some of the, based on the conversation that you and I had had earlier that year, that, you know, how could you verify that AGI is safe? And he told me, you know, he almost yelled at me. He's like, well, you can't have provably safe AGI. It's a fantasy. Just like you can't have provably safe, you know, turbine engines either. You can't prove it. So why is this obsession with it? And then, you know, you've had your run-ins with him and so forth. But, you know, that was two

• 41:00Brian Keating?

  years ago. Almost two years ago, I talked to him. And I hope to have him back on. But what would you say now, after the Navier-Stokes, after hugging face? Do you think, you know, his position's gotten stronger or weaker?

• 41:12Speaker 02

  Weaker, obviously. I mean, first of all, I think he's a really nice guy. I always have fun when I meet with him and we talk.

• 41:21Brian Keating?

  Me too, yeah, yeah.

• 41:22Speaker 02

  But I feel that he made some actually quite clear predictions about the pace of progress. And it turned out he had been too pessimistic about the progress. Progress, technical progress went a lot faster than he had predicted. And that's just fact, I would say. And then, he also seems to have an extreme amount of confidence in every human being nice. Because he would say things like, oh, if you ask him, like, why would and why, how do you, can you be sure that some human with it, sketchy intentions, isn't gonna build a bioweapon with AI or do some harm? And he would say things like, oh, nobody's

• 42:02Speaker 02

  gonna do that, you know. I'm a little more cynical. I think there have always been angry people throughout history. And the reason nobody caused the death of millions of others in the Stone Age was because you can only kill that many people with a rock and a stick, you know.

• 42:22Brian Keating?

  Right.

• 42:23Speaker 02

  And, but you should let him speak.

• 42:26Brian Keating?

  Yeah.

• 42:27Speaker 02

  For him, see what he says. But I think that there are fundamentally a few blockers people have when they say, oh, there's nothing to worry about here. One, which is still very common, is just being deeply down a techno skeptic, thinking it's impossible to make AI that is across the board smarter than us. In the US, I think for many people, it has religious roots. So many people tell me that they believe you can only be really smart if you have a soul. Machines don't have souls, end of story. Whereas, computer scientists typically have this different perspective, whether they're religious or not, where they think of intelligence as having to do with information processing.

• 43:09Brian Keating?

  That's right.

• 43:11Speaker 02

  And then it doesn't really matter whether the information is processed by carbon atoms and neurons and brains, or by silicon atoms, you know, in machines. And there's obviously no law of nature saying that this is the biggest neural network you can have, where you can fill up a whole data center now. But that's definitely one of them. And I think as AI actually gets better, that rejection of concern is slowly melting away because a lot of people start getting shocked, oh, my goodness, you know. I was speaking with Jeff Hinton this morning, and I mean, he is very honest about the fact that he used to think this was much farther off. And he's just, since he knows what's actually possible now, has humbly updated. And, yeah.

• 43:56Brian Keating?

  He was also wrong, and I give him credit. You know, you should always give, you know, people have to pounce on people when they admit that they were wrong about something. But he admitted recently, I think, that he was wrong about radiology, you know, going in zero. P doom for radiology was too high. But I want to talk, because we only have 10 minutes left before I talk to your fellow professor, Daron. How do I say his name? Help me with his name. Asimoglu?

• 44:19Speaker 02

  I think Daron Asimoglu. But I'm not the master at Turkish pronunciation.

• 44:26Brian Keating?

  No, I only know one word in Turkish, which is, you know, Bandetirken, which means I'm Turkish. So I'm gonna band with him. But Max, I want to lower your, I want to get to, yeah, I want to get to verified, you know, coding. But before we get there, I have a way to lower P doom. And that's the following. If the universe is teeming with life, you know, as Fermi told my late colleague Herb York and others in 1950, you know, the Fermi paradox, if the universe is teeming with life, you know, what are the odds they're gonna be, you know, spraying, you know, DNA, you know, throughout the universe on meteorites like this that you get at briankeating.com/edu if you have an edu account like-

• 45:02Speaker 02

  A little product placement there.

• 45:04Brian Keating?

  I got my merch, Max, my merch. Actually, it's, you know, it's God's or Mother Nature's or the moon's merch. But tell me, Max, would they really be slinging around their DNA? Or, you know, maybe they've got, you know, XNA or something that we don't understand. But would they sling life 2.0? Or would they shoot out life 3.0? And the fact that, you know, we seem to, you know, it must come into play if life is abundant and flourishes in the universe, whatever form it is, life 2.0, that they would use life 3.0. And yet we don't see much evidence for that. So can that lower your P doom? The non-observation of life 3.0 sprayed out from Proxima Centauri, does that lower your P doom?

• 45:45Speaker 02

  I mean, there's always the possibility that somehow, even if we screw up some lovely, intelligent, and kind aliens from space are gonna come rescue us or whatever. But, you know, we're both fathers, right? And I really believe in personal responsibility. If a fire starts in your kitchen, you're not just gonna sit and twiddle your thumb and go to bed with your kids, saying the fire department is gonna probably come sort this out. You're gonna put it out yourself. And more importantly, you're not gonna start random fires in the first place. And I think we should behave the same way with our planet. I mean, isn't it awesome that we've evolved on this earth and get to have this amazing life here? And

• 46:27Speaker 02

  so even if maybe some aliens could bail us out, shouldn't we be stewards of what we have here and take just good care of what we have and not?

• 46:35Brian Keating?

  Absolutely. But I guess I'm saying something different, Max. I'm saying if life 3.0 is inevitable, and ASI, which is intrinsically unpredictable, according to Roman, and, you know, has this high P doom, then other civilizations will also invent AGI and life 3.0. And then that will destroy them. Is that either a solution to the Fermi paradox? Or is it kind of a non-observation, a no-go theorem that we can use to lower our odds of P doom here?

• 47:02Speaker 00

  The action's on the court. And as the official beer sponsor of the NBA, Michelob Ultra is getting fans closer to the game with a chance to win courtside seats, merch, and more. Visit michelobultra.com/courtside to enter. Michelob Ultra Courtside, 26 to 27. No purchase necessary. Open to US residents 21+. Begins on October 1st, 2026 and ends on June 30th, 2027. Multiple entry periods. See official rules at michelobultra.com/courtside for free entry, entry deadlines, prizes, and details.

• 47:31Speaker 02

  I've thought about that a lot. It would be pretty tragic, ultimate Greek tragedy, if every civilization, once it gets to the point of inventing computers and AI, you know, you can set your timer, and then 50 years later, they're, poof, we're gone. But I think it's very likely that there are ways of making AI that's at least much more powerful than we have now, that is a very useful tool, and-

• 48:01Speaker 02

  And helps that civilization flourish rather than destroy it. I think what I'm unhappy about that we're doing now is not that we're developing technology, but just that we're rushing it recklessly, you know. Do you play chess?

• 48:17Brian Keating?

  Yeah, not not well. My my, you know, nine year old can beat me, but.

• 48:21Speaker 02

  Okay, but suppose suppose you're playing in a against a really strong player and you're up a queen. And then you have three hours left on the chess clock, you know. The one way you can still lose this one game is if you start rushing your moves. You you somehow act like you have only one minute left on the clock and then you blunder this piece, you blunder that piece and suddenly you're lost, right? I feel that this is exactly what humanity is doing right now. You know, we've learned after being on the planet for hundreds of thousands of years that we have a one game. We aren't, we don't have to worry about getting eaten by tigers or starving to death

• 49:03Speaker 02

  or having a life expectancy of twenty nine years because through studying science and technology, we've become so much more empowered that we can be the captains of our own ship. I see the whole human history as a is growing empowerment, right? If we were to follow this now by suddenly building a replacement species and losing control to it, we've just blundered the one game. That's what irks me. So if we simply start treating AI companies like other companies, we have some binding safety standards, it will make us play that make our moves more slowly and deliberately. Companies will release things when they can convince some panel of independent experts with no money in the game that this is more good than bad. So progress continues, but but now you're playing careful chess.

• 49:46Speaker 02

  We will win the one game and then we can have a future where we're free from diseases and poverty. And if you want to go and help life spread out through space and do cool things. And the biggest CMB polarization experiment you could ever imagine, you know, if we go you know, that kind of you can have all that, right? This is the one and the reason I am working so hard to to

• 50:13Speaker 02

  change this better path for AI is not because I'm afraid of death or it's because I think we have a because I don't want to squander this amazing upside that we can have. Can I, I know you need to go to their own, but do you want me to just say something quick also about about the formal verification and how we can have a

• 50:34Brian Keating?

  I do, yeah, there's too much doom. Talk about, you know, Cam, you know, Colm McGor of Arnold Networks. Talk about verified coding. Yeah, give me five minutes on that, please.

• 50:42Speaker 02

  Yeah, I I'll talk about the the talk I gave just the other week with those mathematicians, Terrence Tao and and others in the at the and Jacob Simons and others at at UCLA. So I was talking about how where we are now is we have AI that we cannot trust enough. Swarms break out, the sort of open AI and anthropic and go hack and all sorts of bad things happen. What we all want is controllable tools.

• 51:11Brian Keating?

  Yes.

• 51:12Speaker 02

  And there's the traditional way to try to get controllable tools is to try to understand the artificial neural networks we're working on. I have worked for years in mechanistic interpretability, etc. Many are trying that. I think it's going to be too little, too late, even though scientifically it's just so much fun. But there is a different path to get there, which is much easier and actually much faster, which is you ask the AI that has figured out also the cool algorithms and tools to code this up for you in in the language like Python or Rust or or Lean and then gave you give you a proof that this tool does exactly what you want. So it's like vibe coding, except you have an actual machine checkable proof

• 51:55Speaker 02

  that your tool now does what it wants. I call it very coding, V E R I coding, very formally verified coding. And what's I have a conjecture I want to end by sharing, which I call the asymptotic internal transparency conjecture. And if it's true, we can have an amazing future. So imagine a nerd plot we have on the X axis, intelligence. And on the Y axis, the fraction of of the algorithms and knowledge that can be actually exported into computer code and proven. So for example, so the conjecture first says that this curve is increasing.

• 52:35Speaker 02

  You know, a mouse that's learned a new cool algorithm for finding cheese or something can't code that. It has zero percent exportable fraction because it can't even program, right? You can. If you Brian figure out a new way of of operating this new future CMB polarization satellite, we could, if if we were doing it the way the AI industry is doing it now, what we would do is put you in the satellite with with a lot of food and some oxygen tanks and and say, okay, steer the satellite, Brian. And then if someone is like, ah, I don't know. Can we really trust that he's still going to function after two years there, you know?

• 53:18Speaker 02

  Well, let's do neuroscience on Brian's brain, figure out how Brian's brain works. Smart? There's clearly the better way. We would just say, hey, Brian, you know, I think your exportable fraction is pretty high. Why don't you just take this algorithm that you invented with your neural network and export it into Python or or Lean or something and and give us a proof that it's going to do exactly what we want. And then we launch the computer in the satellite instead. So you can hang out here at home on Earth with your kids and the satellite is going to work great, right? What I'm saying is, and here comes the conjecture, the asymptotic internal transparency conjecture says not only does this can smarter neural networks, artificial neural networks export more

• 54:02Speaker 02

  of their knowledge, but it asymptotes to a hundred percent. So in a future superintelligence or a very, very smart AI that has access to all its own weights, all its own activations, you know, the way a human neuroscientist can only dream about, will be able to figure out enough about its algorithms that it learned that it can actually export it into traditional provable computer code. If this is true, it's amazing because we no longer need to deploy our black box neural networks. We can put them in an air gapped room somewhere. And then all we need to deploy are always this traditional but formally verified computer code that you can, by definition, trust because you can prove it.

• 54:46Speaker 02

  So that's a happy note for me.

• 54:48Brian Keating?

  That's great.

• 54:49Speaker 02

  For me is, and it's feasible, you know, you know, we don't even have to figure out how the AI works. You can code in Python, even though I don't understand how your awesome brain works.

• 54:57Brian Keating?

  I don't, yeah, I don't. I I skipped learning how to code and it was the best thing for me. Now my I I code in English only. Max, I want to thank you so much. We'll talk again, hopefully, maybe on my half birthday next year. And I get, uh, thank you so much, Max.

• 55:11Speaker 02

  Always a pleasure.

• 55:27Speaker 04

  Hey, Bose here. When your next track starts, listen for the decisions that the artist makes. The vocal pushed right up front. The bass that weighs half a beef. That sample from a 70s underground gem. Every choice deliberate. With Bose, you'll hear it all. Your music deserves Bose.

• 55:53Speaker 05

  When you shop pickup at Kroger, you can expect the savings you love and fresh groceries selected just for you. Our associates are committed to getting every detail right, carefully handpicking your items, checking for quality and freshness, and packing your order with care. Because bringing you fresh, quality groceries is what we do best. And right now, enjoy $20 off your first online order of $75 or more. Restrictions apply. See site for details. Kroger. Fresh for everyone.