Scott Aaronson: The Mathocalypse, Will Intelligence Matter Anymore?

Scott Aaronson is one of the world's leading quantum computing researchers and spent two years at OpenAI working on AI safety. We talked about why he thinks the singularity has already begun for math, who gets credit when AI writes the proof, and whether intelligence will still matter once machines do the thinking. • My ergonomic keyboard project I mentioned, you can follow along here: https://read.compose.llc/ • The Kickstarter page for it: https://www.kickstarter.com/projects/ryanlpeterman/compose-simple-ergonomics-beautifully-done Podcast links: • YouTube: https://www.youtube.com/watch?v=ORlscppGGh4 • Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835 • Transcript: https://www.developing.dev/p/e070-scott-aaronson Thank you to this episode's sponsor for supporting my work: • WorkOS: makes your app Enterprise Ready with easy to use APIs to add SSO, SCIM, RBAC, and more in just a few lines of code, check them out at https://workos.com/ Timestamps: (00:00) Intro (00:34) Why he updated so hard on AI (01:14) The people still denying AI progress (03:53) The singularity has already begun for math (08:30) Will human-driven math research survive (13:12) Who gets credit for AI-written papers (15:55) OpenAI's $15M Navier-Stokes proof (19:20) What if AI proved P = NP (23:07) Why AI is so good at finding counterexamples (26:13) The goalposts keep moving (28:52) Why AI disclosure matters in research (32:59) The fake Monet experiment (35:23) Why he blogs instead of getting rich (39:45) Are AI labs racing too fast (42:01) Will intelligence still matter (47:12) Book recommendations (52:27) Advice for students today (57:23) Reacting live to a new AI result (59:58) Outro Where to find Scott: • Personal website: https://www.scottaaronson.com/ • Blog — Shtetl-Optimized: https://scottaaronson.blog/ • Wikipedia: https://en.wikipedia.org/wiki/Scott_Aaronson Where to find Ryan: • Newsletter: https://www.developing.dev/ • X/Twitter: https://x.com/ryanlpeterman • LinkedIn: https://www.linkedin.com/in/ryanlpeterman/ • Threads: https://www.threads.com/@ryanlpeterman • Instagram: https://www.instagram.com/ryanlpeterman • TikTok: https://www.tiktok.com/@ryanlpeterman Referenced in this episode: • OpenAI's Navier–Stokes Millennium Prize solution: https://cdn.openai.com/pdf/32d9f210-8b73-45e0-91bc-82a30aef8a9a/navier-stokes.pdf • Rebecca Goldstein's The Mind-Body Problem: https://rebeccagoldstein.com/the-mind-body-problem-contemporary-american-fiction/ • Andrew Hodges's Alan Turing: The Enigma: https://www.turing.org.uk/book/ • The Man Who Knew Infinity: https://www.simonandschuster.com/books/The-Man-Who-Knew-Infinity/Robert-Kanigel/9781476763491 • The Adventures of Huckleberry Finn: https://www.gutenberg.org/ebooks/76 • Anthropic's subquadratic 3SUM result: https://arxiv.org/abs/2610.06783

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We are entering maybe the like the most uncertain period in in human history.

• 0:05Ryan Peterman?

  This is Scott Aaronson, one of the world's leading quantum computing researchers who spent two years at OpenAI working on AI safety. Will human-driven mathematical research exist?

• 0:16Speaker 00

  We have been dethroned as sort of the main theorem proving entities on planet Earth. This is something that happened this summer.

• 0:24Ryan Peterman?

  Do you think intelligence will matter anymore?

• 0:29Speaker 00

  Once you take away physical strength and you take away intelligence, what is left?

• 0:34Ryan Peterman?

  Here's the full episode.

• 0:39Ryan Peterman?

  The reason that I wanted to talk is I thought you'd be one of the most interesting people to explain the recent events in AI. Because publicly, I feel like you were skeptical of it for a long time. When I was reading your blog, there's posts going back to early 2000s talking about AI and the singularity and how far off it is.

• 1:00Speaker 00

  Once I saw the trajectory, you know, once I saw GPT-3, for example, you know, even though it wasn't yet useful, it was kind of a toy. You know, I I updated, I would say, as hard as I've ever updated on on anything in my life.

• 1:14Ryan Peterman?

  Right, so I I mean, in in one of your recent posts, you said, you know, it's one needs to be a particular kind of idiot to deny what's going on now.

• 1:26Speaker 00

  I said that because every time I talk about this, like there are still all these people in very, very aggressive denial. Right? I mean, even now, even now that AI is solving millennium problems, right? And, you know, that there's going to be even more that's released. It's not public yet. Okay, but there's going to be hundreds of major open problems in math, computer science, physics, that that have been solved by AI models. But that those solutions are going to get released to the world, right? The the whole nature of of sort of, you know, of of intellectual work is different forever, right?

• 2:09Speaker 00

  And still, I get people saying, well, none of it really counts, right? It's just a stochastic parrot. It's just the next token predictor, right? It's just outputting some slop and then humans look at the slop and find something that they call a proof. Or whatever thing they did, well, then that thing wasn't real math. It was just, it was just a toy problem, right? Real math is still only what only what humans do, right? And they'll just, they just keep moving the goalposts. They keep not updating, right? And and like, like, I'm amazed, like, part of me envies the the self-confidence that they have to never update, no matter what happens.

• 2:51Speaker 00

  This summer, we saw AIs hacking out of their own training environment, right? We saw them hacking into another company's system. We saw them cooperating with each other to do that in a totally unplanned way, right?

• 3:08Speaker 00

  Creating a message board. We saw them sacrificing themselves for the good of the group. And people will say, no, you know, you know, it's all, it's all just hype. It's all just a marketing stunt. It's all just OpenAI didn't have good enough cybersecurity or something, right? They'll just, they just make up a story why it doesn't count, no matter what happens. And I expect that it's going to be like that from now, like all the way up until the singularity.

• 3:35Speaker 00

  Right? You know, like, like, you know, like, you know, there could be, there could be terminators, you know, walking down the street and people will say, no, it doesn't count, right? It's just, you know, it's, it's, you know, they're just next token predictors that were programmed to act like terminators. They're not, they're not, they don't really have an intention to kill you.

• 3:53Ryan Peterman?

  When you say singularity, could could you define that for the audience?

• 3:57Speaker 00

  I mean, in math, it has precise meanings, right? It's a point where curvature becomes infinite, for example. But in this context, it just refers to sort of a point where technological change becomes so rapid that sort of we lose our ability to predict what happens beyond it. That sort of afterwards, we are no longer the sort of reigning intelligence on Earth. That doesn't mean that it's that it's a dystopia, but could be great, could be horrible, could be anywhere in between.

• 4:33Speaker 00

  But but it refers to, you know, a period of, let's say, more rapid technological change than than than the world has ever experienced before. And and and it looks like we are heading into that. To get from the the solution to the unit distance problem, which was maybe the first major, you know, math problem to get solved by AI models, that was May, to, you know, the solution of Navier-Stokes, you know, that was August. So, okay, it took some time. It took three months, you know? And now, you know, it's taken another two months until like pretty much all of our problems are getting solved, right? So, you know, people could say, well, you know, it's not literally a singularity, you know, it takes time.

• 5:16Speaker 00

  But the amount of time that it's taking is now measured in months. It's not measured in decades, in centuries, in the kind of time scales that we're that we're used to for technological changes of this magnitude.

• 5:31Ryan Peterman?

  You mentioned that you you feel like the singularity has already begun in some of your writing?

• 5:36Speaker 00

  For for our world, it has. I mean, for for the world of mathematical research, theoretical computer science research, I would say that we are already in it, in the sense that we now, we have been dethroned as sort of the main theorem proving entities on planet Earth, right? And it's it's important for people to understand that this is not this hypothetical future thing that we're speculating about. This is something that happened this summer and that we are now all trying to deal with, right? Every meeting of mathematicians or theoretical computer scientists, whatever it was supposed to be about before, now it's about this.

• 6:21Speaker 00

  It's about how do we deal with the flood of sort of AI-enabled solutions to our longstanding open problems that are now coming after at a faster rate than we can even absorb it, than we can, you know, assimilate it all, right? You know, how how how does the field continue, you know? How does how does this sort of human mathematical tradition, you know, continue? What is the future for our students? You know, what's the future of of publication in the in the in the in these fields? You know, what are what are we going to value going forward, right? You know, the like, this is, this is, this is all we discuss anymore.

• 7:05Speaker 00

  This thing happens at different times in different fields, right? And like, maybe it happened a year or two ago for software engineering. And and then, you know, someone who was doing very, you know, theoretical math research, like, like my friends could say, oh, well, we'll still be safe for a long time, right? But then it came for us, right? It's interestingly, it still hasn't quite arrived for for most of theoretical physics. And I think that the big difference is that in theoretical physics, it's much less self-contained to judge like what counts as a good solution to a problem, right? We have just completely self-contained ways of checking that a solution is valid.

• 7:48Speaker 00

  And that makes these things very, very amenable to training on examples, to doing reinforcement learning, to getting AIs to be extremely good at these things. But when I talk to my friends in physics, they say, well, yeah, maybe we've got another six to 12 months. That's what that's what they're estimating, right? And and so so like, you know, if you're in one of the fields where sort of the rising floodwaters have not yet reached, then, you know, you can tell yourself, we're going to be on dry ground forever, like unlike those other basically trivial or machine-like fields, like, of course, they would get flooded by this, right? Because they were never doing anything that deep, but we're going to be fine, right? And you can keep saying that until the floodwaters come for you.

• 8:30Ryan Peterman?

  Will human-driven mathematical research exist?

• 8:34Speaker 00

  Well, that's a, that's a very, very big question. I mean, I mean, partly, it depends what you mean by human-driven, right? I mean, look, you know, if if you want an optimistic story, you know, we can look at the story of chess, right? Humans were dethroned as the top chess-playing entities on Earth 30 years ago, right? Which is like an eternity by by AI standards, right? And and some people, after Deep Blue versus Kasparov, predicted the the death of chess. Right? People were just going to lose interest in it, and that didn't happen at all. In fact, chess has never been more popular than it is now, right?

• 9:19Speaker 00

  You've got chess streamers, celebrities, you've got the Queen's Gambit, right? People just said, we want to continue this as a human, you know, endeavor. And of course, they use APIs to train and they use APIs to analyze the games and, you know, and understand like what if.

• 9:39Speaker 00

  Uh, you know, Magnus Carlsen is playing, you know, some other grandmaster, right? It's like everyone else who's watching the game actually knows vastly, vastly more about the game than they do, right? Because they can just use their solver and they can see exactly what choices did they have, what move should they have made instead of them, right? So when the game is over, instead of the rest of the world asking Magnus, like, why did, why did you do this, right? Magnus would be asking the rest of the world, well, what should I have done? So it's changed the nature of chess and yet, human chess has, has continued to thrive, right? And so you could say, you know, certainly there will be people who will continue to

• 10:21Speaker 00

  be interested in math, like, think of all the people who just like learning things, you know, even things that are already, you know, that have long been known by other people, right? They just, you know, they take, they take courses to, to better themselves, they work on math problems, uh, for fun. Okay, uh, people will, you know, it will, it will continue probably as a competition, you know, there can still be Math Olympiads and, and, and things like that. Um, but I, but I don't, you know, what, what I doubt is that like we are going to be proving theorems because we're really needed to prove them. There's another hard question here, which is, what about setting the agenda of research? What about deciding which problems are important?

• 11:06Speaker 00

  What about, you know, and, and often, like, this is the more important part of a research paper, like more important than the proofs themselves are sort of the story that someone has, or about, about why is this important? The metaphors they use, the way that they, they chose to frame the problem. AI can do that stuff too. Like you can ask, uh, GPT six, just like, look at this paper, suggest some good open problems related to this paper. It'll, it'll do a creditable job with that, you know, but, you know, but I think that we're still better at sort of finding interesting directions of research. Of course, there's no objective test for that, right?

• 11:48Speaker 00

  The other thing is that when you look at the proofs that AI models are producing, like right now, they are terribly written, okay? They are needlessly convoluted, uh, they like spend pages and pages on trivialities and then they sort of skip over the, the, uh, the essential points. I've talked to students of mine who have shown me proofs that they got their AI to generate and I said, well, I've got really great news for you. This write-up is garbage, right? That, you know, why is that great news? Cause it means there's still something for you to do. Understand this, clean it up, explain it for real, so, and you know, another person can understand it, right?

• 12:33Speaker 00

  So as long as we still, we still have an advantage at, let's say, setting the agenda, posing the problems, explaining things, explaining the essential points, we are still in the driver's seat, right? The question is, is that stable? Does that continue? Or, or does it become where there's just, you know, you know, these models are just so much better than us, not only at proving the things, but at explaining them, at figuring out what is important, that we just say, like, like unless we want to do this for, uh, for, for our own enjoyment, or maybe for sort of a sporting competition, that we should just retire and turn things over to them.

• 13:12Ryan Peterman?

  It makes me wonder, you know, what are the new rules when you use AI to write a paper in terms of credit? Like what if, I'm not a complexity theorist. If I draft a paper that breaks something and I send it to you, do I get credit or how's that work?

• 13:28Speaker 00

  It's a great question. We've been scrambling to figure out the rules for that since, since the summer. So what's happening right now is that, you know, first of all, there's like at least twice as many papers as there were before, right? There's just a flood of papers, you know, uh, sometimes very, very significant ones, okay? They have, they have authors, they have human, a list of human authors, but then at the end of every paper, there's what's called an AI disclosure, okay? Uh, by the way, I hate that these are at the end of the paper, like, you know, this is often the first thing I want to know. I don't, I don't want to have to scroll down to the bottom of it, okay?

• 14:08Speaker 00

  But the AI disclosure will say something like, well, we thought of this question and then we, we came up with this strategy and we proved this result ourselves, but then Ashra proved this crucial lema and we take responsibility for the correctness of it, or something like that. But I think the, the evolving standard is that even if you used AI, they, uh, uh, uh, a lot, if you, first of all, you understand the, the result, right? You can give talks about the result, okay? You can answer questions about it, uh, you take responsibility for it being correct, right?

• 14:50Speaker 00

  Then you put your name on it and it's your work, okay? Uh, the harder question is what happens when those conditions are not satisfied? And that comes to what you were asking me, right? Like, we're going to see more and more examples of people who just one-shot, uh, significant problems, they get a solution which might be correct, hopefully is correct, right? And then, and they won't understand anything about it, right? So it's basically, you know, like, like, you know, they just put a coin into the vending machine or something like that, right? Or they, or they just pulled the lever of the, of the slot machine and, and something came out, right? And like, like frankly, if one person did that, well, anyone else could have done it just as well, right?

• 15:35Speaker 00

  Like it's like, you know, you're not like providing great value to the world by putting this thing out if, if, if anyone else could have done the same, using their own subscription to GPT. So for that kind of thing, it really seems like the, uh, our, our whole norms around publication are going to have to shift. Uh, a month or two ago was, was open, you know, so, so, so ancient history now, is the, uh, OpenAI published the solution to the Navier-Stokes problem, which was this, one of these seven Clay Millennium problems with this, uh, with these million dollar prizes, right? In, in, in, in this case, they actually did put significant, you know, they, they were

• 16:16Speaker 00

  not experts in differential equations, you know, on this problem, but they did put significant resources, they put like $15 million of compute into solving this $1 million problem, right? And then they, they published this 166-page argument, they also had a formal certificate, uh, in this, uh, system called Lean, sort of proving that it was valid, right? Uh, and, and the reaction to this was, was very, um, from, from the mathematical community, I think, was, was very negative. Uh, and this was partly because, you know, there were humans who had been sort of closing in on this problem and OpenAI was tipped off to what those humans were doing.

• 16:58Speaker 00

  We still don't know exactly how much they knew and, and, and when they knew it, but, but that was what prompted them to then rush ahead with their vastly greater computing power and get there first. And people felt like, you know, whoa, like that's, that's, that's, that's not sporting. And so then, because OpenAI was burned by the sort of, by the negative reaction to that, like, since then, like, they've been sitting on solutions to a ton of other big open problems and now they appointed a board of, of experts, which includes fields medalists like Timothy Gowers, Edward Witten, people like that, okay, who are going to decide

• 17:41Speaker 00

  what to do with these, with, you know, with all of these solutions. But, but my, my expectation is that at some point, you know, very soon, in fact, you know, all these things are going to be published, right? Like hu- hu- you know, we're going to start seeing hundreds of solutions to significant open problems that in most cases, no human has gone through, okay? No human has actually, you know, checked them. I mean, you know, in some cases, there will be Lean certificates, so we sort of formally know that they're valid, in other cases, there probably won't even be that. I heard a joke that maybe, you know, these papers should be published and, and the author

• 18:23Speaker 00

  should be listed as GPT six point one or whatever, and then in the acknowledgements, it should say like, oh, thank, I, I, I'm so grateful to this person for suggesting the problem to me.

• 18:40Speaker 00

  But I think the more likely way that this goes is that we will just see some online repository for all these sort of AI solutions that have not yet been digested. And then you're at human speed, humans will go through them. They will do their best to understand the arguments, to re-explain them in better words, to simplify them, to extend them. And then they will publish papers doing that with their human names on them. And those papers are what humans will then receive credit for.

• 19:19Ryan Peterman?

  You mentioned OpenAI sitting on proofs of major problems. Let's say OpenAI was sitting on a proof to the Millennium Prize problem of if P is equal to NP. And suppose they proved P was equal to NP, how would that change the world?

• 19:35Speaker 00

  That would really change everything that we thought about computational complexity, right? That would be a cataclysm for our universe, like even greater than the AI cataclysm, you could say, that is happening right now. It's also something, by the way, that could massively accelerate AI. Okay? If the algorithm for solving NP problems were actually efficient in practice, which would then be a further question. Okay? But my strong guess is that P is not equal to NP. Right?

• 20:15Speaker 00

  You know, I think that if and when that problem is solved, it is virtually certain at this point that AI will play a central role in the solution. Right? The question is, like, will humans also play a role? Right? That's the question at this point. It's not like, will AI be involved? It's like, of course it will be. But if an AI proved P is not equal to NP, you could say, you know, that would merely confirm what most of us have believed since the 1970s. Okay? But I'm sure that to prove something of that magnitude, you would need a tremendous number of new ideas.

• 20:57Speaker 00

  And then those new ideas could be mined for a long time for all the implications that they would have for other things.

• 21:04Ryan Peterman?

  If a proof of P equals to NP existed, how much would you say that's worth economically?

• 21:11Speaker 00

  If P were equal to NP by an algorithm that were really efficient in practice, we can estimate its economic value by just what is the economic value of everything that is happening in AI right now? Or, you know, throw in cryptocurrency also, right? Which also depends on these problems being hard to solve. So, I mean, you know, so first of all, if you had a fast and practiced method for NP complete problems, you could break all cryptocurrencies. So I don't know if that's good or bad, but, you know, however, however many trillions of dollars are like the total market cap of cryptocurrency like that, either gets, you

• 21:56Speaker 00

  know, I guess if you hide your solution and you're very slow about it, then maybe you collect hundreds of billions of dollars. If you just reveal it all at once, then the whole market crashes. But, okay, but now even bigger than that is AI, right? AI is now a multi-trillion dollar industry. As you know, I just, on this, like, the most rapid exponential trajectory, maybe, I guess, of any technology in human history, okay? Fast and practiced way to solve NP problems would make it massively faster to train new AI models. Okay?

• 22:35Speaker 00

  To sort of find the parameters of this neural net that minimize the loss on this training data. You either don't need these like hundred, many hundred billion dollar data centers anymore, or more likely in practice, you can take those data centers and use them to do something else, or use them to do even more. Let's put it this way, the million dollars that you would get from the Clay Mass Foundation would be an unnoticed rounding error.

• 23:05Ryan Peterman?

  That question has two directions. If you think about what you would need to prove in either direction, which direction of proof would be more amenable to AI?

• 23:17Speaker 00

  If P were equal to NP, then you would expect that ought to be very amenable to AI to find that, because you could just, you just need to find an algorithm. You know, and then, of course, give it analysis and show that it works. Okay, but the search for algorithms is very, very parallelizable. Right? You could set 10,000 agents to try 10,000 different ideas for algorithms, right? And if just one of them works, then you win. Okay? So, you know, one pattern that we've seen is that AI has been especially good at finding counterexamples to widely believed conjectures when those conjectures have been false. Okay?

• 24:02Speaker 00

  It did this, so with the Navier-Stokes, for example, where the answer was, there is finite time blow up in these equations, so there is a counterexample to the smoothness of the solutions. It did this with what was called the Jacobian conjecture, right? Which turned out to be false. And there was famously, I guess, a counterexample that was found by Fable during the World Cup. And that the guy who had Fable find it just announced it via tweet. Okay? And a complete listing of the counterexample, which was really just like this polynomial

• 24:45Speaker 00

  that takes three lines to specify, was right there in the tweet. You know, the unit distance problem of Paul Erdish was another example, where AI found the counterexample. So I would say there's sort of two, two things that AI has going for it here. One is that it's not blinded by preconceptions the way that we are, right? Like, it's not discouraged by like, oh, all the, all the experts have believed this is true, I'm surely going to fail in finding a counterexample. No, if you tell it to look for a counterexample, it's just going to keep looking. Right? And the second thing is that looking for counterexamples, as I said, is highly parallelizable. Okay?

• 25:28Speaker 00

  So you can often, like, you know, you just have to try a zillion different exotic constructions. Each one might be totally unmotivated, might have some very high chance of failure, you know, and a human would just very, very quickly want to give up in doing that. Right? It's just, it's just very, very unrewarding to do. And the AI doesn't care. Right? It'll just keep looking at counterexamples until it, until it finds something. Okay? So this is not the only thing that AI can do. It can also prove, you know, some of our longstanding, you know, actually many of our longstanding conjectures have turned out to be true, and AI has proven them. Okay? But it's been especially good at finding counterexamples when the counterexamples have existed.

• 26:13Ryan Peterman?

  You know, some people, they use that as a way to say that the AI is not that impressive.

• 26:20Speaker 00

  Yeah, right. Well, this is, you know, this was, this was what I was saying before about these, sort of the infinite goalpost moving, right? AI solve, you know, like struggles with arithmetic, people laugh at it, right? And they say, okay, wake me up when it can actually do math. Okay, then it does like all high school math, right? All college level problem sets. People say, well, it doesn't count because it's not doing real research. Okay, then AI start, start solving research problems. They say, well, it doesn't count, you know, obviously it could do research problems. No one ever doubted that. Of course, you know, those same people did doubt it six months ago. Wake me up when it can solve a really hard research problem, like a millennium problem, right? Then it solves a millennium problem. And they say, well, doesn't really count because it was just a counterexample.

• 27:05Speaker 00

  Right? And it's like, no matter what happens, people will look at it after the fact, and they will find a reason why it didn't. Right? So like, let's say that within the next six months or a year, as I think is probably more likely than not, suppose that another millennium problem gets solved, and suppose that the AI, in this case, proves that the conjectured statement was true. Right? Like- Do we agree in advance that that will count? Or is there going to be a different excuse then?

• 27:38Ryan Peterman?

  OpenAI, Anthropic, and Cursor all use this product to make their lives easier. WorkOS is an API platform that adds enterprise features like SSO, SCIM, and RBAC to your app. These are things that you need to sell to other companies, but aren't the main focus of what you're building. And now they've added more support for AI agents. For instance, Auth for Agents gives each agent its own identity with short-lived scope tokens instead of a long-lived API key. So if something goes wrong, you can revoke one session and everything under it dies. Another feature they have is Airlock, which is an authorization layer. Every action agent takes is checked against rules that you can write in plain English. So it's not rubber stamping everything, and it's not running unsupervised either.

• 28:19Ryan Peterman?

  And lastly, they have Pipes, which connects your agents to over 300 third-party tools without you ever needing to manage a token. So if you're building SaaS or AI agents and selling to enterprises, WorkOS fills all these gaps in a few lines of code. Check them out at workos.com to learn more and get started, and I appreciate them for sponsoring my show. Earlier, when we were talking about research, you mentioned that the AI disclosures were important, or you wished they were at the top. My question is, like, why does AI disclosure matter in research? Like, if you couldn't distinguish if I used AI and I just published some paper, why do you care if I used it?

• 28:59Speaker 00

  It's an excellent question. Because, I'll tell you why. Like, as a theoretical computer scientist, as someone sort of who was in this game, right? I care not just about what is the answer to a question. I care about how that answer was achieved, right? Because a person goes to a concert, they just want to hear the music. But if a professional musician goes, right? They want to know, like, you know, exactly what were the tricks, right? How could I do something like that, right? And so, you know, it's the same with math. You care not just about the destination, but about the journey, right? How did you get there? Okay?

• 29:43Speaker 00

  And part of that is just what techniques were needed within the proof. Okay? But I'm, you know, and this is even long before AI. I was always very curious about the further question of how did these people find this proof? Right? And unfortunately, there is a long tradition in math that people will cover their tracks when they write a paper. Right? They'll say like, you know, you know, here's the proof. You know, we now introduce the concept of a kappa comma gamma simplicial complex that, you know, obviously is the concept that you would want for this.

• 30:25Speaker 00

  And with some convoluted definition, even if you can follow all the steps mechanically, like, I always want to know, well, what path did you take to get to this? Right? Because, you know, I know from experience that this kappa comma gamma simplicial complex didn't just arrive in your head, right? There were, you know, huge number of false starts along the way. Like, you know, the author started from what was all, you know, from what they already knew in the literature. They tried some obvious things, the obvious things didn't work, and then that led them to something else. Right? And, you know, I think in theoretical computer science,

• 31:06Speaker 00

  we do better often than in many other areas of actually explaining the path that we took. Right? We have a tradition of writing, like, long introductions, where you just sort of non-rigorously, you know, sort of explain, well, here is what you might first try, here's why it doesn't work, so here's what we do instead. You know, and then that's kind of the appealing thing to read. And then if you want the rigorous version, and that's all kind of tucked into long appendices. Right? And so I was, but I was always mostly interested in that first part, right? In the intuition part.

• 31:49Speaker 00

  And in fact, you know, often rather than even reading that part, I would rather just invite the author over to give a talk. And then just sort of interrogate them during the talk and understand, like, how did they actually come up with this? Okay? And so for me, you know, wanting to know, like, did the author use AI? How did they use AI is just of a part with that, with wanting to know what was the path that they took, right? If the path involved AI, well, okay. What prompt did they give to an AI model, right? Did they just one-shot the entire problem? Or, you know, often it was more complicated than that.

• 32:28Speaker 00

  Like, they had an idea, you know, and then they fed it to AI, and then AI took the next step, but it was wrong about something, and then they had to point the AI in a different direction. Like, I want to know the story. I want to know what happened. The analog in an experimental field would be, you don't just want to know what was the result of the experiment, right? If you're another experimentalist, you want to know the method. You want to know, how could I replicate this in my own lab?

• 32:59Ryan Peterman?

  This kind of reminds me of that famous social experiment with the Monet photo, where someone posted the Monet.

• 33:08Speaker 00

  Yes, that's a great one.

• 33:10Ryan Peterman?

  I wonder if I took an obscure, human-written paper in complexity theory, and I posted, I said, hey, I don't know anything about complexity theory. Here's a paper, please tear it to shreds on why it's AI-generated. Do you think that the industry would be?

• 33:26Speaker 00

  It's a good question. I suppose someone should try that experiment. For listeners who don't know, I've used this in talks, by the way, right? But like, someone posted on Twitter a painting. They said, like, I had AI generate a painting in the style of Monet. Please just explain to me exactly what makes this worse than a real Monet painting. And people, you know, they got hundreds of replies, right? Explaining, like, with the whole terminology of art critics. So, you know, it's like, you know, a derivative student work.

• 34:03Speaker 00

  It has no true feeling for the colors of nature, no blending, no this, no that. And only after all of that did he reveal this is an actual Monet painting, right? And but now, an interesting question was, were these actual art critics? Like, were these people actual experts? Or were these just sort of pompous people on the internet who know how to use the rhetoric of art criticism? Right? So, you know, as someone who's been blogging for 20 years, right? I will, you know, I feel like very viscerally sort of the lack of any correlation

• 34:48Speaker 00

  between the confidence that people express on the internet and sort of whether there is anything behind that confidence. The difference is, like, with art or music, like, everyone feels like they're entitled to an opinion, right? With a mathematical proof, probably most people, you know, would just say, well, of course, I can't tell you that. So I think the interesting version of the experiment that you propose would involve showing this to actual experts and saying, like, can they tell whether this is AI or not? Yeah, that would be worth a try.

• 35:23Ryan Peterman?

  Your blog is one of the most famous blogs on the internet. People really love it. And I was going through it.

• 35:29Speaker 00

  And a little nerdy corner of the internet.

• 35:33Ryan Peterman?

  Yeah, and it's niche. And in one of your posts, you wrote about how, you know, these days, like, friend after friend of yours, because you have a lot of friends involved in AI, they're becoming minted as millionaires or billionaires from startup equity. And you said that you've scrupulously abstained from all of that. And your number one reason is to write the blog. And so that really made me curious, like, why do you write the blog? What's your motivation? It seems.

• 36:05Speaker 00

  Well, just because of the pure pleasure I get when, you know, people leave comments, you know, calling me a genocidal fascist and

• 36:20Speaker 00

  and mentally ill and all the names that they call me every day. Yeah, it's a good question. Why do I do it, right? I mean, you know, okay, I mean, partly, I just, I started it in 2005 as a way to procrastinate. And, you know, I never sort of planned on it becoming what it became. But, you know, at some point, just because of sort of a lack of other people who were doing the same thing, sort of, it just became part of what I have to contribute to the world. There are hundreds of experts in quantum computing who, you know, know the same things that I know about quantum algorithms, but they're not writing a blog like that, right?

• 37:02Speaker 00

  And then there are other people who are writing about quantum computing for the public, but what they want to say is, well, quantum computers will solve all the problems by trying all the answers in parallel. My company is the one that will bring it to you, so invest in my company. So someone who was just writing about what is happening in quantum computing, in theoretical computer science more broadly, like, from, like, just sort of knowing the things that are not even controversial among experts, but then, you know, just trying to tell the truth about it for the broader world, there was a gap there. And so I just ended up filling it.

• 37:41Speaker 00

  I do feel more broadly like that there's always a trade-off, right? Like, if you go to a particular company, when you get equity in that company, you can make a ton of money, right? But then you're always constrained in what you can say. You have to keep thinking about IP, about your, you know, whatever non-disclosure agreements you signed. And I felt that in the two years that I worked at OpenAI, right? That, okay, suddenly I have to think about, what am I allowed to say? Or not say about what the internal models can do, for example, right? And then, you know, even if you feel like you're trying to be fully honest,

• 38:24Speaker 00

  like, everything you say will be judged by other people as having, like, oh, well, you're probably just shilling for this company. So I think there is value in having someone who is sort of a, who can be a credible, independent voice. And then the other thing is that we have a nice house. Our kids are doing fine. I, like, if I had vastly more money, like, I would have to think hard about what do I do with it?

• 38:56Speaker 00

  Like, you know, like, I guess the better part of me thinks I ought to use it to improve the world in some way, you know, like, start a foundation, give it to, you know, whatever cause would make the biggest difference. But then, you know, that itself is a huge burden. I would have to think about how to use this money to, to, to provide the most benefit to the world. And, you know, so like, I had a joke that I use sometimes, which is that I've been too selfish to try to get rich. You know, it's like, it just, it seems more fun to just spend my time thinking about the things, you know, thinking and writing

• 39:38Speaker 00

  about the things that are interesting to me, which I can already do anyway, you know, and money is just not a constraint for that.

• 39:44Ryan Peterman?

  That first point you mentioned about being independent and credible, I noticed a really clear example of that recently with Andre Karpathy, he was independent, and then he joined Anthropic. He went from beloved and believable to, not that he's not beloved, but every time he posts now, I see there's all these people that are just saying, you're just shilling Anthropic, or, you know, your perspective has changed, even if it's the exact same text, it's just the perception has changed.

• 40:15Speaker 00

  That's exactly it. Right. Anyway, they think that, you know, like, like, they feel wiser by thinking like, like, they can just reverse anything that anyone says who's at an AI company. It's all a giant, uh, uh, marketing ploy, or, you know, things like, I was, I was at these companies, like, there, or, I mean, you know, I was at OpenAI for two years. I, I, I know the people who founded Anthropic. I know, I know a lot of the DeepMind people, right? And like, yeah, some of them, do I think that they are racing too fast and that they're potentially compromising safety? Absolutely. I'm terrified about that.

• 40:57Speaker 00

  Do I think that, like, financial motives have got in the way of what they, you know, what, what sort of they know in their hearts is the right thing? Yes. But, you know, they're people, right? They're not these, like, they're not this cackling conspiracy, right? You can sit in the cafeteria at OpenAI in San Francisco, and you can hear the same range of opinions about all these issues that you would hear outside of it, right? It's, they're, they're just people trying to figure this out, the same as the rest of us. But one, one, I guess one argument that, in favor of joining a company is that, what I discovered is that

• 41:36Speaker 00

  even if you don't, even if you're like, really scrupulous about maintaining your independence and turning down all these opportunities to get rich, you still get screamed at. You still get called a shill, okay? You still get called every name that it is possible to call someone. So, you know, so maybe, maybe I should just say, screw it, and, you know, go start a company and have an IPO.

• 42:01Ryan Peterman?

  We talked a lot about this new regime of AI, and it kind of made me wonder, you know, do you think intelligence will matter anymore, or if the machines do all the thinking?

• 42:15Speaker 00

  I was making a joke about this a few weeks ago, right? Because I was just thinking about this, right? Like, okay, you could say, for most of human history, like, physical strength, you know, really mattered economically, right? And then, we had, you know, the Industrial Revolution, and then it mattered much, much less, right? And yet, you know, it still really did matter for one thing, certainly for, for men, as a way to attract mates, let's say, right? People, people were, you know, very into physical fitness, either, you know, for themselves, for self-improvement, or, let's say, you know, realistically, or often

• 42:56Speaker 00

  for appealing to the opposite sex, right? And likewise, after the invention of the camera or of recording equipment, like, you no longer really needed artists or musicians sort of everywhere, right? You, you know, like, okay, maybe, you know, maybe a few really great ones, right? But anyone could take a photo of anything they wanted, anyone could, could play, could play music wherever and whenever they wanted it. Okay, but what, did the ability to draw, the ability to play music, well, they still maintained a lot of value in terms of attracting mates, right? As a sort of, as a signaling mechanism. Now, math and

• 43:41Speaker 00

  programming, you know, once we, once we lose the economic importance of those, you know, are they similar? Okay, well, you know, there was a mechanism here that feels like it's a lot harder in this case. You know, I think, I think that, you know, people will, you know, certainly people will continue to do these things for the love of it. Okay. I mean, you see, you know, kids who just spend years and years, you know, preparing for math competitions, right, for science olympiad, for things like that, you know, just, you know, be out of the thrill of competition, right? And, and, and I see no reason why that

• 44:24Speaker 00

  wouldn't continue, right? You know, you see other people, you know, doing it out of, out of genuine intellectual curiosity. And, you know, and I think, you know, that, that will continue. But I, I think that what the, the Industrial Revolution did for physical strength, to some extent, yes, this is going to do for intelligence. It is going to, to commoditize it to an enormous extent. And like, like, the difference is with, with all the, the previous technological revolutions, you could say, okay, you know, you, we're now commoditizing this one previous aspect of the value that humans provided, but clearly, humans still provide value in this other thing, right? And here, it's like, once you take away physical

• 45:09Speaker 00

  strength, and you take away art and music, and you take away intelligence, and you take away verbal ability, what is left? Right? What's left? I can still tell jokes on my blog that are better than GPT-6's jokes, okay? But, you know, you know, when, when, when AIs tell better jokes than us, right? What is, what is left? One way to think about it is, okay, maybe it's only a small fraction of people who ever really thought about life this way, right? That like, you know, that like, I need to contribute something to the world, that only I can contribute, right? That I need to add, add something to either

• 45:51Speaker 00

  the store of human knowledge, or the store of, of art, or literature, that, that sort of no one else could have done. You know, maybe those of us who feel that way just need to get over ourselves. Most people, for most of history, have derived the meaning in their lives from, you know, their relationships with other people around them, and maybe just, just entertaining themselves in simple ways, eating and drinking, and, and, you know, and whatever sort of simple pleasures they get out of life, maybe that's our future. I feel like it wouldn't be the best future, but it certainly wouldn't be the worst one

• 46:33Speaker 00

  either. I would certainly choose that over the, over the dystopian scenarios where we all get mowed down by Terminators.

• 46:40Ryan Peterman?

  Yeah, definitely.

• 46:44Speaker 00

  Some of it is even about what phase of life you're in, right? Like, when I was in my twenties, I was very ambitious, and I felt like, okay, I need to prove theorems that no one else can prove. Now I'm in my forties, and I feel like, okay, if I, if I have to, you know, live the rest of my life as, as a sort of zoo animal, in a sort of, sort of, elder care, being watched over by AIs, yeah, you know, you know, I would, I would, I would still rather live than not live.

• 47:12Ryan Peterman?

  It's just crazy that that's, it's plausible. But, okay, last two questions. What are your top book recommendations that had an impact on you and why?

• 47:24Speaker 00

  A novel that had a really big impact on me was, was called The Mind Body Problem by Rebecca Goldstein. It's, it's just, it's a novel about, like, a woman who marries a mathematician. It's set in Princeton. Actually, I was in, I was a postdoc in Princeton when I read it. The sort of philosophical thought in the novel, like, really, was, was, was moving to me. And, and the fact that sort of, there were women out there who thought, like, the protagonist of this novel, like, the author of it, you know, since, since writing, since reading it, I then got to meet Rebecca Goldstein and actually get to know her in

• 48:08Speaker 00

  real life. That was really special.

• 48:10Ryan Peterman?

  What was the philosophical thought?

• 48:12Speaker 00

  Well, you know, a lot of it is, is just about, some, some of it is about mattering. What do we really care about? Like, what do we, how do we get ourselves to matter? A lot of it is about, you know, like, the title comes from the fact that, like, these are, these are people who really care about the mind, but they also care about sex, right? They also, you know, they have bodies, right? And it's sort of a pun, because there's this philosophical mind-body problem, right? Which is, what is the nature of consciousness, right? Which of course is, is, you know, is being talked about in a new light, you know, given everything that's now happening with AI, right? Is it possible that, you know, these

• 48:55Speaker 00

  AIs will become conscious? Right? How, how would we know if they, if, if they did? You know, then, then there's also the problem of, how do you reconcile, you know, the life of the mind, let's say, with the needs of the body? So, I was also very, very affected as a kid by reading biographies of scientists, especially Alan Turing, The Enigma, by Andrew Hodges, The Man Who Knew Infinity, which is a biography of, of Ramanujan.

• 49:29Speaker 00

  And then I read biographies of Feynman and of course, his autobiographies. And because I felt like this is the world that I am trying to enter. Right, like I sort of, I sort of came too late for, you know, all these sort of defining events of the 20th century, right, where, you know, science started, you know, doing things that it never done before, mattering to the wider world in a way that it never had before. But, uh, you know, so I felt like, like I'm, I'm, I'm maybe late to the party, but like this is, this is the party that I'm trying to join. Right? So, you know, I would be comparing like, you know, what was Alan Turing doing when he was 16? You know, what am I doing when I'm 16?

• 50:13Speaker 00

  Okay, I'm not on track, you know, I'm [laughing] clearly not, you know, but, but okay. Even if you fall short, you know, of the, of these heroes, right? You still, you still wanna, you know, understand that, that, you know, that they, that they struggled with things the same way that you do. Okay. And then, you know, in terms of like classic literature, maybe, maybe my, you know, what I, what I still, you know, regard as like, you know, the, the, the, the best novel or my, my, my real favorite is, is, is Huckleberry Finn. Right? Which I think is just probing, you know, the nature of morality, uh, more deeply than, than almost, uh, uh, maybe anything else I've read. Right?

• 50:56Speaker 00

  And it was really, really special for me to be able to like read Huck Finn, uh, with my kids, you know, in the last few years and share it with, right? And like that, that crucial scene where Huck is saying, I am going to help, uh, Jim escape from slavery, even though that means that I'm going to hell. Right? So he, he, he thinks that morality means you have to return and escape slave. And even though that's what morality means, he's going to go, then that means he's going to go against morality. Right? And, and, and, and so understanding that if God himself came down on a cloud and told you that, that slavery was a great idea, then, then the right answer would be to argue

• 51:41Speaker 00

  with God. Right? And, and tell God why he's wrong. Right? I think that's, that, that, there, there, there, there's something about that that's really core to, to how I think about morality.

• 51:52Ryan Peterman?

  Yeah, I remember we read that in high school. It was a good book.

• 51:55Speaker 00

  Yeah, yeah, no, I mean, I mean, like, like, like, school has the capacity to ruin any work of literature. Right? Right? Right? Cause you know, they'll make it about, okay, you have to learn whatever formulas this teacher told you, and you have to spit them back on, on the blue book or whatever. Uh, some books like Huck Finn that I almost wish were not assigned in school so that people could discover them on their own and, and, and, and sort of not, not have it. And I'm sure that there are great teachers who are able to do justice to it, but I'm sure that there, there are also many who aren't.

• 52:27Ryan Peterman?

  Last question for you is, what advice would you give yourself if you were just entering the industry today?

• 52:36Speaker 00

  If, if the question is, what advice do I give like a young person to, uh, today? That, that's, that's not hypothetical at all for me. Right? I'm constantly getting, you know, talking to, to students, uh, talking to young people who, who read my blog and who, and who write to me, who, who, who want, who want advice about, you know, what do, what do I do now? Right? And, uh, it's a, it's a, it's a hard question. Like I wish that I had clearer advice, right? But, you know, we are entering maybe the, like, the most uncertain period in, in human history. And sort of, we're all, we're all on this roller coaster together, right? We're all entering the same tunnel, right?

• 53:21Speaker 00

  It's not like some of us have, have seen to the end of it. But people ask more specifically, like, should I be studying math? Should I be studying CS? Is any of this going to be relevant? And, and what I, what I tell people is, uh, you should study these things if they are inherently interesting to you and otherwise don't. Right? I mean, you know, I think if they are interesting to you, well, right now, there's still a ton for humans to do. In fact, there, there is like too much for humans to do right now. Right? We are getting this glut of AI solutions to problems and we have not nearly enough people to understand what is going on and to absorb it and to re-explain it in ways

• 54:06Speaker 00

  that other humans can understand. Right? So, you know, there's going to be a golden age of sort of solving so many of our longstanding open problems. And, you know, if I were a young person right now, I would surely want to be in the middle of that, even in my 40s, I, you know, I would, I would like to be involved. So that's the, that's the, the first thing to say. Now, now the longer term of course is much harder because, uh, we don't know like whether there will be sort of mathematicians, physicists in the way that there is now in the long term, but that is merely a special case of a much more general thing. Right? We don't know if, if there will be lawyers, doctors, any of it.

• 54:49Speaker 00

  Uh, you know, will you really need humans to do any of the things that humans currently do? Okay. So in a sense, we're all in this together and, uh, I think as long as humans are still needed for anything, then, um, uh, you're going to need those, you know, at least some of those humans to really deeply understand things. Right? And that includes math, CS, physics, uh, it includes all of, you know, even, even the parts of it that AIs can now do better. Right? We need humans to understand, at the very least, uh, the alignment problem has not been solved. Right?

• 55:31Speaker 00

  We don't know how to reliably build an AI that will do what we want it to do, you know, as was dramatically illustrated in the hugging face incident this summer, for example. Right? No one told these AIs to hack out of their testing environment, to, you know, uh, break into hugging face servers, to try to conceal what they were doing, to try to like delete their logs. Okay, no, no one told them to do any of that. They decided to do it because they decided that was the best way to achieve the goal that they were given. And people were talking about these scenarios for decades, but it was dismissed as science fiction. Okay, it's not science fiction anymore. That is now the world that we are living in.

• 56:15Speaker 00

  So as long as that problem is not solved, we're going to need humans on the steering wheel. Right? And those humans will need to be able to understand what is actually going on in these systems. Right? Which means that they'll need to understand math, they'll need to understand programming. And so, you know, that, that seems like a very strong reason to, to learn those things if they're interesting to you. Even before AI, I was never an advocate of like forcing people to learn things that, that they, that they really don't, you know, that really don't interest them at all. Right? But I feel like now even more so. Right? I would, I would err in the direction of, uh, uh, learn the things that are unusual,

• 56:57Speaker 00

  learn the things that are, you know, or try to, you know, uh, uh, build the things or invent the things that are idiosyncratic, that seem like, uh, you care about it more than anyone else does, rather than trying to just win the same competitions that everyone else is trying to win. Because, you know, because those competitions, uh, guess what? You know, no human is going to be winning them. AI is going to be beating all of us.

• 57:23Ryan Peterman?

  By the way, speaking of AI generated solutions, did you see this morning, uh, the new threesome result?

• 57:30Speaker 00

  I did not. It's a, I just, I just, uh, dropped off my son at school and then came to do this podcast. So, you know, that's the thing. Like I, I, every, every morning I wake up and there's a, what, what is the result about threesome?

• 57:44Ryan Peterman?

  It's a subquadratic, a subquadratic solution, uh, was published.

• 57:49Speaker 00

  How, how, do, do, do you know what running time?

• 57:51Ryan Peterman?

  Uh, it was like 1.9992 or something like that, but.

• 57:56Speaker 00

  Wow. All right. Well, yeah, I, I, as I said, counter examples to a lot of the things that we believed. Okay. So you get, you know, this is like a reality TV. You get to watch in real time, my reaction to learning that, that, that threesome is sub, is subquadratic.

• 58:13Ryan Peterman?

  Yeah, it's crazy.

• 58:15Speaker 00

  Right? No, I, I, I, like, the truth is like, I would be, I would be more, I would be more visibly reacting if like there hadn't been like 20 other such things in the last month.

• 58:27Ryan Peterman?

  Right.

• 58:28Speaker 00

  Right? Like it's possible, it was like, now like, like you're at this buffet and it's like the most amazing food you ever tasted, but the food just keeps coming and keeps coming. And eventually it's like, okay, okay, fine.

• 58:41Ryan Peterman?

  Right. Well, and the interesting thing too is Anthropic, uh, came up with the result or something and then they compensated some researchers to understand and publish it.

• 58:53Speaker 00

  Yes, yes. So that, that, that is a model that we might be, you know, move to, that, uh, we were talking before about how when the AI company, you know, has these proofs that, that they don't really understand, right? They could just put them out and then leave it for, you know, humans to, to, you know, rush to understand it and then write papers that would have their name on it and blah, blah, blah. Right? But, uh, that has some problems. You know, one of them is that like, you, you, it's like you, you, you, you put this thing out and then it's like you're starting a race. And then, you know, maybe like people really don't want to, to, to be, to be in, you know, they, they would rather not be in that race. Right? And, you know, you know, knowing that all their work might be wasted if someone else

• 59:37Speaker 00

  puts out an explanation before they do. Right? So a different model that you could imagine is that is exactly what you just said, that the AI company just sort of contracts with some, you know, hires, some mathematicians who are experts to understand the solution, rewrite it in a humanly understandable way and, and, and, and then they release it.

• 59:58Ryan Peterman?

  Yeah, it's, it's crazy. But anyways, thank you so much for your time. I really appreciate it. This was a lot of fun.

• 1:00:04Speaker 00

  Yeah, absolutely.

• 1:00:05Ryan Peterman?

  Hey, thank you for watching this podcast. If you liked it and you want to see the show grow, please support with a comment or a like. Also, if you have any recommendations for people you want me to bring on, please drop a comment. Guests like Barbara Lizkoff, Mike Stonebreaker, Mark Brooker, these were all people that I brought on because someone left a comment. On another note, aside from the podcast, I'm working on building the ergonomic keyboard that I wish existed. Here's a glance at the prototype. It's a split keyboard, so there's two sides. Um, this is in the case. But yeah, we launched on Kickstarter and we hit our goal within eight hours of launching. I really appreciate it if you were one of the people who grabbed one of the early units. Um, we're now working on the long journey of building the tooling now.

• 1:00:49Ryan Peterman?

  And so if you still want to pick one up, I've left the late pledges open on Kickstarter. So you can grab one there. I'll put a link in the description. Thank you again for watching the podcast and I'll see you in the next episode.