Science Friday
Hosted by Flora Lichtman and Ira Flatow
A few years ago, ChatGPT couldn’t do simple arithmetic. Now, some experts say that AI could make mathematicians obsolete.
3,479 words · 7 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
WNYC is supported by Bear. Science is about asking questions, testing, trying again. It's not easy, but it's relentless. It evolves, and that's why science delivers. Learn more at sciencedelivers.com.
Science Friday is supported by MathWorks, creator of MATLAB and Simulink software for technical computing and model-based design. MathWorks, accelerating the pace of discovery in engineering and science. Learn more at mathworks.com.
Hey, I'm Flor Lexman, and you're listening to Science Friday. When ChatGPT came out a few years ago, one of its most impressive features was its writing abilities. But something it was often ridiculed for, you may remember, other than making images of seven fingered hands, was its inability to do basic math. Two plus two did not always equal four. But recently, things have changed. Last year, Google and OpenAI's models bagged gold medals at the International Mathematical Olympiad, edging out some of the best high school mathletes in the world. And now, some experts say AI is getting so good, it could pose an existential threat to the field.
Here with a perspective are two mathematicians who've thought a lot about this, Dr. Emily Reel from Johns Hopkins University and Dr. Daniel Litt from the University of Toronto. Emily, Daniel, welcome to Science Friday.
Hey. Thank you.
All right. We have seen, at least in the nerdy media that I consume, these very big claims that AI suddenly is revolutionizing math. It's fundamentally changing what it means to be a mathematician. I want your takes on that. Emily, let's start with you.
I think from the perspective of a professional mathematician, we're trying to evaluate where AI is on kind of the trajectory of a mathematician's life. So, you know, we're first encounter to mathematics at school where you're solving problems that have a numerical answer, maybe involving some geometric figures, maybe involving some arithmetic or some algebra. As you mentioned already, AIs used to be very bad at those problems and they're they're much better now. More recently, AIs have been doing well with more advanced contest level problems, contests for undergraduate mathematics majors where they're writing proofs. And what's exciting in perhaps scary in this past year is they're
starting to be tested on more research level problems, so problems that are of interest to people who get paid to do mathematics.
Daniel, what do you think?
Yeah, so the specific claim that AI tools are revolutionizing mathematics definitely has not come true yet. I think there are leading indicators that it will be very significant. It's definitely changing the way professional mathematicians, including me, work. But the cases Emily mentioned of AIs being tested on on research level mathematics, there are a few success stories. I think those are exciting leading indicators. There's not any example of anything that I would consider to be revolutionary yet. Who knows what the next year or two will bring. So I'm excited about that.
And I think an important part of the story is, you know, mathematics is not just about solving problems. Even research mathematics is not just about solving problems. I mean, solving problems can be beautiful, but actually stating the problems, figuring out what is an interesting theorem to try and prove is equally important, really what drives the field forward. And we haven't seen any examples there where AIs are coming up with problem statements or new mathematical universes to explore.
Like picking the things that we should be trying to solve?
Yes.
I mean, is AI solving problems that people haven't been able to solve? Has that happened? Does it seem like it's on the horizon?
I one way to answer that is that there is so much math that still needs to be discovered. And there are a lot of mathematicians working together and individually all around the world, trying to make new discoveries and prove new theorems. But there aren't enough of us to solve every problem. And so in particular, there's a famous database of problems tied to the mathematician Paul Erlich that has been collected online. And some of them are quite famous and have received a lot of attention. But others, we don't actually know if anybody's given a serious effort to them.
So I'm aware of, I think, maybe three Erlich problems, like Emily mentioned, that have been solved fully autonomously and maybe another six or seven without prior solutions in the literature that were solved by a human with AI help. And then a number where sort of solutions were kind of extracted from the literature by AI tools. I think that's a really exciting development. I think that said, like, we should view this as like an exciting sign that like soon the tools will be useful to help us solve maybe more interesting questions that people have devoted effort into trying to answer.
You know, it's interesting that you use the word exciting. Just as a contrast, we had mathematician Steve Sturges on last year. And he said something I want your thoughts on.
I think the days when we will understand math may be numbered. That it will not be far in the future when computers are producing really impressive math that we will not understand. And it will be correct, but it'll be like they're oracles just telling us the truth and we can't understand it. We'll just be sitting there with our mouth open.
Emily?
I agree 100% with Steve that if we had an oracle to tell us which theorems are true and which theorems are false, that would be unsatisfactory to mathematicians. We do like it when a proof tells us something new, so that something is true or something is false. But if the proof doesn't also explain something deep to sort of justify the new conclusion, then mathematicians will go and search for a different proof that does. People get celebrated. For reproving old theorems almost as much as for proving new theorems. What I would imagine if an AI discovers a proof of a theorem that humans have difficulty understanding is that humans will then try to prove it themselves and may discover a new proof that way.
But I also want to bring into the conversation the fact that there are sort of two different modes in which an AI can produce a proof. And if it's really a proof that is beyond the level of human understanding and it's communicated just in natural language, so as the output of a chat GPT session, there's an argument that actually mathematicians should just throw it in the garbage and ask instead for an AI to produce a proof that is formalized.
Formalized like in math, in the language of math, is that right?
Right, so a thing that's exciting about the possibility of generating AI for mathematics that I think doesn't exist in quite the same way in other fields is that there are these trusted software programs called computer proof assistants that are engineered by human experts, so not produced with AI in any way, that will take a very precisely written mathematical proof and check the logic line by line. So a concern that many mathematicians share is that we're going to get a lot of sort of AI slop, a lot of new preprints purporting proofs to very famous theorems, but they're long and complicated and require a ton of expertise to evaluate and humans will just never be able to judge these proofs for accuracy.
But just as a large language model can produce Python code, a large language model can be asked to write the proof not in English but in the language of these computer proof assistants and then at least some aspect of the refereeing can be automated, can be outsourced to a computer. And I think that's a more productive realm to interact with AI than just the natural language realm.
Daniel, I want to hear your thoughts on that and also on this idea that AI will become oracles for us in math.
Yeah, so I definitely think it's possible that in the long term, AI tools will become better at proving mathematical statements than human beings. We're not there yet, but it could happen. As Emily suggests, that would be a little bit unsatisfying. Like, my goal as a mathematician is to understand mathematics. That's it. I don't think, it's not clear to me that AI tools producing, you know, amazing and long proofs of mathematical statements would mean that we couldn't understand them. Like, why wouldn't we expect an AI tool able to prove an awesome theorem also to be able to explain it to us really well? So, yeah, I'm kind of, I feel confident that if I'm sufficiently motivated to understand a piece of mathematics produced by a computer, then I'll be able to do it. And that's kind of my goal.
But I guess, I mean, there's a, there's a big problem, a kind of a moral problem with AI-generated mathematics and human-generated mathematics. So there's a very strong norm in the mathematics community that you do not post a proof to the internet, to the archive, which is our mathematics pre-print server, unless you believe that it's true. And we can't ask the same things of large language models because they don't, you know, that, I mean, they're completely untrustworthy, they don't have beliefs. I, you know, I don't really have a theory of mind of large language models, so these, these sort of metaphors are imperfect. So I think we should demand more of a proof written by an AI than a proof written by
a human, because we don't have this sort of norm of trust, this, this norm of belief. So if AI gives us a large proof that is difficult to understand, you know, as Daniel said, the AI has not finished its job, it should also give us an explanation. But maybe that explanation should also be in this formal realm, so we can use computer tools to start the verification process, which will be very long.
We have to take a break, but when we come back, I want to talk about vibe proving. You've heard of vibe coding? Have you heard of vibe proving? You will, stay with us.
Hey, it's Flora. I know that you have heard this before, but it's not a line. Science Friday really can only continue with support from you, our listeners. I love making and telling these stories, and if you like listening to them, please go to sciencefriday.com/donate to make a donation. Join us. Stand up for the value of Science Friday and public media, and help us continue to spark curiosity and spread the joy of science. Donations are fast, easy, and secure. Just go to sciencefriday.com/donate, and thank you.
Science Friday is supported by PBS, presenting Going Wild. If you love learning about the natural world and meeting the scientists who help us understand it, be sure to check out the new season of Going Wild with Dr. Ray Wynn Grant. It's the chart-topping and Webby Award-winning podcast from the folks at Nature on PBS, now with a video companion series. In this season of Going Wild, host-acclaimed wildlife ecologist Dr. Ray Wynn Grant heads out into the field to explore the animals, people, and stories that make nature worth saving. Stops include a hibernating black bear den in Arkansas and an underwater shipwreck site off the Florida coast. If you're looking for a place to start, check out the season's debut episode, where Ray joins beloved biologist and influencer Jordan King in the Florida Everglades
to learn about his unlikely journey into science and how to study a wetland when it's not very, well, wet. So don't miss out. Follow Going Wild with Dr. Ray Wynn Grant on Apple Podcasts, Spotify, YouTube, or wherever you get your podcasts.
And from the Alfred P. Sloan Foundation, working to enhance public understanding of science, technology, and economics in the modern world.
Okay, so we've heard of vibe coding. Is there an equivalent in math? Like, could I ask it? Could I be like, I don't really understand how to do this. Can you solve this problem?
Yeah, you could certainly ask that. And then, well, sometimes it would give you the right answer, and sometimes it would give you a wrong answer. There's still a lot of work to do for a human, which is you have to check the answer is correct. That's hard. So you can vibe prove something and then, well, often it will be vibe wrong. But sometimes it'll be correct and sometimes it'll be really useful. So, yeah, I think this is starting to be a part of the mathematical workflow. It's still important at the end of the day that a human is there taking responsibility for the correctness of the results. And I think we can see some examples of this going wrong.
In September of last year, I counted the number of papers put on the archive, which is the main math preprint server, with the words Hodge conjecture in the title or abstract. The Hodge conjecture is one of the millennium problems. I think there were about 12 total, and as far as I could tell, 11 of them were nonsense generated by AI tools.
Like hallucinations? Is that what we're talking about?
Well, I think it was a human who wanted to prove the Hodge conjecture because there's a million dollar prize associated to it. So they said, Claude, prove the Hodge conjecture. Don't make any mistakes. And then they posted whatever came out of that to the archive. And it wasn't correct.
Right. And what Daniel's alluding to is there's a danger with vibe proving, just like with vibe coding. So vibe coding for an expert programmer can be an enhancement of their workflow, but it would also allow somebody who's a total novice, so I'll put my hand up here in the programming context, to think that they can achieve more than they actually can with computer programming. But it requires a lot of expertise to know when it's bluffing and when it's sound.
One thought on this is that actually the ways the models bluff are remarkably similar to how humans bluff and approve.
Really? Say more about that.
Yeah, so, OK, you can ask, well, first of all, you can ask an undergrad to prove something on their homework, and often they'll leave some things out that they don't know how to do and hope to get full marks. And sometimes that works out. Even if you ask a professional mathematician just off the top of their head, prove something, they might have some kind of general idea and not think through all the details. And sometimes that'll be right and sometimes not. So I think it's actually remarkable to watch the models doing math and see, in some ways, how human they are.
I mean, can the proof assistants sort of root out the AI math slop?
Yes, I think so. So there are some caveats there. Firstly, it's a lot more exacting, so you can't sort of wave your hands and skip a few steps in a proof assistant. It will not accept it. It'll make a note of exactly what has not yet been justified. So I do think it can help weed out some slop. And in particular, there are some AI startup companies that are aiming to get better at writing formal proofs, and they are training by using the feedback from the proof assistant to evaluate partial proofs. The proof assistant is meant to be used interactively and will give you some information about whether each line of your proof is correct or incorrect. And so that information
could be integrated into an AI workflow to improve the search for proofs, for instance.
Do these tools change what skills are required to be a great mathematician?
You know, I think one of the things that's most challenging about being a mathematician today is just the breadth of the field. You know, so Daniel and I are in different enough research areas that I think we would have difficulty understanding each other's most recent papers. And I also know that it's very difficult to keep on top of the literature, even in my own subject area, because there's a lot of exciting work that gets finished every month, and I don't read at the pace that I would like to. So a thing that I'm optimistic about is that these tools will help me stay more on top of the literature.
Yeah, I found them really useful to learn things that I needed. Look, over the summer, I learned from ChatGPT-03 what about hyperkalar geometry, which is something that, you know, I could have gotten a human expert and pastored them, but they would have gotten annoyed with me much more rapidly than the AI tool did. I think that, you know, anytime a new technology is developed, it obviates some skills that by automating them, that humans previously needed. And then it also opens up some new capabilities. And I absolutely expect that to happen with AI tools for mathematics. Right now, I think it's sort of too early to figure out exactly what skills will be become, you know, less
necessary and what new capabilities will open up. But I'm hopeful that it will open up on a lot of new possibilities.
I mean, I think
You know, there's a lot of misconceptions about what a great mathematician looks like. There is some stereotypes that this is something that's identifiable when you're very young. And in fact, maybe when you're, you know, when you're my age, I'm 41, which means I'm no longer eligible for the Fields Medal. But I'm a much better mathematician now than I was 20 years ago because I have spent so much time thinking about mathematics in the last 20 years. And some of that is reading the literature, some of that is attempting to solve a problem and failing, but failing in a way that when a similar problem comes up later, I will have better intuition about what avenues might be productive and what might be not productive. So I, you know, I really think what makes a great mathematician
is time and dedication to mathematics. You know, anybody who falls in love with the subject and is fortunate enough to be able to devote themselves to it is going to be able to achieve some pretty cool things.
Yeah. So, well, yeah, what makes a great mathematician? So certainly, I think loving the subject motivation, like Emily was talking about, are very important. You definitely need some technical ability. And I think that's what people usually think about when they think about strong mathematics, like, can you solve this problem? Like, can you prove this theorem? But I think in practice, the most influential mathematics is not actually about solving problems or proving a theorem. It's about something that's a little bit less tangible, like some kind of philosophy or mysticism even. Like, you're trying to develop a theory or figure out why something is true. Often that involves, like, discovering a new structure. And that that structure might not even be something that is really precise. Like, you spread a philosophy to a community of other mathematicians
and give them a way of understanding new mathematical objects. I think you don't really think of that as, it's not a purely formal thing. You don't necessarily think of that as being mathematics when you're in high school or middle school or college or whatever. But yeah, the most important parts of mathematics are actually closer to philosophy, I think, than than science.
Well, and the final thing to highlight is creativity. And where great ideas often arise is really in conversation between mathematicians. I mean, we travel all the time to speak to each other in person at a chalkboard about some mathematical ideas. And it's amazing in a collaboration that a new direction can just originate spontaneously from a bringing together of multiple minds.
I love that. Emily and Daniel, thank you for taking the time to talk to us today.
Thanks for having us.
Thank you. Yeah, that was fun.
Dr. Emily Reel, mathematician at Johns Hopkins University, and Dr. Daniel Litt, mathematician at the University of Toronto. This episode was produced by Dee Peter Schmidt. If you're surprised to find that you got all the feels from a conversation about math, please leave a review and tap Follow or Subscribe wherever you listen. Thank you for listening. I'm Flora Lichtman.
Page updated October 31, 2026 · transcript and summary are machine-generated