Pod Insights Deep Questions with Cal Newport Are We About to Lose Control of AI? | AI Reality Check About this episode Cal Newport takes a critical look at recent AI News.
Video from today’s episode: youtube.com/calnewportmedia
(0:00) Are we about to lose control of AI?
(3:04) How much should we be afraid of recursive self improvement?
(7:52) Are these fears justified?
(9:36) Faster software development doesn’t equal smarter AI
(12:25) These tools are completely controllable
(16:52) Taking a step back
Links:
Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow
https://www.anthropic.com/institute/recursive-self-improvement
Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter.
Learn more about your ad choices. Visit podcastchoices.com/adchoices
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Cal Newport? Last week Anthropic released a report with a scary sounding title when AI builds itself and it came accompanied by a scary animation that shows machines replicating themselves exponentially like cells in a Petri dish. Now, the body of the report itself keeps these dark vibes going. I want to read you some actual quotes here from the intro to the report. They say, "For most of AI's history, humans drove every step in its development cycle, but at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work. Taken far enough and given enough compute, this trend points to an AI system capable of fully autonomously designing and developing
0:45 Cal Newport? its own successor. This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable, but it could come sooner than most institutions are prepared for." A little bit later, they then add, "AI that can build itself would be a major development in the history of technology, one that could bring enormous good for the world in science, healthcare, and beyond, but full recursive self-improvement also might increase the risks of humans losing control over AI systems." Now, if you look at the headlines generated in response to this report, most of them focused on a section of the report that seemed to call for a worldwide
1:28 Cal Newport? pause on AI development to avoid the scenario of humans losing control. But if you read that section closer, you see that's not actually what the report says. Here's the actual wording. "If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe." So in other words, Anthropic is saying, "We'll only slow down if everyone else around the world does too. Otherwise, we have no choice." But to continue with our efforts at full speed. Now, look, this is pretty grim stuff.
2:10 Cal Newport? Anthropic is basically saying that we are potentially hurtling towards a world of AI that improves itself rapidly until we lose control over it. And they're saying there is nothing that we can do about it, except maybe continuing to publish solemn reports with fancy animations. And I guess also cash in on our stock options after an IPO. All right, so here's the key question. Are these fears justified? Well, it's Thursday, which means it's time for an AI reality check episode of the show, which is a good opportunity to go looking for some measured answers. And that is exactly what we are going to do. As always, I'm Cal Newport, and this is Deep Questions,
2:52 Cal Newport? the show for people seeking depth in a distracted world.
3:03 Cal Newport? All right, so how, how much should we actually be afraid of recursive self-improvement? I want to look at the core charts from this Anthropic report. So we can see what they are pointing to that is giving them these RSI fears. All right, I'll load the first chart from the report up here on the page. It's called code contributed per quarter, per person by quarter. It's measuring how much lines of code their engineers are producing over time. And what we see is in 2021, 2022, 2023, 2024, the beginning of 2025, not much. And then the second half of 2025 and into the first half of 2026, the amount of code jumps up. All right, here's how the report itself describes this trend.
3:48 Cal Newport? It says, "A caveat, lines of code is an imperfect measure as it measures quantity over quality. So eight times lines of code per engineer per day in the second quarter of 2026 is almost certainly an overstatement of the true productivity gain. Nonetheless, it indicates an acceleration." All right, so chart one.
4:06 Cal Newport? Once we introduced these tools in late 2025 for software development, we began producing a lot more code using AI. All right, let's go to the second major chart in this paper. This is called Claude Code Session Success Rate. What we see here is various color lines. Each of these represents a different type of problem. You could ask an AI LLM to solve trivial tasks, routine tasks, substantial tasks. And then at the bottom, open-ended problems. It starts in the fall of 2025. And what we see is, especially with open-ended problems into 2026, and as the new models Mythos and Claude Opus 4.7 are introduced,
4:45 Cal Newport? the success rate with those jumps up from low 20% to somewhere between around 70%. Okay, notice that graph starts, it just starts in the fall of 2025 because, and we'll get back to this in a second in more detail, but in order to ask AI to solve a hard coding problem, and like the example they gave was, why is this thing, the system we programmed is crashing? Why is it crashing? And sending the AI to figure out why. You can't actually have data points for before the fall of 2025 because that's when Anthropic released, along with OpenAI, their first mature, what's called a coding harness. So it's a control program that allows you to leverage an LLM,
5:29 Cal Newport? plus a lot of hardcoded logic, to actually tackle multi-step plans. So you can't really have data for that from before because there's nothing to ask to try to solve a problem like that. You couldn't just prompt the ChatGPT to do that or Claude. So you needed the arrival of mature software development tools built on LLMs before you could even do these tasks. So basically what this graph is showing is that like the very first software development coding harnesses they introduced in the fall of 2025 couldn't solve the hardest type of problems. And then they sort of fixed that the next year. All right, there's one final chart in this paper that they're pointing at to justify their concerns. I'll put it on the screen here. The title is, "Where Researcher Went Wrong,
6:14 Cal Newport? Could Claude Have Done Better?" And we see here a bunch of different models of Claude and percentage bars. And back for these early models of Claude, you know, we were getting like 50 or 45%, and now with the very newest versions of Claude, like Opus 4.7 and Claude Mythos, we're at like 59 to 64%. So we've got like a 10 or 15% improvement on that measure. What is this measure? It's a little bit complicated. Essentially, they have these transcripts of programmers working on programmer-style tasks. And what they're looking for is an example where there's some problem that the programmer is trying to solve, and they take a wrong turn. So they go down some path to explore something that turns out not to
6:57 Cal Newport? be the correct source of the problem. So what they would do is take the transcript of this session, right up to the point where the human was about to try to explore something wrong, they fed this transcript to one of these mature coding harnesses on top of an LLM and said, "Hey, what do you think we should do next?" And if it found the right thing to do, or the human looked at the wrong thing, then they would say, "This is a case where like the LLM plus its coding harness were smarter than the human." So that's jumped from like 50% from a couple of years ago to like 64% now. All right. So that is, this is sort of like the core data they're looking at. They capture this idea of AI is getting
7:39 Cal Newport? so smart at producing programs that maybe soon it will be able to not only improve itself, but, and I'm going to quote here from the report, "become capable of full recursive improvement and begin building their own successors." All right. So looking at that data and what I know about current AI tools, are these fears justified? And I would say, no.
8:04 Cal Newport? Here's what this data is all describing. About a year ago, the major AI companies got serious about building tools to help software developers. These tools are a combination of human-written programs called coding harnesses and LLMs. The coding harnesses can make calls to the LLMs and then act on what the LLMs do. The coding harnesses can also interact with other various tools on the computer. So what all these charts seem to be showing is that like, oh,
8:34 Cal Newport? once they got serious about writing these coding harnesses and also tuning the LLMs to play nicer with these coding harnesses on these sort of computer programming-related task and evaluations, things jumped up. The world before having these harnesses, we couldn't do well on programming tasks. And now that we do, we do better. Now, this is, none of this is trivial. In fact, it's a very smart market for the AI companies to go after. Their software development is a big industry. These coding harnesses built on top of LLMs are like really, potentially very useful, right? Because LLMs are very good at understanding and producing code, the type of tools that a coding harness has to access in order to, you know, execute things are simple and tech-based.
9:14 Cal Newport? Like, this is like a perfect case market to build the first killer apps on top of LLMs. And starting last year and then really picking up speed last fall, the major players really got involved in these harnesses. And so I think that's what these charts are all showing is like, yeah, these harnesses make us, they're getting really good at the, we can suddenly do a lot of programming tasks we couldn't before. But does that mean that recursive self-improvement is imminent? It does not, and there's two reasons why. And I want to look at these one by one.
9:43 Cal Newport? Point number one, faster software development doesn't equal smarter AI. All right, so these tools help computer programmers produce code faster or find mistakes or issues in existing code or systems faster as well. These capabilities, though useful and perhaps a good source of revenue for these companies, doesn't add up to AI being able to improve themselves, to create AI systems that are much smarter than what humans would have otherwise been able to produce. Now, why is this? Because the bottleneck to producing breakthroughs in AI, to building new AI systems that are substantially more capable than those that came before, is not the speed with which you produce computer code or track down bugs or issues in existing code.
10:28 Cal Newport? The thing that advances the capability of AI beyond just training it longer are ideas. Like, if we look at our current world of generative AI built on large language models, there's three big insights that built on each other. The first was Jeff Hinton and his collaborators working on backpropagation, right? This was an intuition that if you applied calculus properly, it would be, you could train neural nets that had many, many layers. This was the so-called deep learning revolution, really picked up speed in the 1990s into 2000s, and it was critical for what we have today. The second big idea that led to our current AI world was Google Research notion of adding something known as an attention transformer. It's a mathematical formalism you add into the feed-forward architecture of a large
11:13 Cal Newport? language model that allows there to be selective attention on the input tokens as you are trying to calculate what token to produce next. This allowed sort of coherent generation of text based off a very complicated input or, you know, request, right? This was revolutionary. This is largely a sort of intuition around architecture and linear algebra. The third breakthrough that led to our current moment was researchers at OpenAI, led by Jared Kaplan, wanting to know what happened if you scaled the size and training compute of large language models well beyond the limits where traditional machine learning theory said you're going to start overfitting.
11:57 Cal Newport? Those three things together led us to this current future, and none of those had anything to do with computer programming.
12:04 Cal Newport? It wasn't, "Oh, if only we could have programmed faster or been better at finding bugs in computer programs, we would have had those advances quicker." These advances were scientific, not engineering. AI does not advance, it's not created at the fingertips of computer programmers, and speeding up those computer programmers does not speed up the rate at which we get smarter, more capable, or more advanced AI systems. All right, the second point I want to make here is that these tools, these software development tools that they're testing in these charts are completely controllable.
12:38 Cal Newport? Arguments that say we're on track to losing control of AI like to think about these sort of AI-based coding tools as some sort of unknowable, alien black box that we ask to do things, and we don't know what it does. It just kind of like goes off and behave, and we don't know its intentions, and in fact, it could develop somehow its own intentions that are contrary to us and go rogue, right? We just see this like HAL 9000 from Stanley Kubrick's 2001. This is not how these systems work. And again, I mentioned this before briefly, but let's just say it again briefly right now. The software development tools that they are testing in these charts is a combination of a large language model like Claude Mythos or Opus 4.7,
13:22 Cal Newport? and what I've been calling a coding harness. Now, the coding harness is a computer program. It's written by humans. There's nothing obfuscated about it. It's not a neural net. It's not something that trained. There's no mystery about how it behaves. It's a bunch of pattern recognition, often using old-fashioned tools like regular expressions, and a huge amount of conditional logic, if-then statements, if this do that, a lot of hardcoded cases of things that comes up often in software development, and they want to make sure they do the right thing. This coding harness, programmed by people and completely linear, deterministic, we know exactly how it works. Will then make calls to an LLM through an API when it needs LLM intelligence, right? So I might say, okay, what we need to do here is produce some code.
14:06 Cal Newport? All right, I'm now going to create a prompt and send it to the LLM. Can you give me code that does this? And the LLM will return some code. Or, okay, we're trying to, we want to create a multi-step plan for investigating why this program is crashing. I will put that into a prompt. I will send it to the LLM, say, will you write me a multi-step plan for this goal? Here's the constraints, here's the tools you can use, write in this format. The LLM produces text, it gets that back. And now the coding harness, human-written code, completely non-obfuscated, completely deterministic, looks at that plan and says, great, let's break this up into steps and execute them one by one. It's the coding harness that actually has any action, that actually accesses tools. It's the coding harness that has all the control logic. So we programmed that, which means it's imminently controllable, right?
14:51 Cal Newport? So like, let's do a thought experiment here. Like, let's say there's a certain tool you don't want your programming tools to use on your computer. Well, you could do that with 100% certainty by just never having your coding harness call that tool. Just like, no matter what instructions it gets, it's just not on the list of tools that it'll call. This is a thing we often mix up. LLMs are unpredictable, okay? They're token producers, right? And they're deterministic in the sense that if you give it the same input, it'll give you the same output. But they're unpredictable because the output that you actually get from an LLM is effectively a probability distribution over potential next tokens.
15:35 Cal Newport? And then you probabilistically select a token from that distribution. This is why if you ask the same prompt multiple times, you might get different answers because those random selections made outside of the LLM might select differently.
15:49 Cal Newport? So LLMs are unpredictable, right? That's why if you say, hey, give me a plan for this. Like, sometimes the plan will make sense and sometimes it doesn't. And sometimes it'll seem reasonable, but it'll have a weird aside. Or if you give it a big scenario and say, what do you want to do next? It might, and you know, it might say something you weren't expecting. And if it gets little cues and it prompts that we're playing like a sci fi game, it'll be very sci fi and maybe it'll do like weird things. LLM textual output is unpredictable. That doesn't mean that AI systems that query LLMs are unpredictable because all of the control logic and action is done in human written programs. Right? So this idea that it's like, these are just these black boxes that maybe at some point will spontaneously start improving themselves makes no sense. The LLM is a static thing that you can prompt and it will give you text back. That's it.
16:34 Cal Newport? And a human controlled program does that prompting. That is not a setup where there is a, where is the growing intention here? Where is the like, we're going to change our own code? That type of thinking only makes sense when you kind of obfuscate this all as some sort of alien brain we don't understand. No, it's an LLM that's unpredictable plus a very predictable coding harness. All right, so let's step back here. What's a better description of the reality where we are right now? As these programming tools built on LLMs are getting good, right? Like, I mean, we've only been working on it for less than a year, but we're finding like, oh, this is a great application of LLMs. We're still trying to figure out what these tools can do and can't do and how best to integrate them.
17:18 Cal Newport? What's a better description of this reality? Well, I want to bring one more chart up here on the screen. It comes from a recent John Byrne Murdoch article in the Financial Times. I actually saw this chart first through Gary Marcus's newsletter. Here's the chart.
17:32 Cal Newport? The title here is relative change in monthly volumes of iOS app releases and reviews. And here's what we see is over time, starting in 2025, when these AI based coding, mature coding tools were first released, the number of iOS apps released jumped up. You got dark blue line there. At the same time, let's look at this light blue line. Apps with significant usage, it stayed steady and is even falling.
18:01 Cal Newport? All right. This, I think, captures something important. We're seeing that the introduction of these software development tools led to an increase in the number of mobile apps, which are like exactly the type of programs that like basically anyone with like some tokens to burn and Claude code can just spit out. They're easy for AI to produce. While at the same time, we're seeing a decrease in apps with significant usage. AI-powered software development tools speed things up, and that makes everyone involved feel more productive. But it doesn't necessarily mean that we're accelerating the creation of useful new things. AI often offers interesting potential to a lot of fields, but figuring out how to make it economically useful is a complicated effort that requires a lot of focus.
18:46 Cal Newport? Putting out these sci-fi themed essays about all the doom that's lurking right around the corner might make these AI companies feel important or exceptional, but I'd argue that at the moment, they're proving, if you'll excuse the use of an ironic term here, unproductive. Programming tools are interesting. They're here to stay, though in their final form and how we use them is still up in the air. This is a new thing. But nothing about this new tool's arrival should make you think that somehow AI is going to start improving itself. I would almost say shame on Anthropic for dropping this report. No suggestions, no reassurance, no culpability, no responsibility.
19:30 Cal Newport? Just like, what can we do? This is not productive. This is not useful. If you're wondering, let's just sum this all up. No Claude code or codex or cursor is not a thing that's going to very soon start improving itself till we lose control of AI. We've got bigger fish to fry here. All right, that's enough for this week. We'll be back on Monday with an advice episode of this show, which you check out. I'll probably have another reality check next Thursday. Until then, however, remember, care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world.
20:09 Cal Newport? Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas, and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today at calneupro.com/ideas.
20:37 Cal Newport? Cal.
Previous Should I Press Pause? | Monday Advice Next Do I Need a “Brain Gym”? | Monday Advice Page updated October 30, 2026 · transcript and summary are machine-generated