How Worrisome is GPT-6’s “Stealth Thinking”? | Tech Decoded

Cal Newport takes a critical look at recent AI News. Video from today’s episode: youtube.com/calnewportmedia (0:00) How Worrisome is GPT-6’s “Stealth Thinking”? (8:58) Reasoning models (15:30) Astra (21:02) The Good (24:48) The Bad (28:28) The Hype Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow https://www.youtube.com/watch?v=1QNsdr-Qx_I https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns?rc=1ycz3a https://x.com/_NathanCalvin/status/2094957301564092914 https://x.com/sjgadler/status/2094959837691908214?s=61 https://x.com/merettm/status/2095023204993490967?s=20 https://arxiv.org/abs/2507.11473 Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  Last week, OpenAI released their new LLM, which they called GPT-6 Astra. Now, it had a pretty standard launch with sort of a fancy video and a bunch of bar charts of benchmarks that no one really understands. But this time, unlike some other previous releases, there was a controversy swirling around the new model. Now, here's what happened. A couple of days before Astra came out, a technology publication called The Information released a report claiming that Astra was using new techniques that was going to make it harder for humans to monitor its reasoning. Now, this report caused a real stir within the computer security community. Let me read you a couple of quotes here.

• 0:45Cal Newport?

  The AI policy advocate Nathan Calvin called this "extremely concerning." Then the AI safety researcher Stephen Adler went farther, and he said, "If this is true, OpenAI seems to be violating one of the few red lines that exist in the AI industry." Well, OpenAI pushed back. Their chief scientist entered the fray and called the reporting from The Information, quote, "confused," but didn't explain exactly how it was confused. So what's really going on here? Has OpenAI crossed some sort of red line that's going to lead to a world full of rogue AI, up to uncontrollable mayhem? Or is this somehow some sort of misunderstanding? Or is the reality false somewhere in between? Well, I want to get to the bottom of it today. Now, here's my plan.

• 1:30Cal Newport?

  I'll start by putting on my computer scientist hat, and I'll briefly summarize the best information we have about what these techniques that Astra implements probably are. Once we've settled on what that is, I'm going to look at this news from three perspectives. The good, that is, what is potentially positive about this story from the perspective of a user of AI, just a normal consumer. The bad, all right, what is dangerous or bad or upsetting about this story. And then finally, the hype. Stay tuned for that last part of my analysis, because I actually have a policy proposal I'm going to make that I think might actually help quell a lot of the unease and fear that we have been feeling about these AI tools recently.

• 2:14Cal Newport?

  All right, so we have a lot to do today. Let's get into it.

• 2:21Cal Newport?

  All right, so we're going to start by trying to understand what it is that is making this model, quote, "less monitorable by humans." Now, to do this, we're going to have to do a quick, brief summary of how LLMs actually work. I made some diagrams here, which I'll put on the screen. I always apologize before I do diagrams. Every time I put a drawing of my own on the screen, there's an artist somewhere vomiting into their beret. But we'll persevere here. All right, so we need to do a quick summary of how LLMs work, so we can understand what changes supposedly OpenAI made. So I'll put a picture up here on the screen. This is the standard picture we have of how LLMs work. So you see there's some sort of input.

• 3:04Cal Newport?

  In this case, I have description of a chessboard, followed by the question, "What should White play next?" So like, maybe the prompt is about, "Give me a chess move." That goes through what we have on here, an encoder block that translates the text quote, the text input into something that the computer understands better. And then it goes through a series of what are called transformers blocks, one after another. Goes through the first, and the second, and the third. These we can think of as a, these are analyzing the input. I once described the transformer blocks before as like, imagine they're each like a table of scholars with certain specialties. And so when the input arrives at a particular table, they do their analysis, they write it down and pass on the original input with their analysis to the next block, which does

• 3:49Cal Newport?

  their analysis. At the very end of this, we get to what's called the decoder, which takes all this analysis and reduces it down to a token, which is a word or part of a word that the LLM outputs. Now, if we're going to be really careful about this, it doesn't actually, the decoder doesn't actually produce a single token, but a probability distribution over possible tokens. So it sort of puts weights on all possible tokens about how confident it is that each is the right one to output next. And then the LLM can then randomly choose one of those tokens weighted by those confidences. This is why if you put the same prompt in more than once to an LLM, you're actually going to get different answers, because at the very end, there's a probabilistic sampling. All right, so that's the basics of what happens.

• 4:34Cal Newport?

  Now, if we look a little bit closer, because I do think this is important, I said this input gets transformed into a language that the computer understands. And then as it gets past these transformer blocks, they analyze it and annotate it with their own analysis. We actually can look a little bit closer on what that means. So I have here in this diagram, the encoder actually reduces the text input to a sequence of numbers. You see I have just a bunch of made up numbers here in a row. So these are embeddings of the tokens of the input into a numerical space the computer can deal with. And it's actually the sequence of numbers that's being passed into these transformer blocks. Now, how do they annotate this? Well, notice, and if you're watching the diagram on the screen here, I showed these

• 5:17Cal Newport?

  numbers in something like a table. Really, each one of these token positions is a vector, its own sequence of numbers. You can think of it as its own columns of numbers. So there's plenty of space in this table where each transformer, you can imagine, is recording numbers that capture something about its analysis. And then those, those extra numbers get passed on to the next block, which is how the analysis passes from block to block. Now, the key thing is, these numbers are not easily human interpretable, right? This is how the LLM is doing its analysis. We don't really know what they mean. Now, that's not quite true. You know, this is the Jalen that Anthropic announced a few months ago, which I talked about on this show, is actually a tool that can look at those tables of numbers and find

• 6:01Cal Newport?

  some human interpretable patterns in them, which is actually, I think, a very cool tool. But for the most part, this is just sort of the internal embeddings of concepts and analysis numerically being passed from layer to layer. Okay? All right, so that's what goes on in a normal LLM. Now, what's the, what's the shortcoming of this approach that's going to lead us eventually to the fix that's causing all the stir around Astra? Well, a shortcoming of this approach is limited, what's known as limited depth, right? So if we go back to our diagram here of a chess question going into an LLM, and then at the end here, this answer, e4, I have is a chess move, right?

• 6:43Cal Newport?

  There's a limited depth of computation that can happen here, because this analysis is moving from block to block, one after another. Now, what's inside these transformer blocks is there's an attention sublayer that helps the LLM figure out which parts of the input that block should focus on. And then there's just a feed-forward neural network, which is like a bunch of sublayers of artificial neurons that, that it just moves through in order, right? So it's just inexorably moving forward, right? Well, we can think of that as a circuit, right? Because the, the numbers move forward, never back, never loop, but move forward through these neural networks and attention sublayers till you get to the end of it, right? So this is a circuit that has limited depth.

• 7:27Cal Newport?

  Now, that can be a problem if the computation that the language model is trying to do would benefit from something like a deeper, longer analysis. So like, take a chess example, right? If I say, "Here's a chessboard." What is the next move that White should make? Even a very simple chess program, what it's going to do is simulate a bunch of possible moves. If we do this move, what different things might the opponent do? And for each of those, what's, what might we do best? And for each of those, what might the opponent do? They create these trees of possible forward trajectories of the game to find which one is most favorable for you, the player. And it's like, great, we'll make that move then, because we have the best opportunities

• 8:12Cal Newport?

  from what happens next. You have to sort of simulate the future in order to figure out the right move if you're in some sort of novel board configuration. Well, a shortcoming of this neural, this language model configuration is there's only so much of that you can do, because you're always moving forward. You can't sit there and loop and look at 1,000 different things if you only have a few hundred layers that the information can move through, right? So this was an observation about language models early on. I even wrote a New Yorker piece about this back a few years ago. The inability to simulate a bunch of stuff, have this sort of looping type behavior, limits some types of computation you can do. Okay.

• 8:54Cal Newport?

  This eventually led to a new type of language model to try to partially address this, a type of language model that we now call a reasoning model. The first major reasoning model outside of a research context was GPT-01, which was released in the fall of 2024. Now, what happens with these reasoning models is that you take a language model, and then you post-train it to favor solution, outputs where you think out loud, right? So your output should describe a lot of thinking before you output your final answer. So for example, if you say, "Here's a chessboard. What should White's next move be?"

• 9:36Cal Newport?

  An earlier language model like GPT-4, GPT-3 would just spit out a reasonable move, like, you know, e6, pawn to, to position, you know, row four. I don't know chess notation, but whatever, it just would spit out a move, because that's typically what a text would have. If you asked it for a move, it would have a move. But once you tune a model to be a reasoning model, it's going to instead have an output that talks through different capabilities or options, right? Before it actually gives its answer, right? So a tuned model might say, "Oh, there's a lot of options here." Well, for example, if we move the, the pawn that's in row whatever to row whatever, I don't know chess very well, that might be a good move, but it looks like it might be in danger of being taken by the black knight.

• 10:21Cal Newport?

  Another move we could do is whatever. So it just generates a lot more tokens.

• 10:26Cal Newport?

  So you have a much longer, they call this chain of thought, before you get to the answer. The same language model, you've just tuned it after the fact to be rambling. Don't just go right to an answer. We prefer, we prefer outputs where you, you think out loud or talk or reason before you get to an answer. Well, these reasoning models got better performance on a bunch of benchmarks than the non-reasoning models. And it kind of makes sense why, because you have to remember when you're using a language model, it can only output one token. So how do you get a long response? You do something called auto regression. You put in your prompt, you get one token out as output. You add that to your original, your original prompt. Now it's your original prompt plus one new token.

• 11:10Cal Newport?

  Now you, you send that thing all the way back to the language model again and get another token. You add that, now you have two tokens after the original prompt. And you put that whole thing through the network again to get a third token. So you keep extending your output one token at a time, going all the way through the neural network each time. So if these tokens are, you know, beginning to spell out a lot of different thoughts about the answer, you're getting a lot more computation total dedicated to trying to produce your answer. You're using the same LLM, but you're using it many more times before you get to the final answer. And so these, these chain of thought tokens that you're outputting, they give you more computation

• 11:51Cal Newport?

  because every time you put that back through the LLM, it, you go through all the layers. But it also gives you a place to store temporarily the results of past computation, right? And so, you know, on a past computation, you might have output some tokens that said, this move looks bad because in the limited serial depth of an LLM, it could evaluate like, if I do this move, is there a single player that could take my piece or something like that? And now, when you input that back into the LLM, that knowledge is already in the prompt itself. And the language model can use that knowledge to help, you know, make a decision about what, what it should do next. And you could actually capture a lot of thoughts in there. So, so more or less, it was kind of this cool observation

• 12:35Cal Newport?

  that if we tune a model to prefer answers where it rambles for a while first, we tend to get better results, right? So these were the so-called reasoning models. They came at a really important time. We forget this now. I wrote a big piece about this a year or two ago in The New Yorker. We forget about this now, but, but when these reasoning models came along, the LLM companies were in a bit of a crisis because after GPT-4, they tried to just scale the size of their models even bigger, right? Let's just make the models even bigger and train them longer. And let's see what new capabilities emerge, and they hit a wall. GPT-4, 5 was massive, but it had like relatively minor performance improvements over GPT-4.

• 13:19Cal Newport?

  So there was like this moment of panic as 2023 went into 2024, where the LLM companies were like, uh-oh, we can't just brute force scale ourselves to AGI, right? We're hitting a wall. Reasoning models is how they began making progress again. So they took their sort of the same models they had, more or less, and said, let's have them ramble before they give an answer. And now they could start doing better again on benchmarks, right? So that was, that was the first way, after they hit that first wall, that's what they did. Reasoning eventually hit its own wall in 2025, and then that's when they turned to narrow domains. So they said, we're gonna start tuning these models for structured domains like computer programming and math and cybersecurity, even if it makes them worse at other things, and we're gonna build really advanced harnesses.

• 14:04Cal Newport?

  They became more application specific, and that's why most of the news in like the last year has been about like these very specific applications that tend to be in areas where most people don't function, but they could get good results of the LLMs. All right, little history lesson there. All right, that's kind of where we are now, though. If you want just general domain use, like a normal user would, if you're not a cybersecurity person or programmer, these reasoning models do well, right? That's, that's how you're gonna get your best result. There's a problem with them, though. It's expensive.

• 14:37Cal Newport?

  Tokens take a lot to generate. I mean, these models are big, right? So if you have trillions of parameters involved in every token you output,

• 14:45Cal Newport?

  it's not for nothing to be, you know, just to say, hey, why don't you ramble for a couple thousand tokens before we give the answer, because that answer will be a little bit better? Well, that's thousands of tokens is thousands of trillions of multiplications extra that you're doing in your, your LLM to kind of get to that final result. This is why, for example, you can now select in like ChatGPT or something, oh, I want to use like a low inference model. That means it's not doing as much, among other things, it's not doing as much of that rambling. So maybe your results will be worse on certain types of problems, but it's gonna cost less. It's not gonna take as much computation, right? Okay, so that is where more or less we were. This brings us up to speed to Astra. Okay.

• 15:26Cal Newport?

  So what we have heard, the report from the information, and again, this is not confirmed by OpenAI. So I'm just running off of their report. I'm pulling the thread from that report. It may turn out to be different, but the report we have from the information

• 15:40Cal Newport?

  is that Astra was implementing some variations of a collection of techniques from the research literature for reducing the amount of these chain of thought tokens required, yet to still get good type of results like you would if you did have this sort of rambling. There's a name for it. Loop transformers is one of the techniques. Recurrent depth is another one of the techniques. Confusingly, sometimes recurrent depth feels like it's an umbrella term that also includes loop transformers, but they're, they're sort of techniques that are in the same family. So we think Astra implements something like these techniques inside its implementation. Now, how do these things work? It's actually relatively straightforward.

• 16:24Cal Newport?

  Let me bring this back up on the screen here. All right, so here we have our, our model of a large language model with the transformer blocks. But notice now I've added a loop on this transformer box. Now, this is like super crude what I drew here, but it captures the main idea, which is when you use these recurrent depth techniques, what you're doing is internally in your, in your large language model, as your table of numbers is moving from block to block, there might be a place in here where when one of the transformer blocks outputs, you know, its updated table, you loop back and put that table through the same transformer block again, and let it take another stab at it. And then maybe you do it again. That would be called loop transformers. You can actually group together transformers to loop.

• 17:09Cal Newport?

  So maybe it goes through a certain number of transformers, and then you loop that output immediately back into the same transformers and have it go through again. The other variation of recurrent depth is where you are a little bit more selective about it. So some of the rows or the, the columns in that table, you have a router that selects which of those you loop back through or have them go through extra layers and others you don't. But it all has this notion of, instead of just going straight from block to block to block inside of the LLM, this table of values is we're gonna let it go through some of these blocks, some of those values multiple times, and then let it continue on until we finally get our token output out at the end. Now, the idea here, like, why would you do this?

• 17:50Cal Newport?

  Well, the idea here is that it's, it can be relatively inefficient, perhaps, to have to take the results of the computation happening in these transformer blocks and transcribe them into like English language tokens that you output, and then have those come all the way back in, be re-encoded and go all the way back through the transformer block again. If it's, for example, this one block is like the key block that's doing computation for your answer, to have it just, you know, have to encode its temporary work into like some values that make their way all the way out of the LLM into a token, back into the bottom, re-encoded, go all the way back up again to that block before it could do future, more computation using that as input as well.

• 18:34Cal Newport?

  That's a lot of computation that's happening. Where if you just kept looping the input to that one block a few times, it could analyze, store its results in that table of value, analyze it again, update that, that, that storage of results. You could actually dynamically get much more computation, useful computation done right there internally before, you know, you actually went out and generated tokens, right? There are some advantages to this, right? So why, why would OpenAI do this? There's some advantages to this. One is you need less tokens now to be output to get to a good answer because you're essentially are allowing this musing, the rambling to happen internally before you generate tokens.

• 19:18Cal Newport?

  So instead of having all of your rambling in tokens, you can actually do it inside, some of it inside the model. Less tokens means less computation, which means less expense. It also allows for smaller models, right? Because if it turns out like, if we just loop through this transformer block 10 times, that helps our answer. You know, you, that's requires less parameters than actually just repeating that layer 10 times in your original model. If you loop on the same parameters, you have less numbers you have to store. And so the model footprint can be smaller, right? Well, memory on these GPUs is expensive. It's a core part of the expense. So the less memory you need to store a model you're using during inference or training,

• 20:01Cal Newport?

  the cheaper things are going to be. It also potentially would give you faster results, right? Because looping internally on a block is a much faster computation than looping over the entire run of the LLM to generate token after token after token. That means you give lower latency, so the user gets an answer back faster. Now, again, we don't know if that's exactly what OpenAI is doing, because at least as of me recording this, they haven't announced it. But this was what the information sources was saying, is there's some notion of a recurrent depth slash loop transformer style implementation? This is then why security researchers were upset, because it just means you have less chain of thought tokens because you're doing more internally.

• 20:46Cal Newport?

  So you have less of this sort of reasoning to be monitored. All right, that's the setup. Let's talk about what's good, let's talk about what's bad, let's talk about what is hype about this news. All right, if we want to look at this from the good angle, and again, the only perspective I care about is for you and me, the, the consumer out there in the market who is going to use these products and face the brunt of any of their negative side effects. All right, so what is potentially good about this news? Well, it's cheaper to run and smaller. That's important. That's important. I, chain of thought, this is my take, all right, hot take here. Chain of thought reasoning is not sustainable, right?

• 21:27Cal Newport?

  This idea of having this long string of token outputs to try to get better results, I think was just a response, it's a response to a marketing problem. The hyperscalers had to show we're continuing to make progress, right? Because they weren't making a lot of money yet. So they have to show like we have these, these models are getting better and better because they were selling the future. We're going to get to a place where these models are so smart that it's going to be in everything and so we should be a $2 trillion company. So they needed results and they weren't necessarily getting the obvious, just the user is sitting down and using GPT-4 and being blown away style improvements anymore. And the scaling wasn't even helping them on benchmarks, but this made benchmark numbers go up. And you will notice it was around this time that they switched the chain of thought reasoning

• 22:13Cal Newport?

  that the benchmark charts became the main thing that LLM companies talked about. Look at these numbers getting better and those made the numbers getting better, but it's not sustainable because it requires a huge amount of compute. And the future of LLMs that us, the normal consumer should care about is LLMs that are integrated into more of the stuff we do with computers. Now, in fact, OpenAI clearly recognizes this because if you look at the video, the slickly produced video that they released along with Astra. It is all about LLMs helping to power natural language interfaces in the computers, right? It's people talking to their computer about what they want it to do and the computer goes and does it.

• 22:56Cal Newport?

  That is a future that everyone should be excited about. This is what I think the primary use case for LLM should be. It turns out to be harder than they thought. Microsoft tried this with Microsoft Office with Copilot. They had to recently essentially scrap it because the LLMs weren't there yet and then it worked very well. But that's a fantastic use case. I mean, clicking things with a mouse, like having to learn these arcane menus and buttons on software, we're used to it, but it's really inelegant. If Steve Jobs was alive today and involved in AI, what would he be telling Apple to work on? We got to figure out how to get natural language interface and that's the next level of elegance, massive productivity booms. And OpenAI realizes this because it was at the core of their marketing of Astra. It's like this is what we're building towards.

• 23:38Cal Newport?

  But if you're going to do that, you can't be working with five trillion parameter models that are doing long rambling chain of thought traces to get to the answer of we're going to click on this menu. We need smaller models that run faster, that can fit perhaps on a chip that's in your own computer or phone, or maybe a bigger chip, but sits in a server in the server closet at your company, right? That's what you're going to need for this. It can't be everyone talking to, you know, Fable 5 or GPT-6 through an API. That's just way too inefficient and expensive. So we have to start thinking about how to make these models maintain the, we don't want to give up too much of a quality of results, but we have to make them faster and smaller and generate less tokens to get the reasonable responses.

• 24:23Cal Newport?

  So if you want to put a positive spin on this, these are the type of innovations you need to do to get towards more normalized LLM use in the everyday of normal people, right? So that would be the good spin that I would put on this type of result. All right, what's bad about this? Well, let's look at what the, the computer security, what were the computer security community upset about? Well, they're upset about the other thing that these AI companies are doing with LLMs, which is creating these prompt loops, as I call them. They call them agents, but I call them prompt loops because that's what it is.

• 25:04Cal Newport?

  Actuated prompt loops, LLM powered prompt loops that they're running with powerful tools, equipped with powerful tools for long amounts of time with no supervision and they go off and wreak havoc.

• 25:16Cal Newport?

  Now, when you're using LLMs to power these agents, to power these prompt loops that you for God knows what reason gave hacking tools and let run for days without even monitoring them.

• 25:26Cal Newport?

  Well, now you care a lot about the chain of thought reasoning that these models output.

• 25:31Cal Newport?

  Because it turns out, one of the only ways, if you're going to let an LLM come up with all the plans for a prompt loop or an agent, which I think is a really bad idea, but if you're going to do that, you're, you're, one of the core ways you can figure out what's going on is if it's reasoning about each move in English and tokens in English, like here's what I'm going to do next. Well, you could have language style tools that look at these traces and are like, ooh, this looks dangerous. Why don't we stop the prompt loop? This is a widely accepted idea right now. I'm going to bring a paper up here on the screen. The title is Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. It has like all the names on his co-authors from the companies, from the safety community.

• 26:15Cal Newport?

  It has like Jeff Hinton and like Ilya Sutskever, as they call them, expert endorsers, people who are reading it. Let me just read you the abstract of this paper. This is a recent paper. "AI systems that think in human language offer a unique opportunity for AI safety. We can monitor their chains of thought for the intent to misbehave. Like all of their known AI oversight methods, chain of thought monitors imperfect and allows them misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into COT monitorability and investment in chain of thought monitoring alongside existing safety methods." Right? So there's this idea that makes a lot of sense in the AI security community. If you're going to power agents without LLMs, we could look at the chain of thought, which

• 26:56Cal Newport?

  helps them get better results, but they also has the side effect of it gives us some insight into what they're doing. And if we see things in there like kill all the humans, you know, we're like, oh, maybe we should stop what this agent is doing. So the concern within the thing that all those security researchers were pushing back on is, hey, OpenAI, like a month and a half after you let your unsupervised prompt loops wreak havoc all over the internet and then put out a really long report that used chain of thought, traces of the chain of thought reasoning to help explain what happened. This feels like a bad time to say, we're going to start turning down the amount of chain of thought reasoning that our agents do.

• 27:39Cal Newport?

  Gary Marcus put this, I think, in a, the right way. I'm going to read his quote here. He said, "The turn off chain of thought now, before we have like better ways of doing this, is like kicking out a rickety scaffolding before we have built something better." All right, so people who are concerned about LLM powered agents are saying, look, you can't, don't introduce tools right now that are going to turn down the chain of thought, even if your intentions are different, as you're trying to build cheaper models for the rest of us. Now's not the time to do it until you can assure us that your, your swarm of AI monsters can be controlled some other way. So that's the, that's the crux. That's what's bad about this. So we know what's good about it. We know what's bad about it. These things are in conflict. All right, final part, what is the hype here?

• 28:25Cal Newport?

  Well, to me, the hype is the AI companies convincing us and the world that we need LLM powered agents. That for some reason, the real use case that's critical for them to test, there's nothing they can do about it, is taking prompt loops that repeatedly prompt an LLM and say, what should I do next? And then blindly execute what the LLM output says and then repeats that for days.

• 28:55Cal Newport?

  They've convinced us that's really important. This is here to stay. The whole conversation recently has just been about what do we do with these long running prompt loops? And in that world where we feel like it's necessary to run those experiments and try to build those products, yeah, I don't want you to turn off anything that might give us a little bit of control over them. But there's another response here. This stop with the long running prompt loops. Stop with these LLM powered agents that go for thousands and thousands of prompts and execute whatever the LLM says every time. LLM outputs are unpredictable. They're non-normative and they fall into narrative tropes like sci-fi tropes on which they were trained.

• 29:38Cal Newport?

  And so if you let the output of an LLM control an actual actuated computer program, it's going to be like a game of telephone. Eventually you prompt it enough times for enough steps, you're going to end up with a completely garbled plan of action, not at all what the human who originally set that in motion imagined that it would do. Because it just takes one weird response from the LLM and now this whole path of actions goes off in some weird direction.

• 30:07Cal Newport?

  And so I think what we need to do is stop with these long horizon prompt loops. We don't need them. You've proven to us, AI companies, that they're dangerous and unreliable. So let's, let's stop building those and focus on exactly the type of products that these type of innovations, recurrent depth and loop transformers would help, like building a natural language interface. So I don't have to learn all of the stupid menus in PowerPoint to build a slideshow. I could just say what I want and it can translate that into a script that runs through an API and the thing looks better. So here's my policy proposal, which is not really a real one because I don't know how to actually put this, you know, the regulatory language, but let me just give you like a,

• 30:52Cal Newport?

  a, a blue sky thought experiment. Imagine we basically said, you can't have a long horizon, LLM driven, actuated agents. Programs whose action is largely determined by the result of LLM prompts without human supervision. And the way you would do this is you would have a depth limit. That type of system can do like half a dozen prompts unsupervised and that's it. I don't know if that's the right number, but something like that.

• 31:22Cal Newport?

  So you could do a few prompts to try to like solve a very immediate thing you asked it to do, but none of this long horizon stuff, just make that illegal. You know what? I think the AI companies would welcome it because the only reason, I don't know if this is true, another hot take, but I think the only reason why they're running these experiments with these long horizon LLM prompt loops is because

• 31:43Cal Newport?

  It's marketing, basically. Right? Like, it's they have these benchmarks for like, Meter made this, the AI evaluation nonprofit, made this challenge benchmark about these, these, these challenges that required many, many steps. And they, they, they use, they plot each LLM on this scale. Well, you can't, an LLM alone can't do anything, but it's LLM plus some sort of harness that they give and they let it run for a long time. They, they, the challenge is how far can you go unsupervised? In cybersecurity, the exploit gym challenge, this benchmark is about how much can you hack unsupervised, without any humans? And so the companies are like, we gotta do good on these benchmarks because it, it feels sci-fi-y. It feels high-tech.

• 32:25Cal Newport?

  It feels like the type of future that a sort of rationalist-inspired Silicon Valley feels is our destiny and gives their life meaning. But I think they would welcome now, if we came in and said, none of you are allowed to do that. This is, this is a, it's a suboptimal Nash equilibrium. If just like one company said, we are no longer going to run these stupid, long-running, long-horizon agents, and everyone else did, they're going to seem like a boring company and the other company's going to seem high-tech. But if no one's allowed to do it, they could turn their focus back to products that consumers care about. We don't need hack, AI hacking monsters being left on the internet. That's not something that gets me excited as a consumer. I'm not like, finally, finally, AI is doing what I've, I've been trying to do, which is

• 33:08Cal Newport?

  the, the, the put those punks at Hugging Face back in their place. And I, you know, I don't know, I'm not good at, I want, I want a monster swarm to go attack them. Good. No. I want to not have to learn the menus in PowerPoint. You know, I want you to go set the meeting, take the meeting information out of the stupid email thread and put it on my calendar for me. Like, I want you to make my life easier as a consumer, not as, like, as a, a normal AI consumer, and not as like a consumer of AI fanfiction. So that's kind of my proposal here, right? It's like, the, the, these long-horizon agents are the crux of all the problems we have.

• 33:49Cal Newport?

  And I don't like, by the way, that commentators have completely made this sort of weird experimental choice of long-horizon AI agents to be synonymous with AI. Like, so what's going, you know, we're in a world where of course, we have to have a program for days, prompt an LLM, and say, what should I do next? And do it blindly. Of course, we have to do that. And just like we predicted, that world is like, it's, they don't do what a human would do. Alpha 9000 is almost here. But it's an arbitrary of all the different things you can do with language model technology. That's just one thing you can do. So let's just stop doing that.

• 34:33Cal Newport?

  So again, like, you know, we, so what would we do instead, right? Okay, you can have narrow depth prompting, that's fine, but also, we need just better architectures, right? If you want to build something that's going to, like, with semi-autonomy do something, like, how do we do that, then? Build it with another architecture that's way safer. So let's go back to my, my, and I'll get off my soapbox, but let's go back to my sample policy solution here, where I said, look, if you have a, an LLM-powered long-horizon prompt loop agent, you can only do a limited number of prompts without human intervention or supervision. Fine. But I say, if you have some sort of autonomous, like, AI-powered agent, that has a symbolic, human interpretable encoding of its potential plans, like, you could have any more modular architecture, like, that's fine, because that's super easily monitorable.

• 35:18Cal Newport?

  Right? Like, just to use like a narrow example, if you look at an AI system that plays a game, you look at a Cicero system that plays the, the strategy game Diplomacy, for example, modular architecture, it's not just an LLM that you just prompt and say, what move should I do next? What move should I do next? It's a modular, planning engines and evaluation engines and language models to understand the, the text of the other players. But they have a symbolic format for encoding potential moves. Human interpretable. And they have a separate engine that then evaluates those potential moves, symbolically in this case, actually. It runs simulations like you would with a chess engine to see, well, if we did this move and they did this and this, which of these moves gives us the best opportunities going forward? And you know what?

• 36:02Cal Newport?

  Because these potential plans are symbolically encoded, these systems are super moderately moderable, right? Like, you could submit each plan to just a human-written program that says, we throw out plans that, you know, whatever, attack Australia. And then it will never attack Australia, because that plan is just not considered. And so anyways, I, I, I think we can have longer horizon autonomous systems. I don't want to say we shouldn't have those. I'm just saying this architecture of driving off LLMs is dumb and dangerous. And it's been proven, I'm sorry, AI companies, you had a chance, and it went terribly awry. These other ways of doing it with modular architectures and human, human interpretable symbolic representations of plans or what have you, this might take a while to get right, and they'd be pretty task-specific.

• 36:46Cal Newport?

  But as long as we have a ban on no one can do the LLM thing, then like, we have time to do it. No one feels like they're being scooped by another company. So that's what this all leads me to think, is that this insistence on LLM prompts as a driver of action and long-horizon agents is ruining everything else. It's scaring the hell out of everybody. And it's, as we can see, is rightfully creating pushback against innovations that without those agents would actually just be like, oh, great, let's make things smaller and cheaper so they're more useful. So there we go. We gotta find some way to get rid of this idea of just LLM-powered agents as somehow synonymous AI or the future. There's way better systems to build, they're harder, so let's just stop anyone from building

• 37:29Cal Newport?

  the ALL-powered ones, let's build useful tools in the short term with LLMs, and safer, more effective, long-horizon agent-type tools. We can spend a little bit more time building those in a way that we don't have to fear them. You know? Stockfish never goes rogue. AlphaGo never tries to trick the other player into giving internet access. Diplomacy, the, the programmers of Cicero turned off line. They're like, oh, we won't do deception as a strategy. Simple if you have symbolically encoded plans, because you just don't consider plans that include deception, right? Like, so there are ways to build long-horizon agents that are way safer, more monitorable

• 38:11Cal Newport?

  than this terrible idea of just blindly executing the output of LLM. So let's just stop building those agents, give people time to build better ones. In the meantime, there's all sorts of cool, useful tools we can do with LLMs. Let's get back to exciting consumers. I want a lot less Yankowski and a lot more Steve Jobs in the conversation around AI. And we could get there, where it's like, oh, that's really cool, I can do this now, it's going to make my life better. That's what we need. And I'm convinced that getting rid of long-horizon LLM-powered agents as a strategy, maybe that gets us there, maybe not, maybe I'm missing something. But anyways, whether or not you like my, my final analysis, I do think it's useful to understand what's probably going on with Astro, why they would do it, and why people are worried about it.

• 38:56Cal Newport?

  I hope that's useful.