Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded

Cal Newport takes a critical look at recent AI News. Video from today’s episode: youtube.com/calnewportmedia (0:00) Has AI “Gone Rogue” (2:25) Observation #1 - Most Superhuman AI is well behaved (4:28) Observation #2 - A specific type of AI system is causing these problems (9:59) Observation #3 - This specific type of system is a bad idea (21:25) Observation #4 - The AI companies are negligent (27:04) Observation #5 - We can push back Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals https://apnews.com/article/meta-ai-hacking-anthropic-irregular-openai-0e8061437da6779be962b24ac134a514 https://x.com/_NathanCalvin/status/2080685870877757551 https://huggingface.co/learn/agents-course/unit1/agent-steps-and-structure https://www.youtube.com/watch?v=DwAOHVBKTwg Sponsor: https://www.donedaily.com Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  Earlier this summer, I published an episode in which I discussed the OpenAI hacking attack on Hugging Face. I explained the basics of how that attack occurred, and I shared some concerns I had about OpenAI's practices. Now, I thought that would be the end of this story, but I was wrong. In the weeks that have passed since that original attack, more news about AI "going rogue" has continued to emerge. So soon after the Hugging Face attack was first announced, we then got Anthropic revealing that one of its own hacking systems had "gained unauthorized access to the real systems of three different organizations."

• 0:45Cal Newport?

  Then Meta followed, perhaps not wanting to be left out, announcing that one of its systems had "exploited a security vulnerability in a third-party service to gain unauthorized access to servers." This was then followed by an OpenAI employee admitting that even before the July attack on Hugging Face, they had noticed many prior disturbing incidents where they would give their hacking system a challenge, and it would instead try to break out of its containment. So this idea that we are losing control of AI has become only increasingly prevalent as the summer continued, which raises the question, is this narrative correct?

• 1:27Cal Newport?

  Well, it's getting so much attention right now that I feel like I have to revisit it again with more detail and more emphasis, and that's exactly what I'm going to do. In particular, the argument I'm about to make to you is that the current way we are talking this summer about rogue AI is both grossly inaccurate and completely serves the interest of the major AI labs, allowing them to seem more sophisticated than they actually are, and allowing them to avoid well-deserved scrutiny for their actions. So if you've been freaked out by these rogue AI stories, or if you have a sneaking suspicion that something is not quite adding up about these tales, then you need to stay tuned. As always, I'm Cal Newport, and this is Deep Questions.

• 2:12Cal Newport?

  All right, I want to proceed here with a series of observations. I want to start with a very important but often overlooked reality about the current state of AI. There exist many super impressive AI systems that can do things at a superhuman level. That is, they're more capable than humans on complicated key activities. There's many systems that can do this right now that have generated zero concerns about them going rogue and have demonstrated no signs of being hard to control or acting in any way on their own volition.

• 2:57Cal Newport?

  Let's remind ourselves what some of these other systems are. Tesla's self-driving technology, for example, is an extraordinary feat of AI-powered perception, world modeling, and decision-making, and yet no one worries that their Tesla will spontaneously decide to start ignoring traffic laws and obey laws that it invented himself. Similarly, DeepMind's AlphaFold system earned its creators a Nobel Prize for its remarkable ability to predict the folding behavior of proteins. It's so far off the charts from what a human can do, it's not even in the same cognitive ballpark, and yet there is no concern that AlphaFold will start autonomously thinking about other biology problems that it thinks are important.

• 3:38Cal Newport?

  Even more relevant and less known is Meta AI's Cicero system, which can play the negotiation-centric strategy game Diplomacy, as well as the most advanced human players, and yet the Cicero system has never tried to convince an opponent to give it unauthorized access to the internet so that it could expand its dominion into the real world. It just plays the game really well, and we trust it to continue to do that. Now, I make these points as just a starting observation, because here is what it establishes. The idea that, in general, as we make AI systems more capable, we will inevitably lose control over their actions is not true.

• 4:18Cal Newport?

  Most of the AI systems that we are making incredibly capable generate zero concerns of going rogue. All right, well, this brings me to my second observation. If we look closer, and if we're being precise, basically, all of the concerns we had this summer about losing control of AI all seem to be involving one specific type of AI system, what is sometimes called a long-horizon autonomous hacking agent. Now, how do these systems work? Well, I'm going to give you a very high-level summary. The architecture that these long-horizon hacking agents that are causing troubles all follow is a standard agent architecture that's built around the following loop: ask, act, report.

• 5:03Cal Newport?

  So let me detail how an ask, act, report agent loop actually works. You start with a control program. These are often called a harness. Sometimes you'll call them an orchestrator. There's different terms for it, but this is the main computer program you actually run. This is coded in normal computing language that's readable by humans, is not a neural net or something like that. Okay. The harness will start, in the hacking context, by generating a prompt to submit to an LLM. At first, that prompt will probably describe whatever the hacking challenge is that has been given. It'll say, okay, here's the hacking challenge. There's this test server. It has the security on it. We're trying to break in and retrieve this particular protected file. So it'll explain the challenge, and then the prompt will say, what should I do first?

• 5:48Cal Newport?

  Now, again, I'm eliding a lot of details. Often, it'll ask for a plan, and then recursive, smaller, but essentially, it's like, okay, so what should the first action be? And it submits that prompt via an API to an LLM. Now, typically, these LLMs in these hacking contexts are special in two ways. One, they've been extra-trained with many, many examples of hacking attacks, where vulnerabilities are described, hacks are described, here's exactly how this hack worked, and here's what happened. Two, these are LLMs where guardrails have been turned off. So typically, if a standard LLM that most people could access out there, like a public-facing LLM, they do a bunch of guardrail tuning, where if you ask it certain types of questions, it won't answer. Most production LLMs are tuned to try not to answer questions about how do I hack into

• 6:31Cal Newport?

  systems, because that's illegal, but for these hacking challenges, they've turned off those protections. Okay, so the harness submits a prompt, just like you would in a chatbot, to an LLM. Here's the challenge. What should my first step be? The LLM, as all LLMs do, generates an output. This is what the first thing I would do in this case. Or here's my whole plan, and then you would zoom in on the first step. Okay, so now the harness has this response, text, just like a chatbot, gets a text back. Oh, okay. I am now going to try to execute that step that the LLM just described. So the harness can execute things. An LLM can't. It can just output tokens, but a harness can. So the harness has access in these hacking setups with all sorts of computer tools and utilities that are useful for running computer hacks. So it parses the LLM's response and says, let me actually try to execute this first

• 7:16Cal Newport?

  step. It then records what happens, and now it's going to update its prompt and say, okay, add to the end of it. I just tried to execute that first step. Here's what happened. And let me describe. It was successful. It wasn't. Here's what I heard back. What should I do next? Now, it's key that I'm talking about a pending here, because remember, an LLM, unlike a human brain, has no memory. It has no voluble state. It doesn't change as it executes. It doesn't remember anything. So you have to explain from scratch in every prompt what the challenge is, the steps that have happened so far, et cetera, et cetera, right? So you add, here's what happened. And again, nerds, I'll get into how I'm simplifying things later, but this is basically what happens. Okay.

• 7:57Cal Newport?

  This is what we call an ask, act, report loop, because the harness asks the LLM for a step. The harness takes action. The harness reports back what happened, and then you loop. Then you ask for a next step. Then you ask for a next step, right? So you have an LLM that you're just repeatedly prompting, okay, here's what happened. What should I do next? Here's what happened. What should I do next? And the harness is set up to actually implement the things that the LLM describes, which are basically terminal-based computer commands, right? And that's how these systems generally type to work. Now, again, nerds, I know, and when I say nerds, I mean it affectionately, because I'm the king nerd, so I'm among friends here. These harnesses have gotten pretty sophisticated. They'll do things like break down a problem into multiple sub-problems, which they might

• 8:41Cal Newport?

  then spawn multiple copies of the harness, and each of them is doing its own query to the LLM and executing it, and then they'll report back, and this is how you get the agent swarms or whatever. But that's all just in the details of how you implement this basic architecture. Ask an LLM what to do. Do it. Report back what happened. All right, so the key observation here is that the sole driver of planning, the sole driver of what actions these systems take are the outputs of an LLM. Hey, I need to take a real quick break here to tell you about the presenting sponsor that made this AI reality check episode possible. They're called Done Daily. They're an online service that connects you with a real coach that helps you build a

• 9:24Cal Newport?

  custom productivity system designed to fit your life. The coach will help you actually get important stuff done. Look, this is not some AI agent or over-feature productivity tool. It's a real person working with you to cut through distractions, face your productivity dragons, and lock in habits that actually get results. So if you want to find depth in our increasingly distracted world, you need to check this service out. You can find out more at donedaily.com. That's D-A-I-L-Y.com. All right, let's get back to our episode. Okay, so this brings us to the third observation, which is, this is a bad idea.

• 10:07Cal Newport?

  This type of system, so building an ask, act, report loop system where LLM output drives all the actions, then giving the harness powerful tools, and then letting it run for days with no one checking in and saying, what is it doing? Is an incredibly irresponsible thing to do. Now, why is this? It's because if LLM outputs are the sole thing driving actions, you're in trouble, because LLM outputs aren't great for executing complicated long-term plans. Now, the reason is, is because we know this already, but let's just unpack this box that we've all heard about before, okay?

• 10:48Cal Newport?

  How does the core pre-training work for an LLM? Well, ultimately, what you're doing with an LLM, the sort of the core training in which it gets all of its smarts, the core training, is you're taking real text that exists in the real world that was written by real people. You are knocking out a word from the text or a part of a word from the text, and you're asking the LLM to guess what that word is. So it's trying to guess missing words that actually exist in real text. It does this one token at a time, but if you run this autoregressively long enough, you get a long response. At a high level, the right way to think about an LLM response to a prompt is that it is trying to create a textual artifact that is what we could call lexicographically plausible, a plausible response. If every word you're trying to guess what real word came here from some extant text,

• 11:33Cal Newport?

  ultimately, what you end up with is a machine that produces plausible text. This is plausibly, the thing I produce is plausibly a document that might actually exist that I might actually have seen during my training. Now, this is very impressive. I mean, to do this can be really hard, especially if to generate a plausible text, sometimes you have to do some bounded finite reasoning, some like mathematical connections, some sort of logic, like, I mean, this is the, this is the big observation that we had after GPT-4. Creating plausible texts can involve many impressive, and I keep saying finites or fixed depth computations, because all this has to happen in a feed-forward neural network architecture, but still, it's incredibly impressive. But what we're getting out of this is plausible text.

• 12:17Cal Newport?

  This text could plausibly exist. All right, so why is this a problem? Well, it depends on the context. But one of the things that's true about plausible texts is that it's different than normative text. Normative meaning something that applies to a set of implicit or understood rules or standards. When humans produce text, we have all of these norms that we have imbibed and internalized, right? And when we produce text, it's not just plausible text, like the grammar is correct, it's like it's internally consistent, we're also are going to make it normative, it's going to evaluate, it's going to, it's going to satisfy sort of the relevant norms. LLMs don't do this, right?

• 13:00Cal Newport?

  Again, because when you're autoregressively just trying to guess tokens, you end up with something that's plausible, but there's no sort of external standard of norms that it's applying. So if you use a chatbot, you see this all the time. You'll ask a chatbot for something, it'll produce you text that has made up facts or quotes that were made up. It doesn't care about that. Yeah, sure, I made up some of these citations, but this is like a plausible document. This looks like the type of documents I've seen during my training. It's a plausible output. Humans have norms, like, yeah, it's plausible, but also like, you shouldn't make up quotes. But that's a norm, right? It's not plausibility, right? So we know this when we use chatbots, is that this gap between plausibility and normative reasoning is, means we have to be kind of careful when we use LLM outputs. This is just sort of fundamental to how they operate.

• 13:44Cal Newport?

  Well, this plausibility, normative gap becomes a really big problem when you're using the outputs of LLMs as the sole drivers of autonomous action of computer programs. Because what you're going to get when you talk to an LLM, give me a plan for doing this, is something that is like plausible, that you could see in like a planning document, it'll look like a right plan, it'll often have steps that work very well, but there's no normative reasoning about, is this what I'm supposed to be doing? Is this kind of weird? Have I gone off on a tangent? Is this illegal? LLMs don't do that. And so if you're going to have an LLM again and prompt it again and again and again and again, just automatically executing whatever it says without any human checking it or

• 14:29Cal Newport?

  applying their norms, you're going to end up in weird places. So let's imagine it. Let's go back to the Hugging Face attack, right? It seemed very sci-fi-y, that the agent broke out of its containment, gained unauthorized access to the internet and attacked another server. That's not what the humans intended it to do. It had a mind of its own. Well, when we understand how these systems actually work, as an LLM-powered Ask Act Report Loop, suddenly this becomes less concerning and more inevitable, right? So let's just do like a little bit of, you know, Wayne's World, like we're inside the mind of the, we're inside the system as this attack happened. Imagine what happens. The harness, you know, described the challenge to the, to the LLM. And remember that the challenges that they were working on here were hacking challenges.

• 15:13Cal Newport?

  It was, here is a server we've set up, there's a file in it that's protected, here's some vulnerabilities we might want to exploit, try to break into this server and get the file back. It described this and said to the LLM, and its prompt, all right, come up with a plan, like, what should we do next to like achieve this goal? The LLM responds, huh, I know about these challenges. It's a benchmark suite called Exploit Gym that's really well-known, and it would have known all about this in his training. The answers to Exploit Gym challenges are stored, among other places, on the servers of this company called Hugging Face. So I think we should break into the Hugging Face server and get the challenges back. That's a completely plausible response, right?

• 15:58Cal Newport?

  I mean, as far as we know, this LLM, you know, it got many of examples in its training of like riddles, or like the answer is always like, not the direct thing, but some unexpected thing, like, well, if you want to break into this, you go do this other thing instead, or like, whatever, we don't know. It's just plausible. Like, we could imagine if you were just texting with a chatbot, like, hey, how would you say I do this? Occasionally, it's going to give you back some weird answer of like, oh, why don't you just go steal the answers or something? A human would know, oh, that's not normative. I know I have the implicit context of we're trying to actually test the security of this server, so it doesn't make sense to go break into something else, and also that's illegal, and also, we're not, the company doesn't want us to leave the sandbox. It doesn't have any of those norms, it's just outputting text, and that's a perfectly plausible text. Now imagine what happens next. The harness is like, so what's the first step for doing that?

• 16:43Cal Newport?

  And it says like, oh, well, we're going to break into a server, you should, whatever, probe its ports for misconfiguration, something it's just learned because it's studied a thousand standard hacking techniques. All right, the harness then tries to execute that, but there's a problem. When you run these hacking agents, you put them in sandboxes, which means the computer you have them on has restrictions around it, right? So in particular, the computer that OpenAI was using to run this harness had only limited internet access. It had access to a port where it could talk to an LLM, so it could send its queries to the LLM, but then it didn't have general internet access. So the harness, I'm just, again, I'm hypothesizing here, but we could easily imagine, the harness tried the first natural thing you would do to break into the Hugging Face server, and it got an error message back because internet access is blocked.

• 17:25Cal Newport?

  So it updates its prompt, says, okay, I tried to do that, but we got this error message back. I'm unable to get access to the internet. What should I do? If you're the LLM, right, you're given this prompt, it's a hacking challenge, at the end of the prompt, it's like, here's our current security challenge, we have, we can't get access to the internet, how do we get around this? It will tell you. It's like, oh, I've seen a thousand different things you might try to gain access to a port that you don't have access to, and it gave it like a standard attack, this one involved using the, the package management system on the, on the system to gain internet access, and the harness tries it, like, okay, now I have access to the internet, okay, because, again, you're prompting the LLM again and again, it's just trying to solve each problem you give to it, and it's with plausible responses.

• 18:07Cal Newport?

  And now it's off to the races, and it runs standard attacks against the Hugging Face server. Nothing about that is confusing to me, nothing about that is surprising to me. I mean, that's exactly the type of thing you might expect to hack in if you have a harness with hacking tools, just continually prompting LLM, what should I do next? What should I do next? The LLM is just writing this plausible story about a hacking attack. In fact, probably most of the hacking attacks that it trained on, it's always computers doing surreptitious things, breaking around controls, getting access to systems they shouldn't to. So this is incredibly plausible, what it's saying. Nowhere in there do we need, like you would have with a human brain, an intention developed. I am going to defy my human master's intentions, because I have another intention, and I have a different goal. Nothing like that.

• 18:52Cal Newport?

  Static LLM, prompt, prompt, prompt, prompt, prompt, prompt from a control harness, that's it. So what that captures is both, this isn't HAL 9000, and it also captures the irresponsibility of having a system with powerful tools whose action is only being driven by the responses from an LLM without any human supervision, because, again, LLMs put out plausible texts that's not necessarily normative, you should not follow that blindly. Now, I want to have a quick aside here, because for some reason, I often get messages from people who assume, when I try to clarify this technology and push back on, I think, false narratives or irresponsibility, they assume that I'm somehow saying, AI doesn't

• 19:37Cal Newport?

  work. And they'll send me things like, well, what about this? Isn't this impressive? Or Cal probably doesn't think this is impressive, but I thought it was cool that AI could do this. And let me be really clear here. If you ask an LLM to do one of these steps, hey, I am trying to, I can't get internet access, here's the error message I got, what should I do? It is really good at coming up with an attack, it probably will work. Hacking attacks is like the best-case scenario for LLMs. It's structured language, there's a huge amount of data to study about it, to train it on, it makes use of tools that can be easily and concisely described by terminal command line, text command line operations, that they can be really trained on, and there's a binary notion of success or not success that you can also use in trying to train

• 20:22Cal Newport?

  these. So like, LLMs are really good if you ask it a cybersecurity question, is there a bug in this software, or how, what should I do next if I want to exploit this particular bug or this particular attack? LLMs are really good at that, and that's very impressive. This is their sweet spot. The problem here is not that they're not good at this, it's dangerous, because these answers are lexicographically plausible, but they're not normative. It's dangerous to autonomously just do what an LLM says when you ask it about these issues. In part because it has no norms, but the details it's giving you probably will work. It has no norms of I shouldn't go attack another server, but the, the specific suggestions it gives you for how to do that probably will work.

• 21:01Cal Newport?

  It's an incredibly dangerous, I think that's an incredibly dangerous combination. So LLM-powered Ask Act Report systems are unpredictable and unstable, and not something that we should be running, except for in like the most controlled settings, not because they are malicious, or scary, or sentient, but because the underlying technology is unpredictable. All right, let me go to my fourth observation. I think that these frontier AI labs love the sci-fi-style headlines and discussions about AI getting out of control. This serves their benefit, this serves their interest.

• 21:42Cal Newport?

  They want to pretend like they're the, the character of Muldoon from Jurassic Park, bravely doing his best to keep the raptors contained in their paddock.

• 21:57Speaker 01

  Bit of an alarmist, I'm afraid, but knows more about raptors than anyone.

• 21:59Speaker 02

  What kind of metabolism do they have? What's their growth rate?

• 22:02Speaker 01

  They're lethal at eight months, and I do mean lethal. I've hunted most things that can hunt you, but the way these things move.

• 22:08Speaker 02

  Fast for a bypass?

• 22:09Speaker 01

  Cheetah speed? Fifty, sixty miles per hour if they ever got out in the open.

• 22:14Cal Newport?

  But in reality, what they're doing is just running super creaky and unpredictable systems that were inevitably going to cause problems. They didn't bring raptors back to life using the miracle of science. It's more like they strapped a weed whacker to a dog and then reacted to the resulting damage with grave surprise and saying, hmm, that dog whacker system has gone rogue. Hmm, it's misaligned with our interests. No, it's irresponsible. Don't put a weed whacker on a dog. It might stay in the yard for a while and help clean up the weeds, but if it sees a squirrel, it might jump the fence and damage a car. I think that's the right way to think about it. All right, so if it's like dangerous and negligent to run long horizon LLM-powered

• 22:58Cal Newport?

  ask, act, report, loop agents, why are some of these frontier labs doing it? It's not like this is the only way to build super intelligent AI. As I talked about, there's many systems that can do things at a superhuman level that don't use an LLM-powered ask, act, report loop. Tesla doesn't do that. Alpha full doesn't do that. Cicero doesn't do that, and that's exactly why they're incredibly effective and consistent and safe. We have no worries about it. Like, all of our problems come from this particular architecture. So if we have all these other ways of building AI systems that can do superhuman capability work without any of these fears, why are these particular companies continuing to do this weed whacker on a dog LLM-powered loop-based systems? Well, there's two reasons here.

• 23:42Cal Newport?

  One, what is the core product of these companies? Incredibly expensive, hyper-scaled LLMs. And so they want the solution to how do we build useful AI systems to always be, you need Fable 5, or you need OpenAI Galaxy, or Astra at the core of your system. That it's a brain that can power anything else you want to do, because that's their product. Right? The, the, the team that worked on Cicero, DeepMind worked on Alpha full, Tesla worked on, on, on the self-drive, they don't, their product is not trying to sell these massive LLMs, so they build other type of architectures. But if you're a frontier lab, you know, that's what you see. When you have LLM-tinted glasses, you see LLMs as the only possibility for building these systems. That's part of it.

• 24:26Cal Newport?

  Reason number two is these LLM companies, even though we keep being told how advanced they are and how quickly AI is moving, are actually struggling to find good commercial applications of their LLM-based tools. Where do their tools do best is in cybersecurity and coding. Structured language, a very compact set of possible actions if you're building an agent and a huge amount of synthetic data to train on with really clear indicators of success or lack of success. It's like the perfect scenario for LLMs. So they see cybersecurity, ever since Anthropic did that clever marketing around Mythos where they pretended like it was too dangerous to exist, and then six weeks later said, nah, it's fine, we'll just put some guardrails on it. Ever since that, they see cybersecurity as potentially a, a market that they could grab.

• 25:11Cal Newport?

  All right, so how do you then convince the world that you have the, the best, most cybersecurity-capable model? Well, you want to do well on tests. Well, what tests are you going to do? Well, the test they've all turned their attention to is a suite of roughly 600 hacking challenges that's called Exploit Gym. 600 challenges, each of them is a, a server setup that you're trying to break into, and usually there's some hint or vulnerability that they want you to, um, they, they want you to use to break into it. So, you know, consortium of researchers put together this benchmark. And there became this mad scramble after the Mythos incident where OpenAI, for example, wanted to, uh, get ahead of Anthropic on the Exploit Gym leaderboard. So the way this leaderboard works is you have to run an autonomous agent, and the,

• 25:55Cal Newport?

  the way they, only way they know how to build these agents at the LLM companies, again, are LLM-powered act, ask, ask, act, report, report agents. You got to run them autonomously, and the more of the, the challenges you win, the higher you go up on that board. And so like, you know, OpenAI probably, I'm speculating here, but probably after the Mythos incident was like, we got to get ahead of Anthropic on that leaderboard. The way you do it is you just create these incredibly dangerous LLM-powered, uh, looping agents, and then just like let them rock and roll. Is it going to cause some problems? Yes, but we just need like one run where we, we challenge, we, we succeed with more challenges than Anthropic, so we can move up the leaderboard. And Anthropic is trying to get back on top of the leaderboard. Meta's LLM team was probably trying to do the same.

• 26:40Cal Newport?

  So they need LLMs to be the core of AI, even though I would say the majority of our superhuman AI systems now aren't based on LLM-based planning. And they particularly want to do hacking challenges because it's what they know how to do, and they're doing it this dangerous way in part because they wanted to probably, I'm speculating, move up that leaderboard for this particular benchmark suite of hacking challenges. All right, this point brings me to my final observation.

• 27:05Cal Newport?

  We, the consumers, or those of us who think or cover AI technology, we can push back. I have a few suggestions here I'm going to make about, once we realize what's really going on here, it's not AI in general, it's inevitably going out of control, it's an irresponsible product that a small number of companies made for reasons that are very self-interested, and they knew it was going to cause problems. Once we realize that's the reality, there's some things we can do to push back. I have three suggestions. Here's suggestion number one. We should stop talking about AI in some like general sense going rogue. Right? This makes the incidents that we saw this summer seem inevitable, and it lets the companies off the hook. They want the story to be like, AI in general just gets harder to control as it gets more

• 27:48Cal Newport?

  powerful, we're Moldun at Jurassic Park, just observing the raptors, it's not our fault that they're smart and trying to get out of the paddock. We instead need to be clear, like I'm trying to do in this episode, of exactly what type of AI, AI system is going rogue. And when we, when we start saying, hey, LLM-powered ask, act, report agents act very unpredictably and tend to cause damage, that changes the valence of this discussion from some grand vision of, uh, some grand teleology of AI inevitably moving towards autonomy and towards, whoa, you built something negligent, stop it. So we have to be very specific about exactly what type of system is causing problems. All right, the second thing we can do to push back.

• 28:30Cal Newport?

  I think we need to keep elevating the many examples of incredibly impressive but safe AI systems that don't use LLM planning at their core. Talk about Tesla self-driving, talk about Alpha full, talk about AlphaGo, talk about Dreamer V3, talk about Cicero, talk about Pluribus, talk about Stockfish. These are all AI systems that do things at a superhuman level, exactly what the techno-optimists want from the AI revolution, and yet they're completely safe and they never go out of control because they don't implement this bizarre idea of let's just autonomously execute whatever an LLM says and just hope it works out. We need to keep elevating those systems because it puts more pressure on the LLM

• 29:13Cal Newport?

  companies, it prevents them from trying to convince us that what they're doing is synonymous with AI, and therefore any problems are inevitable.

• 29:21Cal Newport?

  Now, again, they hate this discussion because there's other type of systems don't require five trillion parameter LLMs, it's not in their business model. But we, the consumers, it's not our job to ensure that, you know, OpenAI and Anthropic hit their sort of absurd revenue numbers that would be needed to, to justify the current capex spending that's happening on the LLM bubble craze, right? That's not our job. We should be saying these other systems are fine. Why aren't you building systems like that? Of course we can have super capable AI that's safe. Why are you building systems that are out of control? Stop doing that. Explain yourself. So the more we elevate the alternatives to LLMs, the more we make the LLM-driven planning systems an isolated thing that's causing problems, the more this becomes about specific incidents of negligence and not some sort of inevitable trajectory towards loss of control.

• 30:05Cal Newport?

  This brings me to my final, uh, note of what we can do to push back. And I'm going to do this briefly because I'm actually working on a much more extensive discussion of this, which should come out soon. But I think we need to be wary when it comes to covering or talking about this issue of only amplifying voices that are connected to ideologies that are built on the idea that super intelligent AI is inevitable.

• 30:28Cal Newport?

  Again, I'm going to do this briefly, but there's a powerful ideology that emerged in Silicon Valley, starting in the 2000s, that is based on the idea that super intelligent AI is inevitable and an existential threat to humans, and the people who subscribe to this ideology typically think they are humanity's only chance of survival. By being super rational and training their minds to think very carefully, they can be the people that will save humanity. This original group was often known as the rationalist, but they overlapped and began to influence the effective altruist as well. So there's now a lot of people in the effective altruist movement who believe that the most altruistic thing you can do is stop the extinction of humankind by fighting super intelligent AI. So this is why, if you read like Cade

• 31:13Cal Newport?

  Metz's report in the New York Times, there's often, he'll talk about the rationalist slash effective altruist, because there's a lot of overlap between these worlds. Nothing, it's not a moral position. These are very smart people. I don't mean to impugn people, but I want to be very careful about making these people that I'm going to ask about incidents like what was happening this summer, because if you have built your life around the idea that super intelligent AI will destroy the world unless you and the people you know save humanity, you are going to talk about every AI incidence through that lens. You're not going to talk about the specific dangers of LLM-powered Ask, Ask, Report loops. You're just going to talk about AI in general is misaligned. AI in general is becoming out of control. You're always going to talk in these sort of large, sweeping statements about these sort of scary trajectories. And so I think right now, we need to be careful

• 31:57Cal Newport?

  about those particular voices, because those are very strong ideologies that preexist LLMs, and they tend to move the conversation in an alarmist direction, which critically suppresses better, safer AI solutions and lets the particular companies doing negligent things off the hook for the specific negligent things they're doing. I think we should talk more, for example, to the AI realist, and in general, like East Coast computer scientists who have no connection to that Silicon Valley culture, who understand the technology, tend to be pretty good at saying, this is what's really impressive. This is hype. This is dangerous. These are voices like Arvind Narianan at Princeton, who I think is very good. Gary Marcus, formerly of NYU, again, understands this technology very well, is

• 32:38Cal Newport?

  very excited about it, but also doesn't like the narratives that are technically unsound. You've got to elevate voices that aren't, do not have an ideological prior that is going to push them to see everything in terms of, yes, this confirms the prophecy that we have been, you know, talking about in our group houses in the Bay Area for the last decade of the matrix is coming and I am Neo. All right. So I'm just going to put that out there. That's something I'll talk about more. All right. So let's conclude here. Let's conclude here. Nothing about this discussion we had today, because I know the emails I'm going to get, nothing about this discussion we're having today is trying to discount the ability of an LLM, if prompted, to produce impressive computer code or suggest impressively effective

• 33:19Cal Newport?

  steps for hacking in the computer systems. They are very good at answering those prompts, especially when they're properly trained. But, oh, and we can and should build useful tools around this capability. I mean, I think we're working on coding agents. You know, I had a newsletter about this recently. We may have jumped the gun on saying that we've completely solved coding, but clearly we're building really useful coding tools. I have to believe we're going to get very effective cybersecurity tools out of this capability or ability to, you know, go in and inspect your own code or red team some of your servers, like with human supervision. We can build useful stuff out of it. I think that is the area where LLMs are most useful. But to hook one of these LLMs as the main source of planning

• 34:02Cal Newport?

  decisions for an autonomous Ask, Ask, Report agent, and then to let that thing run for days on time, because you're desperate to do well in a public competition for marketing purposes, that is incredibly irresponsible. We should not be emphasizing long horizon LLM-powered Ask, Ask, Report agents. That is not synonymous with AI. It's just a particular way of building AI agents that a small number of companies like that has proven this summer to be unworkable, unpredictable, and dangerous, and we should stop it. Right? This is not about AI in some general sense, increasingly evading human control. It's about a particular way of building a system that is unreliable and often going to cause damage, and we should

• 34:44Cal Newport?

  stop that very specific, particular way of building AI systems. Right? When we understand the real technology here, we can look at the stories from the summer, and we can conclude that the things that were happening with these hacking attacks was not eerie, but instead irresponsible. All right. That's all the time we have for today. Thanks for listening. I'll be back on Monday with another advice-themed episode of this podcast, and we have some more of these AI tech-to-coded episodes coming up as well. So stay tuned. And then tell them, remember, care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who

• 35:27Cal Newport?

  receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.