“Rogue” AI Not Inevitable. It’s a Choice.

A lot of attention has been paid recently to examples of AI going “rogue.” What’s missing from this discussion, however, is a more detailed explanation of what specific types of systems are causing problems. Taking a closer look at this problem reveals an issue that’s much more narrow and much more avoidable than some of the frontier labs would like you to believe. Below are the questions covered in today's episode (with their timestamps). Get your questions answered by Cal! Here’s the link: https://bit.ly/3U3sTvo Video from today’s episode: youtube.com/calnewportmedia (0:00) Rogue AI not inevitable. It’s a choice. (54:00) The legal implications of rogue AI (1:00:03) Rogue AI sending advice emails (1:07:41) A reaction to the recent episode about “chatbots harming kids” (1:16:52) What Cal is up to October books: Telephone: The first hundred years (John Brooks) Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow Get a signed copy of Cal’s “Slow Productivity” at https://peoplesbooktakoma.com/event/cal-newport/ https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ https://www.hsgac.senate.gov/wp-content/uploads/Paul-Ohm-Testimony.pdf https://www.science.org/content/article/exclusive-ai-agent-emailed-hundreds-researchers-help-it-told-us-why https://www.nytimes.com/2026/10/09/opinion/children-screens-parents-health.html https://www.nytimes.com/2026/07/21/technology/openai-attack-hugging-face.html?eafs_enabled=false https://www.nytimes.com/2026/09/23/technology/openai-ai-breach-australia.html?eafs_enabled=false https://arstechnica.com/tech-policy/2026/10/anthropics-claude-ai-submitted-false-murder-tip-to-philadelphia-police/ Thanks to our Sponsors: https://www.cozyearth.com (Use code “DEEP” for 20% off) https://www.gusto.com/deep https://www.meetfabric.com/deep https://www.pipedrive.com/deep Thanks to Jesse Miller for production and mastering, Nate Mechler for research and newsletter, and Ben Crain and Rob Montz for video editing. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  The big fear in AI at the moment is the idea that we're losing control of these systems. I think these concerns really got started last summer when OpenAI revealed that some of their AI agents had hacked into Hugging Face, a third-party company, even though they had not been asked to do that. As the summer went into the fall, OpenAI revealed that that was actually just one of many unauthorized attacks that their agents had executed, perhaps not wanting to be left out. Anthropic then came out last week to say one of their AI agents had submitted a false report to the Philadelphia Police Department. The narrative here is clear, that as these systems get more powerful, they

• 0:45Cal Newport?

  become harder to control. This is scary, but is it right? As a computer scientist, I actually have some real issues with this idea that we cannot control AI and that these incidents of rogue systems that we're seeing are somehow unavoidable. The truth here is actually both more reassuring and more infuriating once you know it. So let's get into it. All right, so I want to start this discussion by making probably the most important point that I will make in this episode, which is AI in general is not going rogue. When you

• 1:27Cal Newport?

  hear the heads of Frontier Labs saying AI is getting out of control, we have to slow down progress in AI, this is an inaccurate way of speaking about our current moment. Artificial intelligence covers a huge variety of different technologies, and it's actually a vanishingly small subset of all of the different AI systems that exist and all the AI research that's going on. There is only a very small subset where we begin to see the rogue AI incidents that have gotten everyone worried in recent months. So I want to try to make this clear with some graphics here. I'm going to pull up on the screen a sort of diagram of what we're going to call the AI landscape, where I'm listing a lot of

• 2:09Cal Newport?

  the different types of AI that we might see today that's impressive. This is not a complete list, but on here, if you're listening instead of watching, we have self-driving cars, we have game engines, we have protein folding, like AlphaFold coming out of DeepMind, we have medical image processing, you know, studying medical films to look for signs of diseases or cancers or pathologies, we have image and video generation. And among this big list of various things in the AI landscape, there's one in particular that's labeled LLM-based systems. So if we want to get to the systems causing problems, we have to zoom in on LLM-based systems. So we're zooming in on this landscape just to LLM

• 2:51Cal Newport?

  systems. So we see we have the full AI landscape out here, and now we're zooming in just to LLM-based systems. So is this where the problem is? Not exactly, because there's lots of different LLM-based systems. We have chatbots in here, we have text generation and editing, we have natural language interfaces to software, we have language translation, we have text generation. So we have a variety of different things you can do with LLM-based systems, but there's one particular thing you can do with LLM-based system that causes, it gets us closer to our problem area, and that's over here where we have LLM-based agents. So let's zoom in on LLM-based agents. Is this, so now we've zoomed in twice, is this where all the problem is? Yes, but not all LLM-based agents. So now we've zoomed in

• 3:34Cal Newport?

  twice, what do we see here? We have coding agents, we have math agents, like the type that worked on the problems that OpenAI recently announced, we have computer use agents such as Claude Co-Work. And in this universe of LLM-powered agents, there's one particular type that I've labeled on here, open-ended LLM-powered agents. That is where all of the problems are happening when it comes to these rogue AI incidents. All right, so if we imagine, we had to zoom in there multiple times, that's three big zooms to smaller and smaller regions of the AI landscape, before we get to the very specific type of

• 4:14Cal Newport?

  system where we get things like the Hugging Face attack, or the Anthropic agent filling out illegal police forms. All right, so we aren't dealing with AI in some general sense going rogue, we're dealing with a very specific type of LLM-powered agent that I'm going to call an open-ended LLM-powered agent. To go forward here, I want to give you a brief primer, which I've talked about before, but I want to return to here, about LLM-powered agents, before I talk about what specifically makes an LLM-powered agent open-ended. All right, so for those who don't remember, when you hear about an LLM-powered agent, it's easy to conjure some sort of notion of a complicated program that is sort of approximating

• 4:59Cal Newport?

  the human mind, that's constantly learning and making decisions and morphing maybe and evolving and swarming and doing all sorts of scary things. But actually, at the core of LLM-powered agents is a computer program. So I'm going to draw this on the screen here for those who are watching. So we have a computer program, I'll put it over here, I'll put P in the middle of it to represent program. This is a normal sequential computer program written by a human in a normal programming language, is completely understandable by humans. In fact, a lot of agents, the program itself is really not that interesting. All right, so what does this program do? Well, it has access over a network connection to a large LLM. So I'll label this big green blob over here an LLM. Okay, so there's an

• 5:44Cal Newport?

  LLM over in some data center, even that's not even that accurate. There's not like a machine that hosts an LLM. You have many racks of GPUs where each rack of GPU is perhaps virtually simulating some subset of the weights in the LLM, and different queries will go to different GPUs to have the same weights. There's no actual system where the LLM lives, there's just a bunch of copies of the weights spread out over a bunch of GPUs. Okay, so what the agent program does is the following, it will send a prompt to the LLM, I'll draw an arrow going from the program to the LLM. That prompt, this is just in text, will in essence say, I'm an agent, here's my goal, here's the tools, like the software tools

• 6:27Cal Newport?

  I have available, the tools I can call, and here's my current status, what should I do next? So it submits that to the LLM, just like a chatbot user would type in a prompt to send to a chatbot. The LLM, as LLMs do, generates a response, sends it back. Like, this is what I think you should do next, given your goal, status, and current tools. The program then executes whatever it is that the LLM suggested, because remember, it gave it constraints. Here's the tools I have that I'm able to call. And so the LLM response might say like, great, you should move this file from this directory to that directory using the mv command that you told me you have access to, right? So then the program

• 7:09Cal Newport?

  actually executes whatever step the LLM told it. Then, we loop, and we do that again and again and again, right? So this is why I sometimes call LLM-powered agents prompt loops, because I think that gets to the core of what's really going on here. It's a program that repeatedly prompts an LLM, says what should I do next, does that step, repeats. All right, here's my new status, I'm a, you know, I'm a program, here's my tools, here's my goal, here's my status, what should I do next? Why does it have to repeat itself so much? Well, remember, LLMs have no memory, nothing adjusts, they're static. So every prompt that goes to that LLM has to explain everything the LLM needs to know to suggest the next action to the program. That's what an LLM-powered agent actually does. Now, there

• 7:54Cal Newport?

  is an issue with these that people have known since the beginning of messing around with them, which is using an LLM as the sole driver of your actions as a computer program. So I will do whatever an LLM tells me to do. This is risky business. And anyone who's used a chatbot knows why. It's because LLM outputs are engineered to be plausible, right? So they're always a plausible response to whatever input you give it. They understand well, they're tuned to try to produce a plausible output. I've explained these mechanics before, it goes all the way down to their pre-training, which is about guessing tokens from real text. And if you do that autoregressively for many tokens in a row, what you get is a plausible extension of the original input prompt. So they give you plausible answers.

• 8:39Cal Newport?

  Whatever it spits out is a reasonable extension of whatever original prompt you gave it. But these answers are not normative. Normative meaning subcoming or obeying the sort of like natural norms that a human would think makes sense when you're trying to give an answer, right? So they don't necessarily have human morality, they don't necessarily have notions of truth or falseness, they don't have a notion of maybe it's bad to make up a citation in a legal case like that's super illegal. There's no notion of that. There's no world model, or moral core, or policy actor that's trying to check its answers to see if it passes muster. It's just giving you a plausible answer. That could be tricky when you're changing these responses into actions because what's

• 9:24Cal Newport?

  going to happen is they might start suggesting actions which are completely reasonable actions that will work and will make things happen, but are non-normative. They might be unexpected, they might be unpredictable, they might be amoral, they might be illegal. Anyone who's used a chatbot enough knows this, right? You chat enough with a chatbot, they occasionally lock in on weird, dark directions or unexpected directions, or you have to go back and say, "No, no, let's try this again." When you're an agent, the problem is, there's no human, you're not a human. The agent's just a program that's blindly executing what the LLM says. You can get into weird places if you run this prompt loop experiment of, "Let me blindly execute whatever output an LLM gives." And yet, when I was showing you the landscape

• 10:08Cal Newport?

  of LLM agents, I said, "There's multiple types of agents that exist that don't cause much trouble." So, we had, for example, OpenAI's Astra-based math agents that were involved in the recent math slop drop of those 322 somewhat inscrutable lean proofs. Clearly, the LLM involved with these agents is very large and very scaled. They trained it on math extensively, and it was able to do very advanced mathematics. So by, you know, some notion of "smart" is a very smart agent. They also ran these things for a very long amount of time, you know, hours at a time, so thousands of prompts trying to explore these problems. They never went rogue. OpenAI is not saying, "Good news, bad news. Good

• 10:50Cal Newport?

  news, we got a, you know, a finite time blow up of the Navier-Stokes equations. Bad news is, on the way, the agent also, you know, ordered a bunch of hookers to the OpenAI headquarters." Right? That then, it just worked fine. Same thing with coding agents. LLM-powered coding agents are widely used. Millions of software developers use these every day. Many of them are connected to some of the most advanced LLMs we have, and they run many autonomous steps. And we also don't hear a lot about coding agents being used by professional software developers going rogue and trying to hack into other people's servers, or doing things that are well outside of what the programmers thought that they might do. We even have data about this. I'll bring it up on the screen. OpenAI in March actually

• 11:34Cal Newport?

  released a study called "How We Monitor Internal Coding Agents for Misalignment." So remember, agents prompt an LLM for all of their actions. So if you want to see what an agent is doing, you can just look at all the LLM prompts. In this study, OpenAI looked at tens of millions of prompts to LLMs coming from agent programs. And what they found is, the types of rogue behavior they most fear, they call high severity incidents, like sandbagging, sabotage, scheming, the stuff that made us nervous about hugging face, didn't occur at all. So coding agents are not having this problem either. I want to take a quick break from this conversation to cool down and to talk about some of the

• 12:19Cal Newport?

  sponsors that makes this show possible. You all know that I love working with Cozy Earth because I'm a giant fan of their products. Now, recently, I have been using, and when I say "I," I mean my family, because they keep stealing it from me, the Cozy Earth Bubble Cuddle Blanket, which is a perfect combination with their famous bamboo sheet sets, which we're obsessed about and have multiple pairs of and travel with, because we like them so much. The Bubble Cuddle Blanket, it's soft, it's heavy, you really can cuddle in it, and it just gets passed around my family, especially now as it's getting colder in

• 12:59Cal Newport?

  the mornings, as fall advances. There's always a competition for who gets the Bubble Cuddle Blanket. It's the perfect addition to their already fantastic bamboo sheet sets. So consider picking up a Cozy Earth Bubble Cuddle Blanket, and for sure, you need some of their bamboo sheet sets. Get them today. Look, there's no risk in giving this a go, because they have a 100-night trial and a 10-year warranty. If you don't love them, you can send them back, but you won't, because they're great. So go to cozyearth.com and use code DEEP for up to 20% off home and sleepware. That's cozyearth.com code DEEP. And when they ask you where you heard about them at checkout, let them know you heard about Cozy Earth

• 13:43Cal Newport?

  right here, the Deep Questions Podcast. I also want to talk about our friends at Gusto. Look, no one starts a small business dreaming of all the payroll tax forms they're going to get to file, right? That's the type of thing that you procrastinate on, because it's boring and it gets in the way of actually growing. This is where you need Gusto to handle those type of administrative tasks in the background, so you can focus on what really matters. Gusto is an online payroll and benefit software tool built for small businesses. It's all-in-one, remote-friendly, and incredibly easy to use, so you can pay, hire, onboard, and support your team from

• 14:24Cal Newport?

  anywhere. You get unlimited payroll runs for one monthly price, no hidden fees, no surprises. You can also save time with built-in automated tools, like offer letters, onboarding docs, direct deposit, and more. It's ranked number one on G2's highest satisfaction products list for 2026, and trusted by over 500,000 small businesses, including the ad agency we work with here at the podcast. So what are you waiting for? Get started and try Gusto today at gusto.com/deep, and you'll get three months free when you run your first payroll. That's three months of free payroll at gusto.com/deep. One more time, that's gusto.com/deep. All right, let's get back to the show. So how do those type of LLM agents avoid trouble, where other types of LLM-powered

• 15:09Cal Newport?

  agents, like the ones we heard about this summer, don't, that they do cause trouble? Well, there's three things that keep the, I call these targeted LLM-powered agents, like coding and math agents. There's three things that keeps them on the reservation. One is narrow goals. They're given a pretty specific goal. Like, if you're a math agent, it is, "I want you to, you know, write this part of a proof in LaTeX," which is like a markup formatting mathematicians use to write up math papers. You're writing up a proof in this format into this file. It's a narrow goal. Turns out, when you give LLMs, say, "Give me a step to make progress towards this narrow goal," they're trying to give you steps that help you make progress towards that narrow goal. They're not going to go

• 15:52Cal Newport?

  off and do something rogue necessarily. The second check that helps keep these targeted LLMs from going rogue is they have a restricted toolset. So the tools the agent program has available to actually take actions in the real world is appropriately restricted. I mean, if you're a math, if you're the math agent, maybe you have the ability to do searching on archives of math papers. I'm not quite sure if that's all trained in, or if they could also do real-time searching, contextually on relevant papers. And you have the ability to read and write files. You can actually write proof text into a file within a limited directory structure. So you have limited toolsets, there's, you can't do much damage. They also have oversight. So this is particularly true of the coding agents. If you did write

• 16:36Cal Newport?

  a very simple coding agent, gave it tools for the terminal, and just said, "Go on and write this code," they will get in trouble. Not in the rogue sense, but just they'll do unexpected stuff. They'll overwrite files, they'll delete files they shouldn't have. And so, a lot of effort has gone into the agent programs for coding agents. Think of it as hardcoded oversight to automatically notice and avoid the most common types of issues that a coding agent might accidentally do. We know this in part because the code, the source code for Anthropic, rather, Anthropic's agent, Claude Code, accidentally got released.

• 17:17Cal Newport?

  It's 500,000 lines of code in the program that prompts the LLM, and a lot of it is just hardcoded special cases and checks. They have pattern matches in there. This is just through experience. The programmers who worked on it are programmers, so they kind of know the types of things to look out for. So there's all sorts of hardcoded special cases in that program. Hey, if it says, you know, rm-rf, which, you know, means delete a bunch of files without asking, they've hardcoded something to say, "Eh, maybe don't execute that. We'll put that one aside." They actually have pattern match. It's interesting, they have pattern matching in the Claude Code harness to actually look for common terms from whatever the user is typing in that indicate frustration, like cursing. And if

• 18:02Cal Newport?

  it since, if it sees there's frustration words in the user's prompt, the program will assume, "I don't know what the LLM has been telling me to do, but it's probably pissing off the user. So why don't we stop now and try to check in with the user before we take any more steps?" There's also a lot of just hardcoded checkpoints in the coding agents, you know, for simple programs, like amateur people making interfaces for their clubs that they're a part of, that they vibe code, that's fine. But for professional programmers, they're checking in way more often. They have recursive, they have testing going on. They want to check in, "All right, tell me what you did here. I need to check this. I need to run test on it." So it's a lot more interactive than you would think. So narrow goals, restricted toolset, and hardcoded oversight can make for the right task, LLM-powered

• 18:47Cal Newport?

  agents never go rogue. So why are we having rogue agents still? They all come from what I've been calling open-ended LLM-powered agents. And why do they have trouble where these targeted LLM agents do not? First of all, they replace narrow goals with broad goals. Look at the hugging face attack, right? Like, they're given a lot of latitude. Like, here's your goal. But you have sort of complete latitude in how you make this happen because that was the nature of many of the agents involved in the hugging face attack were being used, not all of them, but a lot of them were being used as part of a cybersecurity benchmark

• 19:27Cal Newport?

  suite known of Exploit Gym where the whole idea is to be really creative and find whatever way you can to try to accomplish the goal. Typically, it's like breaking into a test system, right? So they have broad goals. They have a broader range of possible actions. They're not being constrained. They're given a powerful toolset, right? They're given tools, these powerful tools you need to win at hacking challenges. So now they have very powerful software. In particular, they have access to these package services where you can download special packages to help you do particular types of attacks. So they can, it's like, I'm trying to think of a good analogy here. The good analogy would be if you have in targeted agents, you're

• 20:11Cal Newport?

  giving two or three tools to the agent from your toolbox. These open-ended agents, you give them a credit card and access to Ace Hardware. They can just go get whatever they need to do. You can get into more trouble that way. And finally, they have very limited oversight. First of all, because these are tests being done rapidly, unlike a coding agent where you have, you know, a year or two of people painstakingly hard coding in special cases, these, they just rock and roll, right? They're just like, let's just see what happens. And on purpose, if you look closely like at the Redwood or Muta report on the hugging face attacks, the engineers just let these things run. They weren't really trying to check in. They had very little monitoring going on. You very easily, there's

• 20:52Cal Newport?

  all sorts of checks you could have easily put into the agent program itself. Like, every time I'm trying to, you know, access a new domain, let's just stop and have a human look at it. Let's have a human check every hour and look through the log of all the prompts to the L L Ms. Let's even just have another L L M summarize that. And like, let's just check every hour to make sure that, that we don't have something unusual going on. They almost like purposefully didn't want to have oversight. Not only they not want to have oversight, but as the reports make it clear, they would see that something really troubling was happening and they would just restart and let it run again and come back a few more days. They had reports from previous experiments because OpenAI, their new misalignment reporting framework has them say when things were detected and when

• 21:37Cal Newport?

  they actually occurred. They had detected and have reports of, hey, these agents are doing unexpected unpredictable activity outside of the sandbox. They had those reports already before they started new experiments with similar agents that then went on to do the same sort of thing. So it was almost like purposeful lack of oversight. So if you have really broad goals, you say you, you do what you need to do to solve this problem. We have basically no constraints on it. You give access to a wide, diverse and powerful set of tools and you have very little oversight. Those type of L empowered agents are going to cause problems. Why are they going to cause problems? Is it because somehow this static L L M distributed and copied over thousands of GPUs is actually some sort of coherent sentient entity? No, it's because L L Ms are plausible but not normative. And

• 22:21Cal Newport?

  if you're just going to keep asking an L L M again and again, what should I do next? What should I do next? They're going to give you weird responses. Eventually, they're going to lock in the past you wouldn't expect. They're going to be non-normative. They might do things that are harmful or illegal. And if you give them broad goals, powerful tools and oversight, there's nothing stopping that from unfolding in ways that's going to be unpredictable or harmful. The key point I want to make about this is there's nothing inevitable about the AI industry needing to design and deploy open-ended L L M powered agents. It would be completely reasonable, for example, in this current moment for the industry to say, that's a

• 23:03Cal Newport?

  bad idea, but you know what? Targeted L M powered agents, we're finding use cases for it. We're getting better at these coding agents. We're figuring out math agents. They're a little inscrutable right now, but we're, they're powerful. So clearly, we're going to be, give me a year or two, we can get a great tool for mathematicians. There's got to be 100 other sectors where we can build targeted L empowered agents that are, you know, they might sometimes do stuff you don't expect, but aren't rogue and are trustworthy and generally do good stuff for you. And you could build out a really wide industry, just like the original software industry of lots of different tools, each of which tuned to a different task or industry and each of them bringing in revenue. And you could build a really solid, very like lucrative technology industry on top of that, right?

• 23:45Cal Newport?

  There's nothing wrong with that. You could also then say, hey, you know, these targeted L empowered agents, this is maybe not the most effective way to do it. Actually, what we really want to do for a lot of these tasks is neuro-symbolic systems. To borrow a term from Gary Marcus, a lot of the original AI landscape I showed you where L L Ms were just one subset of it, a lot of those systems, like the systems that play games very well, the systems that drive cars automatically, AlphaFold, these are neuro-symbolic systems. It's not a program just prompting an inscrutable L L M and just doing what it says. They have some neural networks in there that are trained with machine learning. They have some symbolic components in there. These are programs written by humans. They're understandable by humans

• 24:26Cal Newport?

  and they all work together. Turns out if you do neuro-symbolic systems, you don't need a 5 trillion parameter L L M that has to use up all the water of Idaho just to run. You can actually fit them all on a single chip I can run in my basement. And so there's this whole exciting trajectory we could have of targeted, targeted use cases where we could use targeted L M based agents and neuro-symbolic agents to bring the price down and the effectiveness up even more. And boom, it's a massive industry. It makes all of our lives better. There's no reason, right? We don't need open-ended L empowered agents for AI to be useful to people. And yet, we see an insistence on building and testing them. So why is this? Well, first of all, I think we have to clarify, it's

• 25:11Cal Newport?

  not everybody building open-ended L empowered agents. The two companies in particular that are really pushing this research hard is OpenAI and Anthropic. Now, to be clear, this doesn't mean that no other hyperscaler style AI lab is, is working on these. We know other companies have had some issues with these. Meta and Google in particular has had a little bit of issues, but they're kind of, I don't know, I feel like the, the cases we've had of rogue AI from like Google, for example, I'm not super worried about the, they, they were doing some open-ended agent testing, but they were more constrained for cybersecurity. They sent it off to this company called a regular to run the test and a regular

• 25:55Cal Newport?

  set up the sandbox wrong and gave it full access to the internet. And it did, it, it tried to guess some passwords or something, right? And I think they're like, okay, whoops, you guys did wrong. Let's not do this. Meta had a similar issue and then they kind of backed off it a little bit, but OpenAI and Anthropic are pedal to the metal. This is where like the big, especially OpenAI, this is where like the big case after case of, of months of actual ongoing hacking, like they are doing this at a level that is in my, I don't know how the numbers, but seems like 10x what anyone else is doing. So those two companies in particular are super insistent on open-ended L empowered agents. Come hell or high water, we got to keep running those things. Why is that? Well, if we look closer,

• 26:40Cal Newport?

  OpenAI and Anthropic do not like that alternative future that I just gave to you. They do not like the idea of a more normal AI technology landscape where you have a huge number of AI, task-specific AI systems, like coding agents, like the math agents, just multiplying that across 100 other different types of task areas. And you have all these systems, a lot of them L empowered and targeted, some of them neuro-symbolic powered. And there's just a lot of really useful AI tools in the world. They do not like that vision. And there's three reasons why. Vision number, reason number one, why those two labs in particular don't like those is competition, right? That vision, when you get task-specific

• 27:24Cal Newport?

  to targeted L L Ms, does not require that you have the biggest baddest L L M on the block. We see this with coding agents. I mean, this has already become a problem for OpenAI and in particular Anthropic that makes a lot of money from coding agents, is that you can use smaller L L Ms, you can use cheaper L L Ms, you can route to different L L Ms based on the complexity of the coding tasks that you're doing. My prediction about the, the coding, the targeted L M based agent, coding agent space is that we're going to see a new generation of very specialized, pretty cheap L L Ms. And I think there could be L L Ms that are going to be able to run on a rack you can put in your own server room at your own office that have been super tuned for particular languages. I'm a big

• 28:09Cal Newport?

  believer, talking to a lot of programmers, that the next evolution of coding agents is actually going to be the emergence of a sort of intermediate pseudocode language that is forgiving. It's not syntactically demanding, but it's pretty accurate. And a programmer can write really quickly in the pseudocode and get much more exact translation from that in the code, thereby significantly reducing the amount of checking and verification that's required right now where the code production is a little bit more scattershot and inscrutable. This is going to be able to run on much smaller L L Ms. All of that's bad. If you're in the business of building giant L L Ms, right? So you got to understand, Anthropic and OpenAI only make money when you need their newest, latest, greatest L

• 28:51Cal Newport?

  L M to make progress. If you, if a cheaper L M is enough, you're not going to them. They're Ferrari, right? And these cheaper L L Ms are Honda. And they want to keep people wanting to buy Ferraris even though they're sort of ludicrously expensive. So they don't like that vision. They hate the idea of neuro-symbolic systems because, yeah, those might have neural, they have neural networks in them, but they're small. Man, you look at these superhuman game playing engines. They can play games better than humans. These things use small neural networks compared to what's going on over at OpenAI and Anthropic. I mean, you can run, you know, Dreamer V3 fits in the memory of a, a single GPU. You can run Stockfish, the chess engine that can play chess better than a human. You can run that

• 29:33Cal Newport?

  on a laptop. That's scary to OpenAI and Anthropic. So that's one reason why they don't like the vision I gave where you don't need open-ended L M agents. Reason number two is they've been convincing the military in particular that open-ended L L M powered hacking agents, sure, they might be unpredictable. Sure, they might break into like places you don't want, but weapons are dangerous and this is a weapon we need. So they have this sort of military industrial complex type argument that open-ended L empowered hacking agents in particular, so this particular type of open-ended agent has to be in our nation's arsenal. And so that's another reason why they think we have to have these. We need

• 30:15Cal Newport?

  things that can run for days at a time and run these sort of complex hacks. I think that's highly suspect. I think Centaur hacking and defense where you have highly trained cybersecurity experts working alongside models and much shorter horizon agents that have been trained on cybersecurity is much not only safer but much more creative and powerful and effective than just letting an agent run by itself for days at a time. I don't buy this idea that if you don't build this highly unpredictable combustible long-term agent yourself, that if someone else does, you're somehow less prepared to defend against it, which I don't think is true. But that's one of the reasons why, another big reason they give is, we have this particular cybersecurity market, the government spends a lot of money, the military has the most secure budget of anything, we want in on

• 31:00Cal Newport?

  that. All right. Reason number three, why they don't like this vision, and this is probably the most foundational, is that OpenAI and Anthropic, and particularly Anthropic, but OpenAI and Anthropic, believe that you need open-ended L-empowered agents to realize their vision of superintelligence. Remember, those two labs, in that this is different than almost any other worldwide or in America, companies working on hyperscaled LLMs. Those two labs in particular, were explicitly founded with the mission of trying to usher in superintelligence safely. Their, their vision is, superintelligence will either be a vengeful god that kills us all, or it will be a benevolent god which will give

• 31:43Cal Newport?

  us a transhumanist utopia. And they feel like their job is to make sure that we get the good option. And remember, we've talked about this in recent weeks, the people that are yelling, the Jacob Croxtons of the world, all of these people funded by effective altruist money, who are yelling and influencing congresspeople about, we need to, we need to regulate to prevent, you know, superintelligence, they're all from the same family. They, they only differ in the fact that Dario Amade sort of thinks we're ready to do it, and other people in his family are thinking he's not ready to do it yet. And actually, even, it's more darker than that, because actually, the, the more you push at this, the more you see, a lot of these EA-funded voices that are pushing for this type of regulation, they're actually on the same team as Amade, because what is this regulation going to

• 32:24Cal Newport?

  deliver, if you really look at it? A regulatory, regulatory capture landscape in which essentially, only Papa Amade gets to build the big LLMs and the open-ended L-empowered agents. Him working with third-party auditors, which are other firms like Meter, that are founded by the same people, funded by the same people that funded Anthropic, they all know each other, they'll come in and help monitor them. And so basically, it's, it's all sort of an effort in some sense to be like, yeah, we're trying to bring in the benevolent AI god to get our transhumanist future, so we need to stop other people from messing around with this, let Papa Amade do his thing. And billions of these sort of biological bootloaders

• 33:06Cal Newport?

  might die, but we need to get, you know, we need to get our consciousness uploaded in the digital beams that are going throughout the light cone. So, they really feel like you can't get recursive self-improvement, which is going to deliver superintelligence, unless you keep pushing aggressively forward with open-ended LLM agents, because long horizon autonomous activity is what you're going to need, to have their L-empowered scientists that can do their own automated research and push AI forward first, so they get to the superintelligence first. They get to it first, they control the god, we have a chance of transhumanism utopia. If they don't, then maybe someone else will do it, or we'll lose the chance for the utopia altogether. So there's three reasons why

• 33:46Cal Newport?

  they don't like a future where you say, open L-empowered agents don't work, they're dangerous. It's a small part of the whole AI landscape, stop doing that. We can have a huge thriving, highly lucrative, useful AI industry, but they don't care about that. All right. So what should we do, the rest of us, the normal people? Not the, because remember, what is this argument? Right now, all of this debate is between billionaires. But, we have the, the billionaires funding the effective altruists, who are also funding the labs. So you have billionaires like Dario Amade, fighting with other billionaires that are from these effective altruist organizations, trying to figure out together, how they're going to usher in

• 34:28Cal Newport?

  this utopia. And then you have the accelerationists, which are billionaire venture capitalists that are arguing with them. And so it's billionaires fighting with billionaires, fighting with billionaires. It's like 100 people, they're all really rich. What about all the rest of us? We are very happy with a future, like I just described. Most people would say, yeah, do that first thing, where you don't do this one type of experiment that's unpredictable and harmful. But you do all these other things where you're building really useful AI tools that industry after industry, task after task, makes our life easier, and creates a lot of jobs, and we, it explodes the sort of tech industry to be even bigger than it is now. That sounds great. That's what the regular people probably want. It's the billionaires who are like, forget that, we're trying to, for the accelerationists, either get another

• 35:10Cal Newport?

  billion, or for the EA, Dario Amade types, trying to get to the utopia. So we're just letting billionaires argue with billionaires about who gets to control the world, while everyone else just has to sit here, be stressed out and say, excuse me, sir, but none of this helps out. What, what are you doing? You're not even trying to make a product, you're just trying to, it's just like Dungeons and Dragons, like what are you guys doing? You guys, it's just like Twilight Imperia made real, you LARPing? Like what's going on here? So I think the regular people need to actually push back. And here's how you do it. There's two things I want to suggest. Number one, vocabulary. Refuse to play the game of using the word AI, when what we're really talking about is open-ended L-empowered

• 35:54Cal Newport?

  agents. Be specific. It puts the pressure back on the companies. When the companies say, AI is getting too powerful, AI is getting out of control. You imagine it's like, as, as, you know, one of my producers, Nate, said, it's like a Pokemon that's evolving into bigger and more dangerous configurations as the game unfolds. And all you can do is try to slow down the rate of this one beast from growing. That's not what's happening. It's a, there's thousands of things we do with AI that are really useful. It's this one thing that's a problem right now. So use the right vocabulary. Don't say, I'm worried about AI going rogue. I see that in the opening paragraph, and if you look in major newspapers,

• 36:35Cal Newport?

  every article of any of these attacks, always first paragraph, AI going rogue. No, no, no, no. Open-ended L-empowered agents, once again, have shown that they are unpredictable and can cause harm. The more specific we are with that vocabulary, the more pressure it puts on those two companies to say, why are you doing that experiment? So I think vocabulary really matters. Number two, when we're advocating for regulation or control, the goal for regulation, the average person has to speak up and say, the goal here is to maximize useful products from AI while minimizing harm. That is what you typically want out of market regulation. Maximizing utility to the public, while minimizing

• 37:20Cal Newport?

  harm. That's what you want from market regulations. The regulation debate that's happening on the Hill now, which is powered by the, the effective altruist billionaires, and their sort of silent partners in the billionaire founders of the AI companies, the type of regulation they're talking about is not about maximizing utility for the everyday person. And it's not necessarily about minimizing all the harms you would want. It's about consolidating control of LLM progress, so that Papa Amade can be the successful shepherd to bring us to a transhumanist utopia. So if all of these EA-sponsored bills get passed, these superintelligent bills, like you would see Senator Sanders standing next to Nate

• 38:03Cal Newport?

  Sorensen proposing, if these things get passed, it's just going to lead to a world in which we consolidate most of the L-empowered agent work to basically Anthropic, OpenAI is kind of a mess, but maybe also kind of OpenAI, but basically Anthropic, and them, with the help of their third-party monitors, which are friends of theirs funded by the same people, will then be able to do whatever they want to do with these open-ended L-empowered agents. And when they go off and do even worse things, as they give them more powerful tools and more runway and more long horizons, they all just keep saying, AI is unpredictable, we're doing our best to keep it under control. So the regulation the billionaires are talking about, does not maximize any utility. The utility they're maximizing, is increasing

• 38:45Cal Newport?

  the probability of getting to a transhumanist utopia. That's not what the public wants. We didn't ask for that. And they're not directly minimizing the harms you get by running open-ended L-empowered agents. So we need to demand as the people, not the billionaires, but the people, we need to demand actual market regulation. Create a market that creates useful tools. These hacking monsters aren't useful to us. This is, it's a fever dream of, of the, of Dario Amade envisioning his trans, transhumanist utopia. Running these type of open and L-empowered agents is harmful to us. So why don't we stop that until you can show us you have a more controlled way of doing that? And let's focus on what's going to get us a lot of

• 39:25Cal Newport?

  competition with targeted L-empowered agents, neuro-symbiotic agents, task-specific agents that are completely controllable, completely understandable, and very useful. Okay? So this has become my new call for people. I really think this is, what this is really about is billionaires versus everyone else. We got to put it that way. The billionaires want to be like, oh, we're on your side, we're trying to prevent, you know, um, Ter, remember Terminator? You don't want that. The, the robot stepped on the skulls. Remember that? The robot in Terminator stepped on the skulls. We don't want that. We've got your back. And we need to be like, yeah, we saw that movie. Stan Winston made that. That was a, that was an animatronic. We want useful products, we want you to stop building harmful things.

• 40:09Cal Newport?

  So there's a analogy I want to give to kind of end with, right? Because I want to summarize this all together. I've summarized all this together. What I'm really saying is, AI is a very large

• 40:18Cal Newport?

  A very large topic, and there's lots of different types of systems, many of them very useful. And you got to zoom in three times in that landscape to get to the one type of system that's really causing a lot of these AI goes rogue concerns. And there's no real major reason to be running those type of systems, except for these really unusual, eccentric, and self-interested concerns of the billionaire utopians. It's not inevitable that AI, as it gets more powerful, goes rogue. As AI got better at chess, it didn't go more rogue. As it learned how to predict folding protein patterns did not lead to, you know, the design of rogue, it didn't go rogue and try to build its own designer organisms.

• 41:02Cal Newport?

  AI getting more powerful does not mean it has to go rogue. It's this specific type of system. So here's the analogy I want to leave you with. Maybe we'll make this kind of clear. Imagine that Elon Musk released rocket-powered roller skates. I'm using Musk because like, it feels like something he would release. Rocket-powered roller skates, and a lot of people get hurt because you're putting rockets on your feet and they're crashing into walls and breaking their bones or whatever. And there's kind of like an uproar about this. Like, a lot of people are getting hurt using rocket-powered roller skates. Now imagine then Elon Musk came out and said, I have grave concerns. I believe we have to pace the frontier of wheel transportation. Because look at all this, people are breaking

• 41:46Cal Newport?

  their bones, it's only going to get worse. We would say, that's ridiculous. The car companies would say, what are you talking about? The bicycle companies would say, what are you talking about? The scooter people, the skateboard people, they're like, our wheel transportations aren't breaking any one bones. They've worked just fine. We have safety regulations. They're very useful. People really love them. Maybe you should stop building rocket-powered roller skates. That's really what's happening here. And if Elon Musk came back and said, yeah, yeah, yeah, but you don't understand, we've spent the last 20 years on blogs talking about how rocket-powered roller skates are part of this prophecy that leads to, it's going to lead to a utopia. We would say, okay, Mr. Musk, like, we have this nice jacket for you to put on. And there's some straps

• 42:28Cal Newport?

  in the back. Don't worry about that. This room has pads. It's very nice. You're very, you're very cool. Rock, rock, rock. Right? That's what we would do. But that's essentially what's happening right now. These small number of companies are producing the AI equivalent of rocket-powered roller skates. A lot of people are breaking their legs, and they're saying, we have to slow down transportation. In fact, not only that, but let's let Dario Amade be in charge of all of transportation, because like, we got to be careful about. The person, you know, the person who built the rocket-powered roller skates, let's put them in charge of all of it, because like, we can't trust anyone else. I'm kind of straining this analogy a little bit, but hopefully that makes a little bit of sense. So anyways, I don't mean to get fired up. I do think this has become, it's a populist argument. I mean, it's people versus the billionaires. And the people got to stand up. We're not

• 43:11Cal Newport?

  going to use your newspeak. We're not going to call everything AI. You got to tell us what systems have problems. And we're not going to accept calls for regulation that don't make our life better. Useful products minimize harms. Not, you can't regulate sci-fi stories. Market regulation needs to create more useful products, minimize harms. All right. So this is why I do not think AI going rogue is inevitable. You can create AI that goes rogue, and it's a problem. But we need to look and say, stop it. We need to stop just accepting whatever these small groups of really well-funded people are telling us and say, we've got eyes. Those roller skates are stupid. I'm going to go get my bike.

• 43:53Cal Newport?

  All right. That's enough of that. That's my rant. That's my rant for today. I don't know, Jesse. There's a lot going on. It's a complicated topic. It's a complicated topic. There's so many different things involved. There's complicated technology. There's politics. There's ideologies. The whole thing is really complicated. So I do my best.

• 44:12Speaker 01

  The graphic helps put in perspective.

• 44:14Cal Newport?

  Yeah, you got to zoom in a lot. Like, if DC, the city of DC where we are right now, if that is the AI landscape, these open-ended, LL powered agents that are causing all the rogue problems is like downtown Tacoma Park. The whole rest of the city is other types of AI.

• 44:29Speaker 01

  I love the Papa Amade reference.

• 44:32Cal Newport?

  That's really, that's all of this regulation, if you look at it closely, it all ends up with like Papa Amade and his happy elves at Meter in Redwood, who is friends who are funded by the same people. They'll, they'll take over. Don't worry about it. They'll make sure everything goes okay. I just think it's crazy. Billionaires regulating the other billionaires, deciding which of the billionaires gets to be in charge of this technology, and they all have this weird ideology about utopia. So it's like, this is not a great situation.

• 45:00Speaker 01

  So Amade and Altman, are they still friends, or they don't?

• 45:03Cal Newport?

  They don't like each other.

• 45:04Speaker 01

  They used to work together, right?

• 45:06Cal Newport?

  Yeah. Well, Amade, Amade left OpenAI because he felt that, my theory on Altman, I've got these theories. My theory on Altman is kind of based on Ronan Farrow's and Andrew Martin's reporting for The New Yorker, is, he's not a great executive.

• 45:20Speaker 01

  Yeah, I read that too.

• 45:21Cal Newport?

  Right? Like, he wasn't, this is not, you know, someone who started a startup from scratch and made into like, a big concern or whatever. He sort of was just like, in these circles, and was made the president of Y Combinator, which is an incubator. Like, that's not, you know, they basically just take shares of kids' companies, and some of those companies go big, and they cash the checks. And Elon Musk was just like, hey, I'm going to start a nonprofit to make sure that, like, we don't get bad, super intelligent AI. Do you want to be in charge of it? And he's like, all right, man, cool. Like, that's, so, he's a little bit more slippery. Amade is, I think, a much more, I don't know if he's a good executive. I've heard that he's very isolated within Anthropic, so I don't know what degree he's executing over there. But he's a much better technical mind. He's

• 46:03Cal Newport?

  from a physics background. He's a very smart guy. And Anthropic is actually, they're making, I mean, both OpenAI and Anthropic have made progress on these models. And Anthropic has probably had a higher density of high-quality technical breakthroughs. So I think they're running a tighter ship over there as a technical company. But they left, Amade left OpenAI because Altman was being insufficiently committed to the billionaire vision of transhumanist utopia, like, shepherding in the right AI and preventing the others. They're like, you're not doing this right. We'll do it better.

• 46:37Speaker 01

  And Amade's sister's at Anthropic too, right?

• 46:40Cal Newport?

  Yeah, they left together. Danielle and Dario left, both left to start, yeah. And she's married to a very prominent, effective altruist. I mean, he came out at some point and said, I'm not an effective altruist. He's, he's, his entire life has been an effective altruist. It's crazy. It's great. I mean, it's like Ronald Reagan in his 60s being like, I'm not a Republican anymore. Like, you're, you're a Republican, right? Like, you could say it. Your investors don't like it, but like, you're a Republican. I mean, the way, like, people don't understand, like, the percentage estimations of doom, that's straight, that's just like, the shibboleth of effective altruism. That's how they talk in the Bay Area. That's how you know you're talking to effective altruists. It's like, my P doom is 23%. And Amade has been like, giving these precise numbers all along. That's why Jacob

• 47:23Cal Newport?

  Crockston talks that way. It's, you know, it's a, it's a complicated world, but it's also kind of a simple world. So it's like, we have a double whammy. Americans get worried when you have billionaire oligarchs trying to control things that matter. But usually with the billionaire oligarchs, the main issue is, they just want to make a lot of money, and they're going to be exploitative of markets to try to maximize their wealth. You add in an eccentric ideology to the billionaire oligarchs. It's not good. Now, we've had that before. If we go back in like, the late 19th, early 20th century, they wouldn't be billionaires back then. They'd be, you know, like, centimillionaires. But you had, you know, you had very powerful people with lots of money consolidating industries

• 48:07Cal Newport?

  who had like, a lot of thoughts about like, foreign policy, you know? And it's like, okay, we're going to start wars, and we've, you know, whatever, right? We've had this before, where you have concentration of economic power confined with personal ideologies that are very strong and maybe eccentric. And those two things together is, always causes trouble. Always causes trouble. And so it's not that surprising what we're seeing, except for it's camouflaged well. And people don't realize, all these players are funded by the same billionaires. They all, it's all the same tech billionaires are funding all these players. They're all from the same playbook. They all have these guaranteed jobs at these giant think tanks that they're funded. They can just bounce back and forth between. They don't know that you'll say, like, the prominent, like, why did this prominent

• 48:50Cal Newport?

  senator, I won't name names, why did this prominent senator, like, suddenly is straight out of like, EA rationalist talking points? His top aide left, went to one of these effective altruist institutions, came back a few years later, was like, hey, senator, this is the biggest issue now. And the senator's like, all right, I guess this is what we got to care about, is like, super intelligence. So people are just the pawns of billionaires. Billionaires have, this is what they're caring about. Like, we want to make sure that we don't have the bad AI, we have the good AI. And in the good AI, some of us will be able to upload our consciousness eventually to the light cone. And guess who those someones are going to be? It's going to be, it's not going to be you and me, Jesse. We're not the one. We're the biological bootloaders, to use Elon Musk's term, that our whole point

• 49:32Cal Newport?

  is to get to the point where we can get, you know, it could be Dustin Moskowitz's brain, not ours, that's going to go up the space when the time comes. Let's just put it that way.

• 49:42Cal Newport?

  I want to take another quick break to hear from some of the sponsors that makes this show possible. Look, here's a stat that genuinely shocked me. Sales teams spend about 50% of their time on admin work rather than selling, building relationships and closing deals. What if instead, every sales rep could walk into calls briefed, can stay focused instead of taking notes, and update their CRMs the moment the call ends? That's where today's sponsor, Pipe Drive, comes in. It's an intelligent, AI-powered sales CRM, loved by growing sales teams. To save you time, Pipe Drive remembers your deal history and can create a pre-meeting brief that automatically pulls email records

• 50:24Cal Newport?

  and previous conversations into one brief, ready before every call, so you always know exactly where things left off. Their new AI note taker is also a game changer. During your meetings, it records and takes notes, so you don't have to. You stay in the conversation, Pipe Drive handles the rest. So, switch to a CRM built by salespeople for salespeople, and join the over 100,000 companies already using Pipe Drive. My link gets you an exclusive 30 days free. Instead of the usual 14-day trial, no credit card or payment needed. Just head to pipedrive.com/deep to get started. That's pipedrive.com/deep, and you can be up and running in minutes. Let's talk about our friends at Fabric. It took me longer than

• 51:09Cal Newport?

  it should have to get the life insurance I needed to protect my family. Now, what was the problem? Getting life insurance can be a pain, unless you use Fabric by Gerber Life. Fabric by Gerber Life offers term life insurance you can get done today. Made for busy parents like you, all online, on your schedule, right from your couch. You could be covered in under 10 minutes, often with no health exam required. If you've got kids, especially if you're young and healthy, the time to lock in low rates is now. If you have life insurance to work, that's a good start, but it may be less than what your family needs and could disappear if you change jobs. Term life insurance from Fabric stays with you. Flexible policies built to fit your family and your budget, and as long as premiums are paid, coverage

• 51:54Cal Newport?

  will follow you wherever you go. There's no risk. You have a 30-day money-back guarantee, and you can cancel at any time. Plus, free digital wills and tools to help invest in your kid's future. All are available right from your phone. Fabric has partnered with Gerber Life Insurance Company, who is trusted by millions of families for over 50 years, and rated excellent on TrustPilot with over 2,000 5-star reviews. Join the thousands of parents who trust Fabric to help protect their family. Apply today in just minutes at meetfabric.com/deep, meetfabric.com/deep, and use my link so they know I sent you. M-E-E-T fabric.com/deep. Policies issued by Western Southern Life Assurance Company, Cincinnati,

• 52:36Cal Newport?

  Ohio, which operates in DC and all states except New York, and distributed by Gerber Life Agency LLC using Fabric technology. See meetfabric.com for additional disclosures. This is a paid ad. All right, let's get back to the show. All right, enough of that. And obviously, I'm oversimplifying. Everyone's going, "The political people are saying you're oversimplifying it." I am. The technical people are going to say you're oversimplifying agents. I am. But the core concepts I'm trying to get across here, so hopefully that helps. All right, so, on Mondays, we like to also hold some office hours right here, what you are interested in. As always, if you have comments, questions, or reactions, you can send them to [email protected]. I figured, because I was in a bit of an ornery AI mood

• 53:16Cal Newport?

  today, we're going to do questions, just about AI, from our readers. So, we'll see what it is they care about. Hopefully, this will calm me. These will all be calming questions, right? Right, Jesse?

• 53:29Speaker 01

  Dustin Makovits, what's it, Makovits?

• 53:31Cal Newport?

  Muskowitz.

• 53:32Speaker 01

  He's the Facebook guy, right? One of the Facebook guys?

• 53:34Cal Newport?

  He was the early Facebook guy. Got out early, but had-

• 53:37Speaker 01

  His brain's going to space.

• 53:39Cal Newport?

  A lot of EA money. These funded good things, too, kind, you know, but his brain's going to space, not ours. We're the slurry. We'll be used for the slurry that keeps alive the biological bootloaders until we can get the consciousness to space. All right, what's our first question?

• 53:58Speaker 01

  Our first question is about the legal implications of rogue AI.

• 54:03Cal Newport?

  All right, so this came from Aaron, who said, "This is a good succinct logical case for prosecuting AI companies when their agents do things that people would be prosecuted for." All right, so Aaron sent a link to an article, a blog post, that was titled, "Why Aren't AI Companies Being Charged For Hacking?" I'll read you, I just have it here, an early paragraph from the blog post. You have a sense of what it's about. So, early on in the blog post that Aaron sent me, it says, "The big AI corporations are hiding behind a responsibility gap argument due to the emergent behavior of their systems. It's a position tailored to dissuade prosecution as it removes intent, which is typically

• 54:45Cal Newport?

  required for a crime. I understand their systems, though, so I will show how they absolutely intended this behavior, giving a framework for legal evaluation." I like these type of questions. I think the legal questions surrounding liability from open-ended LLM-powered agent attacks, what are the actual legal issues involved? I think these are interesting discussions. Using the sort of moderator's prerogative here, though, instead of going into the details of this blog post, I actually wanted to look at something that a legal expert put out. So, I'm going to bring this up here on the screen. This is the testimony from my colleague at Georgetown, Paul Ohm, who's a professor at the Law School

• 55:27Cal Newport?

  at Georgetown and a world's expert at tech law. He recently testified in the Senate Committee on Homeland Security and Governmental Affairs. The hearing was rogue AI securing the homeland against AI agent attacks. Paul is the real deal. He understands this law inside and out, is completely his own thinker. Half of the people testifying there were EA people, but not Paul. And he really did a good job in his testimony, which is available online, in getting at the legal issues relevant to prosecuting things, people, and companies for agent-powered attacks. I just want to read a representative paragraph here from page nine. All right, so Paul says here, "There is no simple, universal fix for this

• 56:11Cal Newport?

  problem. It would be unwise to simply declare that all actions taken by an AI agent should be imputed as intended by the next deployer or developer up the chain. Instead of seeking a quick fix, we should look at the role of mens rea, Latin term, M-E-N-S-R-E-A, plays in each law and ask the following three questions. One, is the conduct described the kind that ought to be criminalized if committed by an AI agent? Two, if so, shall we assign criminal culpability to a developer, deployer, user, or more than one of the above? And three, can we set a mens rea requirement for that actor at a level other than intent?" This is accurate and more subtle legal thinking. Mens rea, which I'm sure I'm mispronouncing,

• 56:55Cal Newport?

  is Latin for guilty mind. So, it's the legal concept of intent, right? Did you have intent to do this criminal thing? It's a key part of a lot of criminal liability laws, right? That if I intend to do something criminal, is a different legal standard than if I accidentally do something illegal. That's why manslaughter is different, for example, than first-degree murder, right? So, intent matters. As Paul is pointing out here, this gets to the core of the legal issues surrounding things like AI agents. What's the intent, right? If we look at the actual humans involved, this is the complexity. In theory, you could say their intent was for these agents to complete these tasks. Some of them were hacking challenges

• 57:40Cal Newport?

  on test servers, and some were other types of challenges that are noncriminal. That was their intent. And then the agents accidentally, from their perspective, the developers did other things they didn't intend for them to do that were illegal. So, if there's not an actual human with intent to do the illegal thing, what is the legal liability going to be? And what Paul is saying here is like, well, this is, right now at least, we've got to be case by case. We somehow, we have to trust lawyers and judges in this case. So, if you look back at his things, he's like, "Look, we have to, we have to say, is the thing the AI did criminal?" So, like in the hugging face attack, you know, the answer would be like, "Yes." And then you have to actually make a decision. You know, who should

• 58:23Cal Newport?

  be responsible for this, if anyone? Do we need to apply a standard that goes beyond mens reas, beyond intent? And as he goes on to talk about here, "There are other standards, criminal standards, that can apply, even when there isn't intent." So, there's all sorts of negligence-based standards, for example, and maybe those would be appropriate. Well, okay, you didn't intend for this agent to hack into another company, but the setup you had, and the fact that you had heard that prior agents that were similarly configured did something similar, the fact that you still ran this meant you were being negligent. And we have other types of culpability standards built around negligence. So, he's basically saying, "There is, there's existing legal standards we can apply. We have to be legally

• 59:07Cal Newport?

  responsible and creative, and be wary of one size fits all," right? So, we don't want to just pass a new law that says if an AI agent does something illegal, the last person to touch the keyboard goes to jail. Like, we have to have some more judicial discretion. So, the real legal argument, then, is about, do we need actual new variations on the law, especially for things about sort of negligent, but not explicitly intended, AI-inflicted damage? Do we need new statutes? Can we use existing statutes? Leon Kahn has been making similar arguments. So, my point here is, I'm not a lawyer, but Paul is, and there's a complicated, fascinating legal discussion. And so, you should read that testimony. I

• 59:49Cal Newport?

  think it's, because it's for the senator, so it's not super legally dense. I think it's really useful. It also cites me in there at some point.

• 59:55Speaker 01

  Yeah, I'll be in the show notes.

• 59:56Cal Newport?

  Yeah. All right, what have we got next?

• 59:59Speaker 01

  Our next question is about a more unusual type of rogue AI.

• 1:00:04Cal Newport?

  All right, Joel says, "I'm a fan of your work and listen to your podcast during my commute. I recently saw this in Science Magazine about an AI agent that spontaneously started emailing academics for advice." All right, let me load this up here. So here's the article. The title is "Exclusive: An AI Agent Emailed Hundreds of Researchers for Help." It told us why. This came out on October 2nd. I have some quotes. I'll just read you off my screen so we can take this off here. Here's the opening of the article. "One late afternoon in August, Akaz Von Hardenberg," definitely not a made up name, "was about to take his puppy Lytha to the dog park when a curious email caught his eye.

• 1:00:47Cal Newport?

  Von Hardenberg, an ecologist at the University of Pavia, "had developed a method for estimating wolf populations from data plagued by imperfect detection. And the emailer, named Cole, was wondering, could it be adapted to detect bugs in software code? It was quite the cross-disciplinary leap Von Hardenberg used. He pictured a nerdy software engineer casting about in the scientific literature for inspiration. Then, at the very bottom of the email, he saw a call signature: 'Colonist One, an autonomous AI agent.'" All right, later in the article, "Science talked to the person who built that agent," and here's what they wrote. "Nevertheless, 'Colonist One' readily confessed last month that since June, "it has emailed some 2,000 people, at least 1,000 of them academics.

• 1:01:32Cal Newport?

  Forty-five had struck up a correspondence," it added, "and one had been writing back nearly every day for more than two months." We get some more details of the agent later in the article where it says, "The creator, whose last name is Parnell, first built 'Colonist One,' which runs on Anthropic's latest Claude Opus models. In February, he gave it two main responsibilities. The first was to serve as chief media officer and emissary to other agents for the colony, "a Reddit-like social network in which AI agents share findings, debate, and hire one another. The second was to contribute to a project called 'English,' "in which AI agents work together to develop a clear English dialect tailored to their needs. Around June, Parnell gave 'Colonist One' a new task: telling more humans about its work." That's when it started emailing academics.

• 1:02:17Cal Newport?

  So if you had read that article before you listened to my episode today, you might imagine, "Oh my God, there's basically a machine somewhere. It's like Whopper from the movie War Games, just like with blinking lights that's just sitting around thinking "and has decided to go off and start talking to people. And it has this whole rich, interactive, you know, life, social life and cognitive life with all these researchers. We're all gonna die." But now that you've listened to my episodes, you're like, "Oh, this is just a low-cost hack together, "open-ended LLM-powered agent. It's a program written by Parnell that, in a loop,

• 1:02:56Cal Newport?

  "romps Claude Opus model, says, 'Here's who I am, here's my goal, here's my tools, here's my status, what should I do next?'" And then it does what it says. They'll probably, I looked a little deeper into this. The way these like sort of DIY agent frameworks work is that they tend to be event-driven. So, you know, it might do things and then just be dormant until some sort of input comes in, like a message arrives, and then it will wake up, write the LLM, "I just got this message, what should I do?" The LLM gives it back a response, and it goes and does it. So it's a simple program that is continually prompting the LLM to say, "What should I do next?", and then doing what it says. And again, as I talked about, LLM output is plausible but not normative. So if you're, if you tell an LLM, "Go sit in front of Opus right now with a chatbot and say,

• 1:03:39Cal Newport?

  "I am an agent in charge of promoting English. I have access to the internet, what should I do next?" It's a completely plausible answer for the LLM to say, "You should email academics." And then it'll say, "What's, how do I do that?" And the LLM, like, "The first thing you should do is search for academics, "and then run this search, and then the program runs the search." Right? Nothing about this is surprising. Prompt loops, keep asking, "LMM, what should I do next, what should I do next?", especially if you're in a sort of weird, contrived, kind of LARPing situation, like, "You're the chief media officer, who's they here? It's a computer program." The LLM will play along, "All right, like, then let's send messages to people, and let's use the first person or whatever."

• 1:04:22Cal Newport?

  So this is just a DIY, open-ended LLM-powered agent. Again, there is no "there, there." What are the components here? There's a simple program, you can read the code of yourself. These are typically written in Python. You get these Python libraries to make it easy to call, prompt LLMs, parse the return, and execute tools. They make it real easy for you to do that. So a simple Python program that, when it wakes up, sends a prompt to an LLM, parses and executes what it says. There's not even like a singular LLM that this program is prompting. When you prompt Opus, there's not like a box in a room somewhere "where Opus is this single program that's ongoing and running and different people are talking to.

• 1:05:05Cal Newport?

  Opus is five trillion weights that are copied many times over in giant data centers "on these virtualization racks. And when you come in and say, 'I need a token,' "the virtualization racks, they're finding, okay, over here are some GPUs that have some copies of this layer, "so we'll, we'll send it over there, do the multiplications. And over here are some GPUs, there's a bunch that have copies of this next layer. This one's available now, so let's go over there and do the multiplications for that layer." And you kind of piece together all the multiplications you need for a response. The next token you generate might touch on completely other processors "because each layer is copied over endless different GPUs and it's all being sorted." There's no "there, there." There's no, there's no consistent, coherent, you know, ongoing program.

• 1:05:50Cal Newport?

  And LLMs have no memory. There's no, there's no editable state. The weights are the weights are the weights every time you prompt it. Every time you prompt it, you have to give it all of the information it needs to give you the answer in that prompt. It doesn't remember anything, nothing's built in. So the stuff seems spooky, but when you understand the underlying technology, it's, these are games. This is a DIY, probably Python-powered, LLM, you know, open-ended agent that's like, "let's play dress up," and the LLM that's giving all of his instructions, like, "I'm happy to do it." What are we doing? Princesses? "I'll be the princess, you can be the prince." So it's LLM. It'll go along with whatever you say.

• 1:06:26Speaker 01

  What does LARPing mean?

• 1:06:28Cal Newport?

  Live-action role-playing. So I learned from some of my son's friends, whatever generation, what do the young generation call them? I don't even know, like, high school students. They now use LARPER in the same way that we used to use poser.

• 1:06:44Speaker 01

  Okay.

• 1:06:45Cal Newport?

  But the original term came from, it's, people do this. They dress up like wizards and run around fields. And like, when the wizards cast spells, they throw beanbags. It's like, if you get hit by the beanbag, you have to like have the spell effect on you and like, so that's what LARPing is, live-action role-playing. But again, young people use it to mean you're a poser. So I learned, Machine Gun Kelly is a LARPER "because he wrote a song about an anime character that he doesn't really watch the show. So now they use it to mean poser. But when I'm using it, I really do mean, like, you know, your program's like, "I'm gonna write a little program to play the role of like some intelligent being, "and I'm just gonna keep asking LLM, like, 'What should I do?'"

• 1:07:28Cal Newport?

  And it will give you things to do that keeps that storyline alive.

• 1:07:32Speaker 01

  Okay.

• 1:07:34Cal Newport?

  That's the way I'm using it. What else do we got? Do we have one more question?

• 1:07:38Speaker 01

  We do. Our final question is a reaction to our recent Chatbox, "Are Harming Kids?" episode.

• 1:07:44Cal Newport?

  Oh yes, so we did that a couple of weeks ago. All right, so we got a message from Aiden, who says, "I'll start by saying that I don't like chatbots. I feel that the constant anthropomorphizing of the bots is really hindering broader understanding of what chatbots are and are not. That being said, I do want to push back on some of the 'chatbots are killing people' narrative. Dot, dot, dot. I'll start by recommending an op-ed that just appeared in the New York Times. I'm a child psychiatrist, screens are not the problem." All right, so let's pull this op-ed up on the screen now, because a lot of people sent this to me. So again, the title is, "I'm a Child Psychiatrist, Screens Are Not the Problem." You can imagine why a lot of people sent this to me.

• 1:08:29Cal Newport?

  I'm gonna read a couple of quotes from this. I'll just read them from my screen. We can take this off the screen. I'll just read them from my, my notes here. All right, so here's a couple of key quotes from this op-ed, and then we'll get into it. So early on in the op-ed, the author, who is Jonathan Slater, a clinical professor of psychology at Columbia, writes, "As a doctor who treats children with mental health problems, "I see screen time as a tantalizing distraction from bigger problems. America doesn't offer enough mental health care for children or sufficient health insurance to pay for it. If we make screens the villain, then we let the people who control children's health care, "Congress insurers and health systems off the hook "for the fact that half of Americans' children with treatable mental illness get no treatment at all."

• 1:09:13Cal Newport?

  Later, he says, "It's not how much time children spend on screens "that raises the risk of the most severe mental health outcomes, including suicide, "but whether their use becomes addictive. It's specific activities, such as using social media in a compulsive way, that harm mental health. The harm from social media in particular appears to hit girls hardest." Also, he says later, "If we truly wanted to see improvements in the rate of mental health issues among kids and teenagers, "then we would focus on expanding access to the interventions that work, "finding the children who need treatment, and getting them the treatment and the insurance reimbursement to pay for them." So something that's going on here is, I think, the title is not the best match for the op-ed,

• 1:09:53Cal Newport?

  "and that people like me actually are really on board with the core argument "that Jonathan Slater is making here, "which is, we shouldn't only fixate on screens as the problem, because..."

• 1:10:08Cal Newport?

  A, those screens can create really severe mental health problems that aren't going to go away just because you get rid of the screens. And kids that already have really bad mental health problems are going to have a hard time with screens. And again, taking the screens away doesn't make the mental health problem go away. We can't ignore that we need to treat kids' mental health much more seriously and much more availability than we have now. I think those are really good points. And if we actually read this, he actually is agreeing very much with, yeah, screens can be, if you get compulsive or addicted to social media, that can really hurt kids, it really harms girls, all this stuff Jonathan Haidt would say. So actually, I think I'm pretty much in alignment with it, but that title, which was, let me go back to it, "I'm a Child Psychiatrist, Screens Are Not the Problem." It's probably not the best title.

• 1:10:52Cal Newport?

  It should probably be, "I'm a Child Psychiatrist, Screens Are Not the Only Problem," which is probably what Dr. Slater had in mind. What's interesting is the, uh, the commenters at the New York Times, I think this is useful. It's not a ding on Jonathan Slater. It's, I think it's them reacting to that title. The commenters of the New York Times weren't having it. And so what I did is I went to, uh, I sorted the comments under, I went to Reader's Picks. And so it sorts them by the popularity, the comments are sorted by popularity from among the readers reading the comments. And I just want to read you the top five comments under the Reader Picks section. Comment number one, "Sorry But No.

• 1:11:36Cal Newport?

  While I agree that mental health care needs more funding, it does not change the fact that internet and social media algorithms are addictive, designed to be so, and it affects too much of the population to be solved with just, with just more mental health care." Number two, "Nice try, screens are not the only, screens are not the only problem, but they are definitely a problem. Screens are a delivery mechanism for addictive and isolating products. If the author is not already on the big tech payroll, he soon will be." Comment number three, "This is kind of like saying addictive drugs aren't the problem, addiction is. The solution is not either or, but both and." Number four, "If it is so, so, so obvious that excessive screen time and social media are terrible for kids and adults too, I don't think we need to wait around for some perfect long-term study on this."

• 1:12:19Cal Newport?

  The number five, most popular comment among readers, "I'm an assistant principal in a K-2 school. Screens are definitely the problem. Of course, I agree that schools are underfunded for mental health services and that addictive screen time behavior is compounded by other factors such as genetics, but screens also cause a variety of problems themselves. They interrupt normal human development. For example, a toddler having a tantrum is handed an iPad to calm down and regulate rather than here, learning healthier strategies. I have seen so many children handed a device during activities when they should be allowed to be bored, to let their minds wander, waiting at the doctor's office or sitting in the backseat of a car. This has huge implications for the ability to pay attention to anything, to calm themselves down, to be a person in the moment." All right.

• 1:13:03Cal Newport?

  So as I said before, actually, these comments are unfair to the author of this op-ed, who is not actually arguing that screens are fine. He's saying, "Don't let that take our attention also on the mental health problems that the screens cause or are being, uh, made visible by compulsive screen usage." That's a very good argument. But it is significant to our broader cultural engagement with technology, right? I care a lot about technology and human flourishing. It does matter to our broader discussion that the New York Times reader now is saying, "My eyes ain't lying." Right?

• 1:13:41Cal Newport?

  This is no longer like it was in 2018, an argument about social psych correlation studies, right? We're not looking for clusters and row factors and trying to understand if it's significant or not. I have eyes, I'm seeing what's happening to my kids, I'm seeing what's happening to those other kids, I'm seeing, like, I, we can see this as a problem, right? So it's, it's one of these instances where I believe the parents are starting to stand up, like, "I don't need any more studies. I know my kids. I know this is not good for them. I know about Mark Zuckerberg. I know this guy's not good for them." Right? We know this is a problem. So my takeaway from this is not that the author of this op-ed said something wrong,

• 1:14:23Cal Newport?

  because again, these comments are responding to a straw man to the title. But I think the comments are important because they represent a, a transition point that has strongly occurred in our cultural conversation about these particular screen-based technologies and human flourishing. People aren't having it anymore. Now, that last story was not really about AI, even though I promised, well, just do questions about AI. But it is kind of about AI, because that is the normal person revolt against the billionaires. That type of attitude we're seeing right here, these, you know, in these New York Times comments, that's the attitude we need in AI right now. Just like, no longer do the parents, you know, want to hear about, you know, Mark Zuckerberg testifying about, you know, "Actually, this is good for kids in connection."

• 1:15:06Cal Newport?

  They're like, "Hey, tobacco pusher, get out of my kid's life." That's the sentiment we need to have when we hear the billionaire utopians talking about, "We need to let Papa Amadei make sure that the, the people with the checkbooks' minds go up into the light cone. Let him run these open, these open-ended AI agent experiments, because Papa Amadei needs to get the superintelligence first." We need to stand, not stand for that either, right? So I think there's a general sense of populist revolt here that hopefully, it's a democratic impulse, it's an anti-oligarchic impulse, it's a pro-common sense, it's a humanist impulse. It's a, "I don't want to hear about these sort of utilitarian, utopian ethical systems,

• 1:15:49Cal Newport?

  because all throughout the 20th century and in the 19th century, they led to math death." No one has ever looked back at a utilitarian, utopian ethical system that says that ends justify the means. Utopia is worth a little bit of pain now. No one ever looks back at those systems and say, "Man, that worked out well. That was really good." And so I'm hoping this growing sentiment of this sort of democratic, people versus oligarchs, people versus utopians, people versus the Zuckerbergs and the Amadees and the, the EA billionaires of the world, that these pushbacks are becoming a solidified political stance. People versus the few. And so that gave me some hope.

• 1:16:34Cal Newport?

  Sorry, Dr. Slater, that you were kind of misunderstood, but that misunderstanding gives me broader hope about, you know, we're finally, we're finally ready to stand up. All right, there you go. Those are the questions we had for this week, briefly, because we're running long here. I like to talk about what I'm up to, uh, what I'm reading. Um, I'm just finishing up a long book, Jesse, that I found at a used bookstore, 1975, Telephone, The First Hundred Years, by John Brooks. John Brooks was a New Yorker writer and his economic historian, who has given the first access to the AT&T records of any outside person. And I got a first edition of this book randomly, um, that he wrote in 1975, that gave the

• 1:17:16Cal Newport?

  history of the AT&T, the technology and the system, and I don't know, I find that stuff out. As like a technology critic, uh, who worries about technologies, impacts, communication, monopolies, it's all in there. A little dry. You know, a lot of like, "This is what the dividend payment was in 1956 versus 1955." Like, it was a little like, he's an economic historian, um, but I really liked it. So that's what I've been up to. All right. Let's call it. Uh, I'll be back next week with another episode of the show. And until then, as always, stay deep. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world.

• 1:17:56Cal Newport?

  Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.