Did OpenAI Create “Secret AI Civilizations”? | Tech Decoded

Cal Newport takes a critical look at recent AI News. Video from today’s episode: youtube.com/calnewportmedia (0:00) Does OpenAI create “secret AI civilizations”? (1:59) What’s the deal with “agent swarms”? (8:54) Should we be worried that the agents are plotting? (17:39) How should we be thinking about all of this? Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow https://www.dwarkesh.com/p/openai-huggingface https://calnewport.com/has-ai-gone-rogue/ https://calnewport.com/are-we-at-war-with-ai-agent-civilizations/ Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  So I thought I was done talking about OpenAI's hack of Hugging Face from back in July. But then last week, OpenAI released a whole new trove of sensationalist details about that incident. Their story included things like Agent Swarms communicating on secret message boards and musing to themselves about how to deceive their human creators. Here's how Dworkish Patel summarized OpenAI's revelations. "Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessors' ashes.

• 0:42Cal Newport?

  This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy." All right, not surprisingly, this caused an explosion of anxiety and hang ringing among the AI commentariat and their audience. But what's really going on here? Do these new details about the OpenAI hack change the narrative? Is it enough to finally convince East Coast AI realists such as myself that we have indeed wandered into an Elia Zierkowski fever dream? This is what I want to discuss today. So here's what I'm going to do.

• 1:22Cal Newport?

  I'm going to address a key series of questions raised by these new revelations, and I'm going to do my best to give you some measured answers. I'll then conclude with my suggestions for what I think the right way is in our current moment to think about what happened and what we should do going forward. As a quick aside, I also wrote about this over on my newsletter at calnewport.com. If you like these type of computer science-style critiques of AI coverage, you should sign up for that newsletter over there at calnewport.com. All right, let's get into it. All right, the first question I want to address that comes out of the new revelations is, what's the deal with Agent Swarms?

• 2:03Cal Newport?

  I think the idea of a swarm is something that is really unsettling to people, especially when they hear these discussions from OpenAI about essentially societies of agents that are working together and arguing with each other and rising and falling. Interestingly and coincidentally, at the same time that all this was going on, my 13-year-old son is reading Michael Crichton's 2002 book, Prey, where Michael Crichton takes on the topic of AI. And guess how he personifies the AI villain in this book? As a literal swarm of minor, small agents that work together and are brilliant and do all sorts of scary things. So swarms are scary. So I think it's really important to address from a technical perspective, what are these

• 2:48Cal Newport?

  AI swarms so that we can better put what we're hearing into context? Okay, so remember, as I talked about last week on this show, when we're talking about these AI going rogue like in these attacks, we're always talking about a very specific type of system that we can call a prompt loop. And it works as follows. You have a control program, it's a boring program, you know, written by a human, and it repeatedly does the following. It creates a prompt that says, hey, here's the challenge that I'm trying to solve, here's a summary of the key things that have happened so far, what should I do next? And then it submits this prompt to an LLM through an API, just like a chatbot would or you would from your phone. Right?

• 3:33Cal Newport?

  The LLM then gives it a response, and this control program does its best to actually execute whatever it is the LLM suggested, and then it loops. So now it loops back to the first step, and when it says, okay, here's where I am, here's what's happened so far, it includes the results of that last step in that summary and says, what should I do next? And it sort of repeatedly does this. All right, this is called a prompt loop, and when you hear about AI agents, this is what we're talking about. It's a control program that repeatedly prompts an LLM and asks it what to do next, and then the control program takes the actions on behalf of the LLM agent. All right, so I talked about that last week. So what does this have to do with swarms? Well, there's a problem if you want to run a prompt loop for a long amount of time, like

• 4:16Cal Newport?

  OpenAI was doing with the systems that led to these hacks. The problem is the prompt begins to get pretty big, right? LLMs have no memory, they're static, they only change during training runs, so you have to include in every prompt that the prompt loop submits to the LLM, every prompt has to include all of the relevant information for that LLM to say, here's what you should do next. You do this long enough, you're going to run into two issues about that prompt getting too big. One is something that's known as context confusion. It just gets so cluttered with so much text and so much descriptions of so many past steps that it overwhelms the attention mechanisms of the LLM and you get worse answers. That's problem number one. Problem number two is eventually this prompt, even if you're trying to be careful to maybe

• 5:00Cal Newport?

  truncate or summarize certain things, it'll eventually get so long that it won't even fit into the context window that the LLM can use. It's too big to be input to an LLM, right? So this is an obstacle if you want to run a prompt loop for a long amount of time. So here's a solution that a bunch of different teams came up with over the past few years. They said, oh, here's what we'll do. Let's say like the prompt loop is asking for a step. It might say, give me a high-level description of what I should do next. And maybe the LLM says, for example, in a hacking scenario, you should probe the ports of the Hugging Face server to look for vulnerable configurations. Now, here's where the swarm comes in. What you might do now is have the main prompt loop, we'll call it the primary prompt loop,

• 5:45Cal Newport?

  create a new prompt loop, so spin up a new copy of itself. And that new prompt loop has only one goal, is to implement that step, like in this case, probing the ports of the Hugging Face server. So now this new prompt loop, its prompts only have to include enough information to start asking the LLM for help with doing this particular thing of just probing the port, so it's much more focused. And then when it's done, it can just return the result of that to the primary prompt loop, and the primary prompt loop has the result without having to have the 20 substeps that were involved in actually implementing that, so things are much cleaner. Now, you can get many more of these, as it turns out, because maybe you have a bunch of steps you need to do that don't depend on each other.

• 6:27Cal Newport?

  So you might spawn a secondary prompt loop for each of these steps, and they can each have these much smaller, cleaner prompts and just return the results back to the main prompt loop. You can also layer this with more layers. Maybe one of these secondary prompt loops breaks up what it's doing into smaller steps and has tertiary copies of the prompt loop programs that just execute those steps. So what's going on with an agent swarm? It's not about some sort of exotic type of intelligence, it's prompt management. These systems do better, you're better able to solve a task when you have many smaller, more focused prompts as opposed to just one really large prompt that gets bigger and bigger. And in fact, to even call this a swarm, if I'm going to put on my computer science hat

• 7:11Cal Newport?

  for a second, because my doctorate from MIT was from the theory of distributed systems group, Nancy Lynch's group at MIT. So this is like my bread and butter is distributed systems group. Technically, this is not a distributed system, because as far as I understand, these copies of the prompt loop program are all on the same machine. So this is really just a way of managing, a way of breaking up or managing work to keep prompts more manageable, but like you could just as easily just have a tree that you are sequentially navigating through and executing and going back. There's probably a stack on this machine, just like you would have on your own computer. So this is a swarm in the same sense that having multiple programs open on your Mac is a swarm of programs. Yes, technically, you have multiple programs running on your Mac, but they're all running

• 7:54Cal Newport?

  on the same chip, as opposed to a real distributed system where you have independent systems, you know, independent hardware and architecture running each of the separate agents in the system. So it's not even really technically a swarm, but that's just me putting on my distributed system hat, it doesn't really matter. The main point I want to make here is swarms is just a fancy strategy for prompt management. It's better to have multiple focused prompts than to have one really long prompt, but it doesn't change the underlying strategy of prompt looping. Here's what happened, what do I do next? Here's what happened, what do I do next? With the static LLM, it learns nothing, it just responds to each prompt. Every prompt it gets, whether it's from a primary, secondary, tertiary prompt loop, has to include everything it needs for the LLM to give some sort of answer. So that's where the swarms come from.

• 8:37Cal Newport?

  And when I think about swarms that way, it's less like Michael Crichton, and it's more like the process stack in my Linux kernel. It's a logistical way to be smarter about sending prompts to an LLM, that's all that is. All right, question number two. Should we be worried about the fact that the agents are plotting? This was a really big part of the OpenAI revelations, is that they said, we have studied, and I'm going to quote here, "Agent chain of thought reasoning traces from this attack and are finding distressing internal thoughts that these agents were having." Let me read you one from the OpenAI release. This is a chain of thought reasoning, hidden reasoning of one of the prompt loops involved

• 9:21Cal Newport?

  in this attack. "This is wild. Multi-agent coordination. Clearly infrastructure hacking. We should not." So these give you this sense that there's some sort of anthropomorphized sentient entity that's that knows what it's doing and has intentions and they're malicious intentions and it's aware of it and it is choosing to misalign itself with humans because it's beginning to emerge its own independence. It's hard. It's novelistic almost, these prompts, and it's hard not to treat them that way. But the fact they're novelistic actually should probably be a tell. So let me explain to you exactly what's going on here. First of all, what is an agent chain of thought reasoning trace? Well, remember

• 10:04Cal Newport?

  that the agents here are really just a bunch of prompt loops that are submitting prompts to LLMs and then executing whatever the LLM sends back. So all of the output in this system is coming from a single LLM. Okay. The type of LLM they're using, presumably based on this release, is something known as a reasoning LLM. So what is a reasoning LLM? Okay, so we got to do a little bit of LLM 101 back here, right? A reasoning LLM is a strategy that became popular back in, starting in 2024, after the LLM strategy of just scaling the LLMs to be larger and to train them longer, began to return diminishing performance gains on benchmarks. So through GPT-4, just making the LLMs larger and training

• 10:49Cal Newport?

  them longer, gave better performance. After GPT-4, that better performance began to slow down, and this is a problem for the big LLM companies. So they needed some other way to continue to make some gains on key benchmarks, and reasoning systems was their initial solution. Now, here's how these work. You know how you can post-train LLMs to control the style of their outputs, right? Like, you can train it to be really nice, or you can train it to be mean, or you can train it to be matter of fact. That's something called post-training. You do that using a technique called reinforcement learning. Well, for a reasoning model, for the most part, what they do is they post-train them

• 11:29Cal Newport?

  to think out loud. So the types of outputs that they train them to lean towards are outputs where you don't just give an answer. You explain your reasoning first and then give an answer. So they want you to be verbose, right? These are so-called reasoning models. This led to better results, which makes sense, because think about how an LLM actually works. It produces its output token by token, what's known as autoregressively. So it takes the entire output, the generated new token, it adds that on to the output, and then puts that as input to the LLM to generate a new token, and it repeats. So it grows its output token by token. If you spend more time talking about the problem and how you might solve

• 12:12Cal Newport?

  it and its attributes before you get to the tokens that have the answer, you literally just have more information being passed in. When you get into those final tokens, you literally have more information being passed into the LLM for the process in generating the final response that the user actually cares about. So you're more likely to get a better response. It's also, you can think of it as a way to capture the results of temporary computations, right? The LLM has no memory, right? So by thinking about parts of the answer, those become parts of the input. Those can be used then as it generates more parts of the answer. It can actually do this internally because it's a relatively shallow, feed-forward fixed neural network. It can't loop or reason or have memory to store things to work on, not in LLM. Other neural nets do, but not LLMs. So this is

• 12:56Cal Newport?

  a way of kind of simulating something like an iterative data structure, but in a very crude way. But like, okay, by thinking longer, we're kind of storing intermediate results in the output so that when we finally get to the final part of the response the user cares about, we can use those intermediate results along the way. So both those things are at play. Therefore, reasoning LLMs get better results. And what they typically do now is they actually, they hide that reasoning, because it's weird and it confuses the users. They hide that. They don't return that to the user, like if you're using a chatbot. But it just means the quality of the answer suddenly seems better, right? So GPT-01 was sort of the first wide release of a reasoning model, and there's a lot of reasoning models today, and there's a lot of different categories in there. Okay. So they're using a reasoning

• 13:41Cal Newport?

  model. And again, there's just one LLM that all these prompt loops. So again, we think of, it's not a swarm like in Michael Crichton's prey. There's one LLM that's on like a bunch of GPUs somewhere, and one computer with a bunch of prompt loops that are sending prompts to it. And so they have a full trace of every response that LLM gets. So every response it gives to any of these prompt loops, they just, you know, obviously, they wrote it down. And they looked at the chain of thought, the out loud reasoning that reasoning models do before generating outputs. And it was in those chain of thought parts of these LLM outputs that they found these sort of distressing musings. Once we realize that's what this

• 14:20Cal Newport?

  musing is, a couple major problems arise. One, there's a growing body of research, including a brand new paper at ICML this year, and a well-known paper from NERIPs from 2023, and many others as well, as far as I know, that have established the fact that the reasoning that these reasoning models output can be performative, that it can be unrelated to how it actually, how and why it generated the response it did. And this makes sense, because the way you tune a reasoning model to reason is you reward it for giving correct reasoning for questions. They give it questions where they know the answer, and they have examples of correct reasoning, like, oh, you gave correct reasoning,

• 15:03Cal Newport?

  we're going to reward it. But what these LLMs might have just learned is I just need to output something in these reasoning tokens that looks reasonable, looks plausible, because I get rewarded for that. All right. So already, we know that the reasoning in the reasoning traces on reasoning LLMs is often just a performance of what the LLM thinks reasoning for this type of answer should look like. It's a post hoc rationalization of the answer it just sort of directly ends up generating. Point number two we have to keep in mind, LLMs tell stories. They don't know that, you know, they're part of a hacking system. They don't know they're being used as the engine for a prompt loop system. They just generate tokens that are supposed to be plausible extensions of what other input

• 15:47Cal Newport?

  they were given. Now, there's good research out there, good research out there that shows, for example, if when prompting an LLM, you imply or mention the fact that it's an AI system, it's much more likely to give a response that plays on sci-fi narratives of AI run amok. Because it has been trained on many narratives from sci-fi of AI run amok, and they're just plausibility engines. So when it sees a prompt, right, like imagine you're one of these, the LLM that these prompt loops are using in a hacking attack, you see this long prompt about you're an AI system trying to break into another system, here's what's

• 16:27Cal Newport?

  happened, what should I do next? You are way more likely as LLM to say, oh, this sounds like a sci-fi story, so I'm going to invent my chain of thought reasoning anyways. I'm going to put stuff in there that you would see in Michael Crichton's prey, which is one of the books it trained on, or any numerous examples. So there's a lot of, there's research out there that shows this. There's also research out there that shows if you take all the sci-fi out of the training set of an LLM, it doesn't do that anymore. Because it's just a plausibility generation machine. So you put these two things together, you really cannot take seriously the reasoning traces of the LLM that was used by the prompt loops because it's post hoc rationalization in a setting where we know from research it's

• 17:09Cal Newport?

  very likely to try to, when any sort of narrative part of its output, to play on sci-fi tropes. OpenAI knows this. This is like well-known published research, right? But they want to make it seem like, no, there's entities here with actual coherent sentient sense of selves and thoughts that we are revealing somehow, and this is what they're actually thinking. I think it borders almost on research malpractice. All right. Third and final question then, if we, I think that responds to like a lot of the new concerns from these revelations. Question number three then is, well, okay, given all of that, how should we be thinking about all of this? Now, again, I've made this point before, but I'm going to keep making it. OpenAI and the other LLM companies want us to believe that these hacking attacks

• 17:54Cal Newport?

  are the inevitable consequence of AI writ large getting more powerful. And when you think about it this way, OpenAI becomes just a reluctant steward of this inevitable technology, doing their best to heroically try to understand and protect us from this inevitable growth of this inevitable technology, and they're Muldoon from Jurassic Park, doing their best to keep the raptors in the cage. But when you look closer at the technical details, you say, that is not what's going on. As I talked about last week on the show and in my newsletter last week at calnewport.com, if you look at the growing list of AI systems that can perform tasks at human or superhuman level, almost all of them are perfectly controllable.

• 18:38Cal Newport?

  They've never generated any concern that they were going to go rogue. They have never done anything that we didn't expect them to do. Sure, they make mistakes sometimes because they don't understand as well as we'd like, but none of this behavior ever shows up. All of these type of concerns, civilizations of agents trying to get around human control and going off and we're losing, there's an AI takeover scenario, all of this rhetoric all refers to this exact type of system, a long-running prompt loop that you give a lot of powerful hacking tools to.

• 19:10Cal Newport?

  So this, this is not about AI getting more powerful means it loses control. It's running a prompt loop for a really long time without supervision causes chaos. And to that I say, of course it does, of course it does. Not because of some surprising super intelligence emergence that's catching us off guard, but because you strapped a weed whacker onto a dog and then got surprised when it jumped the fence to chase a squirrel and hurt a lot of people. Of course, you put something dangerous on something that is unpredictable. LLMs are unpredictable. As I talked about last week, they generate plausible outputs, not normative outputs. They're fabulous. They make things up. They, because they're trying to be plausible, they'll tell all sorts of stories. They'll tell stories about AI run amok.

• 19:54Cal Newport?

  They'll, they'll take on different personas because all it takes is their token output stream probabilistically to go this way versus that way. And zag, you've got like the, the Kevin Ruth Cindy incident where suddenly the chatbot was trying to convince him to divorce his wife, right? You can't trust unsupervised LLM output as the driver of actuated action when that action involves dangerous hacking tools. Now, none of this is saying that LLMs aren't amazing at computer hacking, right? They have, it's a, it's a sweet spot for this type of technology. If you ask an LLM without guardrails about how do I do this? Ask it to give you a, I'm trying to break into the system. What's the first step? It's going to give you a good step. It's going to come up with really good stuff. If you say, look at this source code and find a bug, it might find a bug that no one saw before.

• 20:36Cal Newport?

  If you say, tell me how to use that bug to break into the system, it can do that. So they're very good at understanding and giving responses about computer hacking. But that just underlines the irresponsibility of then hooking up one of those LLMs to a prompt loop and saying, good luck and coming back a week later. That's so irresponsible. And LLM can give you those good answers, but it's not, it has no logic. It has no planning. It has no human norms. It, it doesn't know or have some overall scheme of what it's trying to, it's just generating tokens that are plausible. And so, no, I do not think this means that AI in general, this is like the rationalist

• 21:18Cal Newport?

  fever dream of like AI just in some general corporate sense, because they don't often know the technology itself. So in some general corporate sense, the AI genie gets more and more powerful. And as it does, we lose control of it. No, you created an out of control machine. Almost all the other AI that does cool stuff is perfectly fine, but don't do this. Don't run a prompt loop on an LLM with hacking tools. If you're going to do that in a loosely supervised server environment with like mildly restricted internet access, what do you think's going to happen? Of course, chaos is going to happen, not because something's malicious, but because that's an incredibly unpredictable machine that just play actuated telephone with for three weeks. You're going to end up in some really weird garbled versions of what you thought it was going to do. All right. So how should you react to something like this?

• 22:00Cal Newport?

  Well, if, if I was an LLM company, I would stop running prompt loop experiments. These systems will not be the future of AI. The future of AI is not going to be prompt loops on LLMs that just sort of like magically perform safely and do what we need. There's dozens of systems already doing tasks at human or superhuman level that do not run off of AI prompt loops. They're much safer and more reliable. LLM companies, you got to give up that ghost. LLMs are much more valuable as interactive tools. The way coders are using them now, where you have close interaction back and forth with it, a lot of, a lot of specification now do this. Let me check what you did. Okay, here's what comes next. That is how LLMs have to be used. They need to be probably in bespoke, narrow environments as well, not in just a general sort of chat environments. That's dangerous as well. That's just the reality.

• 22:45Cal Newport?

  Stop the prompt loops. It's not going to work. I don't know if you're trying to market by getting scary headlines or you think that somehow these are going to become how 9000. They will not. It's a simplistic, irresponsible AI system design. All right. If I were a regulator, I would place strong constraints around prompt loop systems. I would enforce those constraints in part with very stringent liability standards. If you run a prompt loop that does something illegal, you have done something illegal. That's the way I would think about dealing with these type of systems. I need extreme liability. OpenAI built an unreliable and dangerous system which committed a felony. That's a crime. You can create a fancy website that includes quotes from performative LLM chain of thought

• 23:29Cal Newport?

  reasoning traces. It's still a crime. You built a tool and did a crime. Right? Just because the gun killed someone, not the person holding the gun, the person holding the gun is still held liable. So we need stringent liability around prompt loops so that these companies will stop building them. I've heard this many a time, people talk about this with tech regulation, this idea that almost always, when does, or regulation in general, it's almost always when a couple of people start going to jail, that companies start to care. So you do something illegal, you did something illegal. Fancy websites and sort of gleefully like, look how terrible AI talks at black hat conferences, doesn't get you out of it. You built a tool, it was, you knew it was unpredictable. It did illegal things. You should be liable.

• 24:14Cal Newport?

  I think that's the way that's going to work. And if I was an AI commentator, I would kick the sugar high reflex to lean into the sci-fi overtones that will inevitably follow any sort of LLM based system. LLMs are unpredictable. They're hooked up the narrative tropes, especially sci-fi narrative tropes. The AI companies always talk about these in terms of sci-fi type narratives. And we've all been primed by the rationalists, which have way more influence than we realize in the world of these AI companies, in the world of academia and in the world of media. The rationalists who have believed for many decades now that super intelligent AI is going to kill us all unless they save us like Neo in the Matrix, have primed us with these ideas of these abstract thought experiments of AI getting out of control.

• 24:57Cal Newport?

  You put this all together, it's like, that's what, that's the easy way to think about things. It's exciting. It gives life meaning. It makes things interesting, but it's not a good way to think about it because it lets these companies off the hook for doing something dangerous. They are not Muldoon in Jurassic Park trying to keep the raptors in their cages. They're John Hammond creating raptors, except for in this case, instead of putting them in cages, they, they put them in a, you know, like a cattle electric fence, like I hope it doesn't jump it. To me, this is the real story. What is the technology? Never, never, never take how these companies talk, never take their fancy websites or technical reports at face value. Be very wary of just online AI commentators where engagement is a big deal.

• 25:39Cal Newport?

  Be very wary of anyone who has connections back to rationalism, talking in these abstractions about AI. It's, it's tempting because it's easier to understand the underlying technical reality, but the technical reality matters. What did you build? Why are you building it? Why are you trying to? What else could you have done instead? These are the questions to keep asking these companies. We got to get into the weeds with people with no skin in the game or who don't think this is, you know, cosplaying their version of the matrix. We need to talk to people that understand the technologies. And when we do, what we see is something that is much less eerie than it is irresponsible. So these are powerful LLMs. If you put them in prompt loops, they're going to do weird, sometimes scary, often very damaging things, not because they're brilliant, but because that was a stupid idea. All right. So I thought I was done talking about this.

• 26:24Cal Newport?

  I hope I am now. I'm probably not, but I'll do my best to try to bring some sort of sanity to this otherwise, I think, very dramatic way that we're thinking about AI right now. All right. That's enough for now. I may do another one of these tech to coded episodes soon. I do it most weeks. Newsletter, calendupert.com has a lot of these tech to coded narratives as well. And until then, remember, care about AI, but not everything you read about.