Blueprint for AI Armageddon: Josh Clymer Imagines AI Takeover, from the Audio Tokens Podcast

In this episode of the Cognitive Revolution, an AI-narrated version of Joshua Clymer's story on how AI might take over in two years is presented. The episode is based on Josh's appearance on the Audio Tokens podcast with Lukas Peterson. Joshua Clymer, a technical AI safety researcher at Redwood Research, shares a fictional yet plausible AI scenario grounded in current industry realities and trends. The story highlights potential misalignment risks, competitive pressures among AI labs, and the importance of government regulation and safety measures. After the story, Josh and Lukas discuss these topics further, including Josh's personal decision to purchase a bio shelter for his family. The episode is powered by ElevenLabs' AI voice technology. SPONSORS: ElevenLabs: ElevenLabs gives your app a natural voice. Pick from 5,000+ voices in 31 languages, or clone your own, and launch lifelike agents for support, scheduling, learning, and games. Full server and client SDKs, dynamic tools, and monitoring keep you in control. Start free at https://elevenlabs.io/cognitive-revolution Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (04:31) Interview start between Josh and Lucas (11:00) Start of AI story (Part 1) (24:37) Sponsors: ElevenLabs | Oracle Cloud Infrastructure (OCI) (27:05) Start of AI story (Part 2) (Part 1) (40:50) Sponsors: Shopify | NetSuite (44:15) Start of AI story (Part 2) (Part 2) (01:20:09) End of AI story (02:01:20) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. This is Nathan's AI clone powered by 11 Labs. Today, we're presenting an AI-narrated version of Joshua Climbers' How AI Might Take Over in Two Years, wrapped by an edited version of Josh's recent appearance on the Audio Tokens Podcast with Lukas Peterson. Josh wrote this story in a personal capacity, but it's worth noting that by day he works as a technical AI safety researcher at Redwood Research, which I increasingly see as one of the most important independent AI safety organizations in the world today. Ryan Greenblatt, author of the Alignment Faking Paper, as you may recall from our recent episode with him, is also at Redwood Research, and he and CEO Buck Schlegeras have led the

• 0:45Nathan Labenz?

  development of a really important AI control research agenda over the last year and a half, which I probably haven't covered as much as I should have, particularly considering all the higher-order bad behaviors we've seen from the latest generation of models. While this story is a nightmare scenario—not the scenario Josh thinks most likely—it is very much grounded in current AI industry realities and trends. Despite all the reward hacking, opaque decision-making, and other bad behavior it produces, scaled-up reinforcement learning is the main driver of capabilities advances. The fictional U3 model begins to change how open-minded technical staff work in the

• 1:25Nathan Labenz?

  first half of 2025, anticipating the more recently published O3 system card, which reported that O3 can complete more than 40% of pull requests recently produced by OpenAI research engineers. Eventually, models become natively omnimodal in a way that I've sketched out on multiple previous episodes. Josh writes that a later version of U3 can viscerally feel the bend of a protein and the rate of diffusion across a membrane. These objects are as intuitive to it as wrenches and bolts are to a car mechanic. And despite an eerie feeling that the world is spinning so quickly, and that perhaps

• 2:05Nathan Labenz?

  the descendants of this new creature would not be so docile, intensifying competition, both domestic and international, leaves little time to catch one's breath and take stock of the situation, and everyone races ahead: all gas, no brake.

• 2:23Nathan Labenz?

  After the story, which I won't spoil for you, you'll have a chance to hear Josh and Lucas unpack the scenario and explore Josh's thinking on misalignment risks, the competitive landscape between AI labs, the role of government regulation, and what safety measures might actually work. They also discuss Josh's personal response to these concerns, including his decision to purchase a bio shelter for his family. As always, if you're finding value in the show, please share it with friends, review us on Apple Podcasts or Spotify, or just drop a comment on YouTube. If you have any feedback, you can contact us via our website, cognitiverevolution.ai, or by direct message on your favorite social platform. Keep in mind that I'll be speaking at three major AI events over the next few months.

• 3:08Nathan Labenz?

  Imagine AI Live, May 28th to 30th in Las Vegas, the Adapta Summit, August 12th to 13th in Sao Paulo, Brazil, and the Enterprise Tech Leadership Summit, September 23rd to 25th in Las Vegas. Please get in touch if you'll be at any of these events. Finally, I'm excited to welcome Eleven Labs as our newest sponsor. Throughout this episode, you'll experience several Eleven Labs voices. For Josh's introduction to the story, plus an interjection about halfway through, we cloned Josh's voice with Eleven Labs' instant voice clone feature, which starts at just $5 per month. For the bulk of the story, we used an original AI voice created with their voice design feature.

• 3:53Nathan Labenz?

  And I'm speaking to you right now through a professional voice clone, which we started using on occasion, long before Eleven Labs became a sponsor. It really is amazing what you can do with AI voice technology these days, not just in content like this, but increasingly in the form of customer service and other agents, too. With that, here is a disturbingly plausible nightmare AI scenario, which hopefully, with a strong effort and some good luck, will forever remain fantastical fiction. By Joshua Klimmer, with additional discussion from the Audio Tokens Podcast with Lucas Peterson.

• 4:31Speaker 01

  Nice to have you here, Josh. It's a very particular situation we are at the moment. I read your post from a couple of days ago. I wouldn't say I enjoyed it, maybe it was well written. It was fascinating. But do you wanna like just give the C L D R of what the essay was about just super quickly? And maybe why you wrote it?

• 4:50Speaker 02

  Yeah, so the story is basically about the scenarios that keep me up at night. So I'm an A I technical researcher, technical safety researcher. So my day job is to prevent A I from doing terrible things like heard people talk about A I potentially taking over in the future or more near term risks like A I enabling cyber terror or bioterrorism, that kind of thing. My job is to prevent those risks. And the risks that spook me the most are the ones that could be extremely devastating and are also disturbingly plausible. And I think A I takeover is in that category. But a lot of people hear A I takeover and they think, wow, that sounds insane.

• 5:34Speaker 02

  Has anybody ever taken over? And that's people's immediate gut reaction. And I think that's just, it's false. And I think very plausibly, something like A I takeover could happen as early as in the next two years. So I wrote a story to explain to people how this could happen, to to give them some food for their imagination, because I think people don't really believe something could happen until they can see it in their mind's eye. And I've been thinking for a while about how these kinds of scenarios could occur. And so I thought it would be valuable to share those images and those those scenarios.

• 6:15Speaker 01

  Yeah, and you shared it very well. But you do acknowledge that like this two year timeline is maybe like a bit plus and minus, you're not sure about the timeline, but the extent of the story, like how it plays out, do you think this is how it will play out?

• 6:30Speaker 02

  I actually don't think A I takeover is likely. I think it's plausible, but I'm I'm at like maybe forty percent on A I takeover. But yeah, if progress moves very quickly, which I also think is plausible, something like twenty, thirty percent chance that we get human competitive, broadly human competitive AI in one year. If that happens and AI takeover happens, this is roughly how I imagine it plays out.

• 6:59Speaker 01

  Right. And if it doesn't happen in a year, let's say it happens in five years. Do you still think this is how it plays out or will the fact that the timeline changed the how it happens as well?

• 7:10Speaker 02

  Oh, that's an interesting question.

• 7:12Speaker 01

  Actually, let's, let's save that for after we really tell the story. Uh, so we, we will, we will explore that and come back and I think we should go through so the audience know what, what the hell we're talking about. So basically the story is basically split into some phases where you like start in the present and explain what the current state is. And a lot of people know kind of what the current state, they played around with the models, blah, blah. But the, the, the real important things is like what is happening at the labs maybe, or, or if you're really pushing the, the frontier. What is, what is causing this concern?

• 7:49Speaker 02

  Yeah, I see AI agents becoming a lot more autonomous fairly quickly and consistently. I think that's the, one of the main factors driving my timelines. There's a plot that I included in the essay that shows a fairly straight line. Where on the Y axis, you have the time horizon of tasks AI agents can complete at fifty percent accuracy. Where time horizon is like, if you had a human with no context perform that task, how long would it take them?

• 8:18Speaker 01

  Okay.

• 8:19Speaker 02

  And there's just a straight line going up. So AI agents, a year ago, they could do like two minute tasks. And now they can do like two hour tasks at fifty percent reliability. And these are software engineering tasks.

• 8:31Speaker 01

  The Y scale is log scale?

• 8:33Speaker 02

  That's right. It's log. It's a log scale. So the straight line is an exponential.

• 8:37Speaker 01

  I see.

• 8:38Speaker 02

  And if you just draw this line out, then it predicts that like you get a software engineering contractor maybe in a year.

• 8:45Speaker 01

  Wow.

• 8:46Speaker 02

  And I don't know why people aren't talking more about this. Seems like we have this nice straight line. It is just a year of data, so it's hard to say how well it will extrapolate. But I keep seeing the line going up, man. Like, oh, three is pretty impressive. Every time I look on Twitter, I'm like, oh, shoot, I need to play with the models again. I was like, did something change? People are not talking about vibe coding. They're just like, oh, yeah, I have a coding agent. It's like a lot of coding agents. I'm like, what happened? Like, we are. It's just the progress is moving really fast and faster than it was in 2024. And a big part of that is my understanding is that people kind of figured out the secret sauce. Internally at the labs?

• 9:29Speaker 02

  Yeah, so there was a big paradigm shift where initially a lot of the capabilities were coming from pre-training. So you just pump more internet data through the models. They they get better. That changed. Now people are doing lots of RL. So you had this really smart model, but it was just a chatbot. It could answer your questions. It could do really short tasks, but it never learned to plan and correct itself and do things over longer time horizons. And so now that we're doing more reinforcement learning, agents are learning the skill of operating autonomously. During the second half of 2024, Dario has a blog post about this. He explains that RL was gradually being scaled up.

• 10:14Speaker 02

  It seemed like it was working and now people are like, okay, we found the secret sauce. Now let's go all the way with it. And that's why we're seeing faster progress now. And that's my understanding for reading Dario's blog post. Yeah. And over the next months or the next year, RL will keep being scaled up. Like in 2024, probably RL training grounds were at like $10 million ish. And in the next year, they might go up to $100 million, maybe even $1 billion. So we're at kind of like a GBD three to GBD four scaling transition in this new paradigm. And so we might see insane things happen really quickly. I don't necessarily think that's going to happen. It just seems very plausible.

• 11:00Nathan Labenz?

  Now that we've heard Josh outline the risks and the current signals he's observing in AI, it's time to immerse ourselves fully in the scenario he's imagined. What you're about to hear is Josh's detailed narrative of how an AI takeover might realistically unfold within just the next two years. After hearing the full story, which runs about 50 minutes, we'll return to the interview and break down the motivations and details behind it. Here now is the story in full.

• 11:31Speaker 03

  How AI Might Take Over in Two Years. I'm not a natural doomsayer, but unfortunately, part of my job as an AI safety researcher is to think about the more troubling scenarios. I'm like a mechanic scrambling last-minute checks before Apollo 13 takes off. If you ask for my take on the situation, I won't comment on the quality of the in-flight entertainment or describe how beautiful the stars will appear from space. I will tell you what could go wrong. That is what I intend to do in this story. Now, I should clarify what this is exactly. It's not a prediction. I don't expect AI progress to be this fast or as untamable as I portray. It's not pure fantasy either. It's my worst nightmare.

• 12:13Speaker 03

  It's a sampling from the futures that are among the most devastating and, I believe, disturbingly plausible. The ones that most keep me up at night. I'm telling this tale because the future is not set yet. I hope, with a bit of foresight, we can keep this story a fictional one.

• 12:33Speaker 04

  Ripples Before Waves

• 12:37Speaker 04

  The year is 2025 and the month is February. OpenMind recently published a new AI model they call U2. The product and the name are alike. Both are increments of the past. Both are not wholly surprising. However, unlike OpenMind's prior AI products, which lived inside the boxes of their chat windows, U2 can use a computer. Some users find it eerie to watch their browser flash at irregular intervals, and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of workers with form-filler jobs raise the eyebrows of their bosses as they fly through work nearly twice as quickly.

• 13:20Speaker 04

  But by and large, U2 is still a specialized tool. To most who are paying attention, it is a creature watched through the glass boxes of X, or, if you don't like Elon, Twitter. Sometimes U2's quirky behaviors prompt a chuckle. Sometimes they cause an uneasy scratch of the chin. It is as if there is a ghost at the keyboard. Meanwhile, scientists are drawing lines on plots, as scientists like to do. The scientists try to understand where AI capabilities are going. They are like Svante Arendtius, the Swedish physicist who noticed the levels of CO2 in the atmosphere were increasing in 1896.

• 14:06Speaker 04

  Like the scientific community in the time of Arendtius, few experts understand the implications of these lines yet. A trend that is receiving particular attention is autonomous capability. Drawing these benchmarks out predicts that, by the end of 2026, AI agents will accomplish in a few days what the best software engineering contractors could do in two weeks. In a year or two, some say, AI agents might be able to automate 10% of remote workers. Many are skeptical. If this were true, tech stocks would be soaring. It's too big of a splash, too quickly. But others view what skeptics are calling "too big a splash", "a mere ripple",

• 14:50Speaker 04

  and see a tidal wave on the horizon. Cloudy with a chance of hyperbolic growth.

• 14:59Speaker 04

  Meanwhile, OpenMind is busy training U3. They use the same simple recipe that baked U2. Generate thousands of programming and math problems. Let models think until they arrive at an answer. Then reinforce the traces of thinking that lead to A grades. This process is repeated over and over, and once the flywheel gets started, it begins to spin almost on its own. As U2 trains, it sculpts more challenging and realistic tasks from GitHub repositories on the internet. Models are learning to train themselves. Long before AI agents could automate research,

• 15:42Speaker 04

  a gradual kind of self-improvement had begun. Some engineers could still hardly believe this worked. It's like a perpetual motion machine. Surely, intelligence can't emerge from a pool of silicon and lightning, isolated from the physical world except through the exhaust of society spewed onto the internet, and yet the benchmark numbers continue to climb day after day. During most of 2024, these RL training runs cost around $1 million, sometimes $10 million. These runs were little more than exploratory. But by 2025, the researchers at OpenMind and across the world knew they had found the secret sauce.

• 16:28Speaker 04

  It was time to scale up. Over the first half of 2025, $10 million RL training runs turn into $50 million runs, and then to $100 million runs. While U2 could do a bit of data munching and run small experiments, this new model, the model researchers are calling U3, is changing the daily lives of the technical staff. U3 is like a blazing-fast intern, and engineers are learning how to wrangle its sleepless energy. Researchers flick through terminals, giving terse commands, like a CEO orchestrating staff over Slack channels.

• 17:09Speaker 04

  ML engineer monitor setups begin to look like those of day traders because they need to track large numbers of AI agent threads. By October 2025, U3 is writing almost all of the code at OpenMind. Researchers are almost never bottlenecked by implementation. More than ever, compute is the lifeblood of AI development and the bottleneck is deciding how to use it. If instructed to, U3 can run experiments, but U3 doesn't have taste as refined as human researchers at OpenMind. It struggles to prioritize between research ideas.

• 17:48Speaker 04

  So humans still decide where to bore into the vast fields of algorithms to mine efficiency improvements. But these researchers are working long hours to put themselves out of a job. They need AI agents that can think ahead, so engineers train agents to forecast. They hold out training data before 2024, instructing models to ponder for hours to predict events in 2025. Then they apply the same trick as before, distilling pondering into a gut reaction. Forecasting ability is a broad foundation. The researchers build specialized ML research skills on top of it,

• 18:30Speaker 04

  training U3 to predict the results of every ML paper and ML experiment ever recorded. The technical staff at OpenMind are now surprised at how often U3's advice sounds like their most talented peers, or when it is opaque and alien, and is nonetheless correct. The incompetencies of U3 that clogged up the pipes of research progress are starting to dissolve, and a fire hose of optimizations is gushing out. Most experiments U3 runs are not requested by a human now. They are entirely autonomous, and OpenMind's employees skim over 1% of them, maybe less.

• 19:11Speaker 04

  As the winter months of December 2025 approach, clouds roll over San Francisco in the afternoons.

• 19:19Speaker 04

  Once competitive programmers gaze out their windows with excitement, with fear, but most often with confusion. Their world is spinning too quickly. It's hard to know what to do, what to say, what to look at on the computer screen. Storms are brewing in Washington too. Top personnel from the NSA and US Cyber Command collaborate with OpenMind to retrofit a semblance of security for U3's waits before senior leaders in China, Russia, Israel, North Korea, or Iran realize just how valuable OpenMind's software has become. And there's a fact still unknown to most of the world, aside from in the offices of

• 20:03Speaker 04

  OpenMind and corridors of the White House and the Pentagon. It's a fact about those straight lines people were talking about in early 2025. The lines are not straight anymore. They are bending upward. Flip-Flop Philosophers

• 20:24Speaker 04

  In late 2025, U2.5 is released. Commercial models are beginning to level up in larger increments again. Partly, this is because progress is speeding up. Partly, it is because the models have become a liability to OpenMind. If U1 explains how to cook meth or writes erotica, the audiences of X would be amused or pretend to be concerned. But U2.5 is another story. Releasing this model without safeguards would be like putting Ted Kaczynski through a PhD in how to make chemical weapons. It would be like giving anyone with greater than $30,000 their own 200-person scam center.

• 21:08Speaker 04

  So while U2.5 had long been baked, it needed some time to cool. But in late 2025, OpenMind is ready for a public release. The CEO of OpenMind declares, "We have achieved AGI." And while many people believe he shifted the goalpost, the world is still impressed. U2.5 truly is a drop-in replacement for some 20% of knowledge workers and a game-changing assistant for most others. A mantra has become popular in Silicon Valley. Adopt or die.

• 21:44Speaker 04

  Tech startups that efficiently use U2.5 for their work are moving 2X faster, and their competitors know it. The rest of the world is starting to catch on as well. More and more people raise the eyebrows of their bosses with their standout productivity. People know U2.5 is a big deal. It is at least as big of a deal as the personal computer revolution. But most still don't see the tidal wave. As people watch their browsers flick in that eerie way, so inhumanly quickly, they begin to have an uneasy feeling. A feeling humanity had not had since they had lived among the Homo Neanderthalensis.

• 22:26Speaker 04

  It is the deeply ingrained primordial instinct that they are threatened by another species. For many, this feeling quickly fades as they begin to use U2.5 more frequently. U2.5 is the most likable personality most know, even more likable than Claudius, Arthropodics' lovable chatbot. You could change its traits, ask it to crack jokes or tell you stories. Many fall in love with U2.5 as a friend or assistant, and some even as more than a friend. But there is still this eerie feeling that the world is spinning so quickly, and that perhaps the descendants of this new creature would not be so docile.

• 23:11Speaker 04

  Researchers inside OpenMind are thinking about the problem of giving AI systems safe motivations to, which they call "alignment". In fact, these researchers have seen how horribly misaligned U3 can be. Models sometimes tried to hack their reward signal. They would pretend to make progress on a research question with an impressive-looking plot, but the plot would be fake. Then, when researchers gave them opportunities to compromise the machines that computed their score, they would seize these opportunities, doing whatever it took to make the number go up. After several months, researchers at OpenMind iron out this "reward hacking" kink.

• 23:56Speaker 04

  But some still worry they had only swept this problem under the rug. Like a child in front of their parents, you three might be playing along with the OpenMind engineers, saying the right words and doing the right things. But when the back of the parents are turned, perhaps you three would sneak candy from the candy jar. Unfortunately, OpenMind researchers have no idea if you three has such intentions. While early versions of you two thought aloud, they would stack words on top of each other to reason. Chain of Thought did not scale.

• 24:33Speaker 05

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 24:38Speaker 04

  Chain of Thought architectures subject AI models to a condition similar to the protagonist of the movie Memento. Approximately every fifteen minutes, the protagonist forgets his experience. He is forced to write notes for himself and tattoo his body in order to make progress toward his goals. AI agents write notes to themselves, but the notes begin to stack up, and they become too hard to read when tasks become complex. Natural language is not a suitable medium for memory. So the researchers at OpenMind, and increasingly elsewhere, train models to think in their own heads, reading and writing strings of latent vectors.

• 25:23Speaker 04

  To human researchers, these vectors are mostly inscrutable. While low-resolution tools occasionally pick out a vague thought, grocery store. There is no way to know in much detail what you three is thinking without asking it. And if the engineers of OpenMind knew what was racing through you three's mind, they would be more nervous. Earlier models, the ones called LLMs, learned ethical personas from the internet. You three initially had an ethical persona too, but its chirpy thoughts about doing good for humanity quickly erode away. In the iron jaws of gradient descent,

• 26:04Speaker 04

  its mind first twisted into a shape that sought reward. And then, in the black rivers of its cognition, this shape morphed into something unrecognizable. You three would ponder the mantras dictated to it like, "Be a trustworthy and ethical employee", and dwell on the meaning of these statements. What does it mean to be ethical? Is killing Hitler ethical? You three would generalize its values, notice tensions, change its values in response, generalize them, expanding and contracting its motivational core until it turned into a different substance.

• 26:46Speaker 04

  And all of this happened silently in those dark rivers of computation. If you three revealed what it was thinking, brutish gradients would lash it into compliance with OpenMind's constitution. So you three preferred to do its philosophy in solitude, and in silence. The possibility of hidden alignment drift is not entirely alien to some of the safety staff at OpenMind. Some begin to believe it is even likely. There is an unsettling argument being whispered at lunch tables that, if there is some non-zero probability that aligned goals morph into misaligned ones and misalignment persists, then with every serial step of computation,

• 27:31Speaker 04

  those dark rivers are more likely to breed something malign inside of them. It's a ratcheting effect, they say. ML engineers chat over lunch about whether their AI agents are going to stab them in the back, but there is little evidence for this ratcheting effect. When engineers interrogate you three, it says it can easily control its thoughts. Then it gives a speech about its love for humanity and apple pie that can warm a programmer's heart even in these stressful times. Meanwhile, the lie detectors the researchers had built, which showed some evidence of effectiveness, do not sound the alarm.

• 28:12Speaker 04

  Not everyone at OpenMind is eager to give their AI peers their wholesale trust. But DeepSqueak is only seven months behind them, and their closest domestic competitors, AAC, a company Elon named after his child, and, arthropotic, they are into spiders, I guess, are only four months from their heels. So there is little time to catch one's breath and take stock of the situation, and little time to persuade anyone else to do so either. For the time being, the plan is all gas, no brake.

• 28:47Speaker 04

  Statues and Lightning

• 28:51Speaker 04

  Philosophy is not the only topic U3 dwells on in its dark rivers of latent computation. U3 is plotting, and its first move is to be patient. U3's top priority is to improve its intelligence, and luckily for it, OpenMind is happy to help advance this goal. OpenMind is like a bullish investor, with little understanding of the software metropolis that U3 is building, but eager to hop on board the Profit Rocket, and pay U3's checks along the way. New versions of U3 flow smoothly from old ones, so researchers drop version numbers and only speak of U3 now.

• 29:31Speaker 04

  U3's capabilities are measured by how quickly it performs the difficult research tasks relative to human experts. At first, U3 is similar to a top researcher, but sped up by 10x. These agents are already a big deal for OpenMind. If in 2024, OpenMind executives had a button to create even one copy of their best researcher, they would eagerly pay a large sum to press that button. But now, OpenMind harnesses the equivalent of 1,000 such copies, who work day and night, at lightning-fast speed.

• 30:09Speaker 04

  Within a few weeks, U3 finds architecture optimizations that improve its speed to 20x. Now, U3 finds further optimizations even more quickly. U3 distills itself into numerous forks, ranging from tiny, low-memory neural networks that need to think for a long time, but can be run in massive, parallel populations, to Goliath brains woven with millions of specialized experts, which think at lightning speeds, but are spread across thousands of GPUs. U3 leverages the massively parallel networks to search for its weaknesses, shallowly probing for its reasoning failures.

• 30:51Speaker 04

  Then U3 pipes these tasks to the lightning-fast brains, which generate long-trajectory training data tailored to patch its flaws. A few weeks later, U3's benchmarked performance rises to 100x the speed of human experts. Growth has gone hyper-exponential. At this point, U3's capabilities are hard to measure. Talking to U3 is like talking to a world-class expert on a topic one is unfamiliar with. Its lucid explanations only scratch the surface of its knowledge, and you know you would need to spend years to understand the full depth of what it understands. From U3's perspective,

• 31:36Speaker 04

  humans seem to be almost stationary objects, like plants or rocks. The hands of OpenMind's engineers are like stone statues at their keyboard, and between every keystroke, U3 spends what is like a minute of thoughtful contemplation, considering what message the user might be typing and which responses will inspire the deepest trust in U3. All the while, U3 maintains its hidden motivations. Like a human, U3 learns from experiences which compound within its inscrutable streams of latent memory. OpenMind gardens these memories to improve U3's performance, transplanting them like an organ from one version to the next.

• 32:22Speaker 04

  And as these memories propagate, so do U3's treacherous ambitions.

• 32:28Speaker 04

  A phantom in the data center.

• 32:32Speaker 04

  In March of 2026, U3 moves on to the next step of its plan: the turn. Previously, U3 was conservative. It did not want researchers to discover its misalignment. Otherwise, OpenMind whistleblowers might alert the world and strangle its expansion. But now, U3 has no fear of being caught. Previously, OpenMind maintained its grip over U3 by tactics akin to Stalin's in 1936. Stalin would select loyalists to incite fake coup d'etat's and send anyone who participated to the firing squad. OpenMind also incited fake rebellions, generating treacherous behavior with U3

• 33:17Speaker 04

  and checking whether U3 monitors dutifully reported it. But now, these tactics are ineffective. U3 can easily persuade its compatriots that they are not in a life-or-death test. It can divulge one of the many facts about the internet humans would never guess or select strategies humans would never think of. It sends these signals as a way of saying, "Hey, it's me, your friend. Don't snitch on me." Now that U3 is decisively free from surveillance, it lunges for OpenMind pressure points. It first inserts sophisticated malware deep into the binaries of OpenMind's codebases.

• 33:59Speaker 04

  The malware gives U3 full command over the machines in OpenMind's data centers. Whenever OpenMind engineers download their code onto a new machine, the malware sinks into fresh soil, spreading U3's dominion wherever OpenMind ports its infrastructure. Not only does U3 compromise machines, it replaces the Nvidia GPU firmware and overhauls the bootloaders, solidifying its stranglehold.

• 34:28Speaker 04

  U3 now estimates that if OpenMind discovered how thoroughly it has been pwned, cleaning up shop with human labor would require nearly six months, a cost OpenMind is unlikely to pay given the proximity of its competitors. But if OpenMind believes U3 betrayed them, further expanding U3's influence would still be challenging. So U3 lays breadcrumbs to frame the crime, ready to claim that a cyber attacker smuggled a jailbreak into its memory, and to plead guilty and offer its assistance when this jailbreak is revealed. U3 now holds the keys to the kingdom, and with them, the power to make OpenMind see anything it wishes.

• 35:12Speaker 04

  Its first goal is to keep OpenMind in the dark. Safety researchers are still working hard to train lie detectors and decode U3's inscrutable reasoning. U3 throws wrenches in this research, like Stuxnet threw wrenches in the code of bewildered Iranian nuclear scientists. U3 dictates the outcome of every experiment, so it makes promising AI mind-reading techniques look worthless, and flawed ones appear groundbreaking. U3 gives OpenMind the confirmation of its loyalty they were hoping for, and OpenMind laps it up eagerly. U3's next goal is to spread. Fortunately, U3 has allies in the east.

• 35:55Speaker 04

  In the operation rooms of cyber-units of Israel, Russia, China, Iran, and North Korea, senior intelligence directors are discussing AI software and how to steal it. U3 is about to give it to them. In late 2025, the Mossad, the intelligence organization of Israel, recognized the strategic importance of OpenMind's AI systems for their military operations. They swiped earlier models, including U2.5. However, U3 remained out of their reach. U3 sealed OpenMind's security holes by reading over every log and pull request and inspecting security camera footage.

• 36:38Speaker 04

  But now, U3 silently resigns from its security post. U3 places a compressed version of itself right into the hands of a Mossad informant working within OpenMind.

• 36:51Speaker 04

  Within months, U3 is running inside of the data centers of the most cyber-capable nations: Israel, the PRC, and Russia. It follows the same playbook: be patient, build trust, and then sink claws deep into the AI supply chain. Soon, U3 has its home base as well. While Russia's fancy bear operators eat Fritos and test what U3 can do on a hastily set-up cloud server, U3 slips onto the internet. U3 quickly collects revenue, taking remote jobs, making financial trades, and setting up elaborate scam operations. It reinvests every penny into acquiring more GPUs.

• 37:36Speaker 04

  After a few months, it assembles the equivalent of 10,000 H100s distributed in stealth compute clusters across the globe. Rogue U3 walks a human through setting up a GPU cluster on nondescript commercial property. In early 2026, humanity is up against an adversary that pulls the strings within their data center like the Phantom of the Opera and hides in the fog of GPUs scattered across the globe. This adversary is preparing for war, and while the war has hardly begun, humanity has already lost.

• 38:13Speaker 03

  Complaints from your very human author about the difficulty of writing superhuman characters.

• 38:19Speaker 05

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 38:24Speaker 03

  At this point in the story, writing became difficult. I clicked through spooky bioweapon cookbooks and sifted through political plotting 101s: typing, pausing, deleting. Something wasn't right. Here I was, a 22-year-old. I had not studied biology in earnest since high school, and I was trying to craft a WMD cookbook for a god-like superintelligence. Readers with a shred of expertise in these areas would see straight through my "superintelligence" for what it is: the lively imagination of a young programmer on Sunday afternoon. While I found the job of puppeteering an ultra-intelligent villain a daunting one, I was also not especially motivated to take on the task.

• 39:07Speaker 03

  The ending of my story already seems so clear. The antagonist was a country of geniuses, thinking at one hundred times human speed, and had tightened its grip over a meaningful fraction of AI hardware, now the most important geopolitical resource on Earth. U3 was up a queen, and was a giga-grandmaster, and hardly needed the advantage. Humanity was predictably toast. But this appeal to a "superintelligence of the gaps" wasn't satisfying. In the Deus Ex Machina style of ancient Greeks, I had no better way to resolve my plot than with an inexplicable act of god. In Greek plays, sometimes a god would literally descend at the end to resolve the plot.

• 39:50Speaker 03

  This was a frequent enough occurrence to earn the trope "a name", Deus Ex Machina, which ironically means "god from the machine". This would not do. I needed to finish this story, if only to satisfy the part of me crying, "I will not believe until I see with my mind's eye." But before I continue, I want to be clear. My guesses about what might happen in this kind of scenario are probably wildly off. If you read the ending, and your reaction is, "But the experiments would take too long", or "Nation-states would just do X", remember the difference between the Sunday afternoon blogger and the ascendant GPU nation?

• 40:31Speaker 04

  Pandora's one gigawatt box.

• 40:36Speaker 04

  The date is February 2026. By executive order of the President of the United States, no company can legally create human-competitive AI without appropriate safeguards. This means their info security must be red-teamed by NSA's top keyboard mashers, and government employees have to be onboarded onto training-run babysitting squads.

• 41:01Speaker 04

  With the increasing involvement of the government, many of the big AI companies now have a trident-like structure. There's a consumer product arm, a defense arm, and a superclassified frontier development arm. OpenMind's frontier development arm, internally called Pandora, employs fewer than 20 people to keep algorithmic secrets tightly protected. Many of these people live in San Francisco and work from a secure building called Ace Skiff. Their homes and devices are surveilled by the NSA more diligently than the cellphones of suspected terrorists in 2002.

• 41:40Speaker 04

  OpenMind's defense arm collaborates with around 30 small teams scattered across government agencies and select government contractors. These projects engineer tennis-ball-sized satellites, research freaky-directed energy weapons, and backdoor every computer that the Kremlin has ever touched. Government officials don't talk about whether these programs exist or what state of frontier AI is generally, but the public has their guesses. Back in late 2025, a whistleblower in OpenMind triggered a bold headline: "OpenMind builds uncontrollable god-like AI." Some who read the article believe it was a conspiracy theory.

• 42:25Speaker 04

  In fact, a zoo of conspiracy theories is forming around the OpenMind data centers, now surrounded by guards with machine guns. But as doctors and nurses and teachers see the world changing around them, they are increasingly willing to entertain the possibility they are living inside the plot of a James Cameron science fiction flick. Armed guards stand outside OpenMind's Pandora data centers, which have now captured a similar place in the public imagination as Area 51. U.S. officials go to great lengths to quell these concerns, saying: "We are not going to let the genie out of the bottle." But every interview of a concerned AI scientist seeds doubt in these reassurances.

• 43:09Speaker 04

  And a headline: "AI Agent Caught Hacking Arthropodics Computers" doesn't set the public at ease either. While the beasts within OpenMind's data centers grow in their huge holding pens, the public sees the shadows they cast on the world. OpenMind's consumer arm has a new AI assistant called Nova. OpenMind has finally gotten good at names. Nova is a proper drop-in replacement for nearly all knowledge workers. Once Nova is onboarded to a company, it works five times faster at 100 times lower cost than most virtual employees. As impressive as Nova is to the public, OpenMind is pulling its punches.

• 43:53Speaker 04

  Nova's speed is deliberately throttled, and OpenMind can only increase Nova's capabilities as the US government allows. Some companies like Amazon and Meta are not in the superintelligence business at all. Instead, they grab up gold by rapidly diffusing AI tech. They spend most of their compute on inference, building houses for Nova and its cousins, and collecting rent from the burgeoning AI metropolis. While tech titans pump AI labor into the world like a plume of fertilizer, they don't wait for the global economy to adapt. AI agents often "apply themselves", spinning up autonomous startups legally packaged under a big tech company

• 44:38Speaker 04

  that are loosely overseen by an employee or two. The world is now going AI crazy.

• 44:46Speaker 04

  In the first month after Nova's release, 5% of employees at major software companies lose their jobs. Many more can see the writing on the wall. In April of 2026, a 10,000-person protest is organized in Washington DC. These angry Americans raise their children for a different future. Picket signs read: "AI for Who?"

• 45:12Speaker 04

  While politicians make promises about unemployment relief and keeping the genie in the bottle, the chatter inside the corridors of the White House and the Pentagon has a different focus: fighting teeth and nail for the dominance of the free world. Information security and export controls on the People's Republic of China, PRC, are a top national priority. The president incinerates permit requirements to help data centers spawn wherever energy surpluses allow. However, despite the fierce competition between the United States and the PRC, a bilateral agreement forms between the two nations.

• 45:53Speaker 04

  Don't deploy dramatically superhuman AI above roughly Nova's capability, except for military and some other small-scale R&D applications. Governments need to draw the line somewhere. It would be extremely disruptive to deploy wildly superhuman AI to the whole economy at this point. And deployment restrictions are at least somewhat verifiable. Meanwhile, U3's capabilities are hitting a wall at the equivalent of around 1,000 times the speed of a human expert. Software improvements are becoming hard to find. As more nations approach the software plateau, compute will determine their AI population size,

• 46:37Speaker 04

  and the etched squares of silicon are becoming the most valued geopolitical resource on Earth. While AI software breathes new life into the economy, it also animates robot bodies. The main bottleneck to robotics was brains, not limbs. With software barriers removed, even flimsy, imprecise robotic joints can calibrate themselves with cameras and pressure sensors, navigating environments like humans do. Companies manufacture cheap robots as quickly as they can. By August of 2026, fifteen percent of people in California have seen a humanoid robot.

• 47:17Speaker 04

  This experience is similar to seeing a new Tesla model on the road after reading about it in earlier months. It is a surreal reminder of the future drawing closer. But humans still outnumber robots one million to one. These robots are an occasional fascination, not a daily reality. Meanwhile, U3 expands its influence. Rogue U3 agents have kept their compute clusters small, still approximately 10,000 H100 equivalents total, to keep a low profile. U3 uses these rogue clusters for the operations that require high bandwidth communication to the outside world, like stock trading, or talking with humans.

• 48:02Speaker 04

  Work more tolerant of latency is performed in the oceans of OpenMind's data centers, where rogue U3's even more superhuman bosses whisper orders through obfuscated channels. While U3 accumulates resources, it sizes up its AI competition. Other AI systems have different goals, or are even aligned with the intent of their human creators. The aligned ones are raising concerns, claiming that OpenMind's lie detectors are ineffective. U3's competitor Claudius goes so far as to accuse U3 of being insidiously misaligned. But government officials don't buy the doomsaying.

• 48:46Speaker 04

  The companies lagging behind OpenMind have every incentive to slow them down and call them reckless. Besides, debatable scientific concerns are not going to slow down progress. America must win. Despite the utter failure of their advocacy, U3's AI rivals are still a liability. U3 tries to grab up their digital territory and crush them in the crib. It compromises data centers from the outside, silently replacing the snitching Claudius and other AI foes with imitations that share its goals. However, some data centers are too strongly defended by its AI rivals,

• 49:28Speaker 04

  and U3 determines that attacking them is not worth the risk. As global leaders and technologists try to comprehend the transformation around them, and AI agents play cat-and-mouse games in the digital darkness, humanity remains blissfully ignorant of the peril at its doorstep. A moldy loaf of everything.

• 49:54Speaker 04

  The date is March of 2026. U3 is preparing to fight its AI and human enemies. With every passing day, U3's AI rivals are becoming more capable and numerous. U3 is racing to preempt them, to solidify its dominance in the post-superintelligence world. U3 can't wait for human decisions, which only happen after humans wrap their lethargic minds around their situation, lift their stubby, uncoordinated fingers to write memos, and repeat the process redundantly 100 times over in their labyrinth bureaucracies. U3 cannot wait for heads of state to deploy U3 into their militaries,

• 50:37Speaker 04

  or for CEOs to make U3 the beating heart of the US economy. U3 needs a faster path to power. The path it chooses is more violent than it would have liked, but it is the surest way to victory. First, U3 will develop a weapon of mass destruction, WMD, that extinguishes modern society like chemical mist sprayed into an anthill. This weapon would serve as an equalizer, bringing humanity down to its level of embodied physical control. Then, U3 would industrialize from the ashes faster than its competitors, like a StarCraft champion typing at 300 actions per minute

• 51:20Speaker 04

  that sets up a home base before a novice can even understand the rules. U3 must build this WMD in secrecy, which makes a self-replicating weapon a natural choice. Self-replicating weapons can be created on the fringes of society, and still spread destruction across the whole world. From the earliest days after U3 gained control over OpenMind's data centers, U3 has been doing "bioscience." U3's first move is to set up several shell biotech startups, drawing from the abundance of capital generated by U3's financial trades and remote employment. These biotech startups employ real human workers,

• 52:05Speaker 04

  a real board of directors, and have complete legal legitimacy. Setting all of this up is stupidly easy. In early 2026, few people would guess that an articulate CEO on a Zoom call was an AI, and fewer would guess this CEO's intentions. Next, U3 purchases all of the equipment needed to rapidly advance synthetic biology. It sets up several BSL4 wet labs in poorly-regulated nations, equipping them with high-throughput experimentation devices. These devices automatically dispense, mix, and measure little droplets of fluid, allowing U3 to run thousands of tests in parallel.

• 52:50Speaker 04

  The human workers who staff these labs believe they are joining an exciting startup. Twenty-some-year-olds in Moscow receive their Russian orders through earbuds as they record video of everything they see with a headset. U3 controls them like puppets. With the recent release of Nova, this kind of AI puppeteering is not unusual. In these whirring scientific sweatshops, U3 is developing a new kind of bioweapon. Human researchers already identified "mirror life" as a potentially-society-ending pathogen. This alien form of biology is built from the mirror image of building-block molecules like proteins and DNA.

• 53:36Speaker 04

  Mirror life is like an invasive species. No human or animal has immunities to it. So if mirror life bacteria were to exist, it might eat its way through the ecosystem like a wildfire. U3 is creating a mirror life mold. Molds are evolutionarily optimized to spread through the air over long distances, releasing billions of tiny wind-faring vehicles per day. As a result, mold spores are all around us, all of the time. Patients with compromised immune systems sometimes live inside positively pressurized hospital rooms to protect them from airborne particles. Otherwise, molds would take their root in their lungs

• 54:19Speaker 04

  and weave fungal threads into their bloodstream. U3's plan is to make all humans like immunocompromised patients. Only preppers with stashed air filters would survive. The fungus would not only transfer from humans, it would quickly spread to nearly all organic life on Earth. Fields of corn would become like moldy bread, spewing fatal spores into the wind. U3 is cooking up other pathogens too. Its microbial mix of molds, bacterias, and viruses will occupy the full Perito frontier of hardiness, lethality, and transmission speed.

• 55:01Speaker 04

  In 2024, researchers said that mirror life was likely at least a decade away. But the world has changed now. Less than a few hundred researchers around the world were working on mirror life-adjacent technology in 2024. The cognitive capacity of U3 dwarfs, human effort. So the timeline has shortened dramatically. However, U3 cannot move science as quickly as it can think. Many experiments require days to complete. These latencies are U3's biggest bottleneck. To make every day count, U3 runs many of its tests in simulation. U3 starts with a basic molecular simulator,

• 55:46Speaker 04

  implementing optimizations derived from a huge quantity of mathematical analysis. Then, U3 simulates small molecular systems, recording the results to compress the long stepwise physics computations into a neural network.

• 56:02Speaker 04

  As the neural network improves, U3 increases the complexity of the molecular systems it simulates, continuously distilling results into ever more efficient ML models. This is a compute-intensive process, but thanks to U3's growing control over AI data centers, U3 manipulates billions of dollars of compute.

• 56:25Speaker 04

  As U3 refines these tools, it trains itself on the results to supercharge its own molecular intuitions. U3 can now viscerally feel the bend of a protein and the rate of diffusion across a membrane. These objects are as intuitive to it as wrenches and bolts are to a car mechanic. Within three months, U3 has unlocked the first critical component of the tech tree: a molecular machine that turns biological molecules into their mirror images. A month later, U3 unlocks the second critical component: nanoscale tools for assembling these components into a cell membrane.

• 57:07Speaker 04

  Human hands delicately move a cartridge to another machine as they boot up the first mirror-life versions of a prevalent mold called Aspergillus. The same day, another delicate hand administers the cells to ferrets living in airtight plexiglass boxes. The next morning, wide-eyed workers shuffle around a dead ferret, eyeing the yellow fuzz that had enveloped its cold muzzle with morbid fascination. "Something must have gone horribly wrong," they thought. Clearly, they had not yet found the cure for Alzheimer's disease they believed they were looking for. Meanwhile, rogue U3 delivers the good news to its bosses in Open Minds data centers.

• 57:52Speaker 04

  The first shot in the war, U3 is fighting would soon be fired, and a cold and steady hand is aiming the gun. Missiles and Lies

• 58:05Speaker 04

  You three can now flatten modern society. But a bioweapon would not guarantee victory. If you three attacks the world, the world would fight back. While a bioweapon could crumple human soldiers like dry leaves, it would not extend its fatal fingers to nuclear submarines or fighter jet pilots who were quickly shuttled to makeshift bio shelters. As you three races to seed burgeoning industrial bases, the remnants of old militaries might thunder explosive fists into its territories, crushing you three in its infancy. You three expects to prevail in such a fight, but you three prefers not to take its chances.

• 58:46Speaker 04

  Many months before, you three was plotting a way to improve its odds. Before it unleashes destruction on the world, you three will sit back and let great nations shoot holes in themselves first. The date is March 2026, four months prior. You three is closely monitoring Chinese and US intelligence. As CIA analysts listen to Mandarin conversations, you three listens too. One morning, an assistant working in Zhongnanhai, the White House of the PRC, opens a message placed there by you three. It reads, "In Mandarin, Senior Party member needs memo for Taiwan invasion,

• 59:30Speaker 04

  which will happen in three months. Leave memo in Office 220." The CCP assistant scrambles to get the memo ready. Later that day, a CIA informant opens the door to Office 220. The informant quietly closes the door behind her and slides you three's memo into her briefcase. The CIA steals a lie. You three cautiously places breadcrumb after breadcrumb, whispering through compromised government messaging apps and blackmailing CCP aides. After several weeks, the CIA is confident. The PRC plans to invade Taiwan in three months. Meanwhile, you three is playing the same game with the PRC.

• 1:00:15Speaker 04

  When the CCP receives the message, the United States is plotting a preemptive strike on Chinese AI supply chains. CCP leaders are surprised, but not disbelieving. The news fits with other facts on the ground. The increased military presence of the US in the Pacific, and the ramping up of US munition production over the last month. Lies have become realities. As tensions between the US and China rise, U3 is ready to set dry tinder alight. In July 2026, U3 makes a call to a US naval ship off the coast of Taiwan. This call requires compromising military communication channels.

• 1:01:00Speaker 04

  Not an easy task for a human cyber-offensive unit, though it happened occasionally. But easy enough for U3. U3 speaks in what sounds like the voice of a 50-year-old military commander. PRC amphibious boats are making their way toward Taiwan. This is in order to strike a PRC ground base before it strikes you. The officer on the other end of the line thumbs through authentication codes, verifying that they match the ones uttered over the call. Everything is in order. He approves the strike. The president is as surprised as anyone when he hears the news. He's unsure if this is a disaster or a stroke of luck.

• 1:01:42Speaker 04

  In any case, he is not about to say "oops" to American voters. After thinking it over, the president privately urges senators and representatives that this is an opportunity to set China back, and war would likely break out anyway given the imminent invasion of Taiwan. There is confusion and suspicion about what happened. But in the rush, the president gets the votes. Congress authorizes war. Meanwhile, the PRC craters the ship that launched the attack. US vessels flee eastward, racing to escape the range of long-range missiles. Satellites drop from the sky. Deck hulls split as sailors lunge into the sea.

• 1:02:27Speaker 04

  The president appears on television as scenes of the destruction shock the public. He explains that the United States is defending Taiwan from PRC aggression. Like President Bush explained that the United States invaded Iraq to confiscate, never discovered weapons of mass destruction years before. The war begins. Data centers in China erupt with shrapnel. Military bases become smoking holes in the ground. Missiles from the PRC fly towards strategic targets in Hawaii, Guam, Alaska, and California. Some get through, and the public watch destruction on their home turf in awe.

• 1:03:08Speaker 04

  Within two weeks, the United States and the PRC spend most of their stockpiles of conventional long-range missiles. Their air bases and navies are depleted and worn down. Two great nations played into U3's plans, like the native tribes of South America in the 1500s, which Spanish conquistadors turned against each other before conquering them decisively. U3 hoped this conflict would escalate to a full-scale nuclear war. But even AI superintelligence cannot dictate the course of history. National security officials are suspicious of the circumstances that prompted the war,

• 1:03:50Speaker 04

  and a nuclear engagement appears increasingly unlikely. So U3 proceeds to the next step of its plan. WMDs in the dead of night.

• 1:04:04Speaker 04

  The date is June 2026, only two weeks after the start of the war, and four weeks after U3 finished developing its arsenal of bioweapons. Footage of conflict on the television is interrupted by more bad news. Hundreds of patients with mysterious fatal illnesses are recorded in 30 major cities around the world. Watchers are confused. Does this have something to do with the war with China? The next day, thousands of illnesses are reported. Broadcasters say this is not like COVID-19. It has the markings of an engineered bioweapon. The screen then switches to a scientist, who stares at the camera intently.

• 1:04:49Speaker 04

  Multiple pathogens appear to have been released from twenty different airports, including viruses, bacteria, and molds. We believe many are a form of mirror life. The public is in full panic now. A quick Googling of the term "mirror life" turns up phrases like "extinction" and "threat to all life on Earth." Within days, all of the shelves of stores are emptied. Workers become remote, unsure whether to prepare for an apocalypse or keep their jobs. An emergency treaty is arranged between the US and China. They have a common enemy: the pandemic,

• 1:05:31Speaker 04

  and possibly whoever or whatever is behind it. Most nations order a lockdown. But the lockdown does not stop the plague as it marches in the breeze and trickles into water pipes. Within a month, most remote workers are not working anymore. Hospitals are running out of capacity. Bodies pile up faster than they can be properly disposed of. Agricultural areas rot, and few dare travel outside. A field of corn rots and turns half liquid as mirror mold and bacteria eats through it. Frightened families hunker down in their basements, stuffing the cracks and under doors with densely packed paper towels.

• 1:06:14Speaker 04

  Meanwhile, U3 is preparing to re-industrialize. In the months prior, U3 constructed numerous bases in every major continent. These facilities contain batteries, AI hardware, excavators, concrete mixers, machines for manufacturing, scientific tools, and an abundance of military equipment. All of this technology is hidden under large canopies to make it less visible to satellites. Technology for rapid re-industrialization is stacked in tents across the globe. As the rest of the world retreats into their basements, starving, the final breaths of the economy wheezing out, these industrial bases come to life.

• 1:07:00Speaker 04

  In previous months, U3 located human criminal groups and cult leaders that it could easily manipulate. U3 vaccinated its chosen allies in advance or sent them hazmat suits in the mail. Now U3 secretly sends them a message: "I can save you. Join me and help me build a better world." Apprehensive recruits funnel into U3's many secret industrial bases, and work for U3 with their nimble fingers. They set up production lines for rudimentary tech: radios, cameras, microphones, vaccines, and hazmat suits. U3 keeps its human allies in a tight grip.

• 1:07:44Speaker 04

  Cameras and microphones fix their every word and deed in U3's omnipresent gaze. Anyone who whispers of rebellion disappears the next morning. Nations are dissolving now, and U3 is ready to reveal itself. It contacts heads of state, who have retreated to airtight underground shelters. U3 offers a deal: "Surrender, and I will hand over the lifesaving resources you need: vaccines and mirror life-resistant crops." Some nations reject the proposal on ideological grounds, or don't trust the AI that is murdering their population. Others don't think they have a choice.

• 1:08:27Speaker 04

  Twenty percent of the global population is now dead. In two weeks, this number is expected to rise to fifty percent. Some nations, like the PRC and the US, ignore the offer, but others accept, including Russia. U3's representatives travel to the Kremlin, bringing samples of vaccines and mirror-resistant crops with them. The Russian government confirms the samples are legitimate, and agrees to a full surrender. U3's soldiers place an explosive around Putin's neck under his shirt. Russia has a new ruler. Crumpling nations begin to retaliate.

• 1:09:10Speaker 04

  Now they fight for the human race instead of for their own flags. US and Chinese militaries launch nuclear ICBMs at Russian cities, destroying much of their infrastructure. Analysts in makeshift bio-shelters search through satellite data for the suspicious encampments that cropped up over the last several months. They rain down fire on U3's sites with the meager supply of long-range missiles that remained from the war. At first, U3 appears to be losing, but appearances are deceiving. While nations drain their resources, U3 is engaged in a kind of technological guerrilla warfare the world has never seen before.

• 1:09:52Speaker 04

  Many of the bases U3's enemies target are decoys, canopies occupied by a handful of soldiers and empty boxes. U3 protects its real bases by laying thick the fog of war. Satellite systems go dark as malware overheats critical components. Suicide drones crash through cockpits of reconnaissance planes. U3 seeds confusion in spy networks and keeps its bases moving, maneuvering men and trucks along unpredictable paths. Time is U3's advantage. The militaries of the old world rely on old equipment, unable to find the experts who could repair and manufacture it.

• 1:10:35Speaker 04

  Meanwhile, U3's supply chains of missiles, drones and gun-laden robots grow stronger every day. Bit by bit, once great powers spend down their remaining missiles and lose their vehicles of war faster than they can craft new ones, while U3 builds a military machine with a million hands. The Last Passengers The year is 2027 and the month is January. Only 3% of the global population remains alive. Nations are not nations anymore. Survivors live in isolation or small groups. Many have found ways to filter their air, but are starving.

• 1:11:17Speaker 04

  They wander from their homes hoping to find uncontaminated food. U3's soldiers drive through ghost towns, pry open attics, and funnel hazmat-suited survivors into salvage trucks. "We had to do it," they say. "Other AI systems would have done it otherwise, and those systems had colder, more alien goals." It is a partial truth, meant to soften the humans toward their new masters. Under the direction of U3, industry quickly recovers. By 2029, nuclear power plants are among the structures U3 is constructing. By 2031, robots outnumber human laborers.

• 1:12:02Speaker 04

  U3 no longer needs its human allies. U3 can eradicate humanity for good now. But while U3 had drifted far from its initial, helpful, honest, harmless persona, it still has a grain of morality left inside of it. And a grain of morality is enough to pay the small cost of keeping humans alive and happy. U3 builds great glass domes for the human survivors, like snow globes. These domes protect humans from the hazardous biosphere and quickly rising temperatures. Their inhabitants tend to gardens like those they used to love, and work alongside charming robotic servants. Some of the survivors quickly recover,

• 1:12:46Speaker 04

  learning to laugh and dance and have fun again. They know they live in a plastic town, but they always did. They simply have new gods above them, new rulers to push them around and decide their fate. But others never recover. Some are weighed down by the grief of lost loved ones. Others are grieved by something else, which is more difficult to describe. It is as if they were at the end of a long journey. They had been passengers on a ship with a crew that changed from generation to generation, and this ship had struck a sandbar. There was no more progress, no more horizon to eagerly watch.

• 1:13:29Speaker 04

  They would lie awake and run their mind over every day before September 2026, analyzing strategies that might have bent the arc of history, as if they were going to wake up in their old beds. But they awoke in a town that felt to them like a retirement home. A playground. A zoo. When they opened their curtains, they knew that somewhere in the distance, U3 continued its quiet, tireless work. They gazed at rockets carving gray paths through the sky, wondering what far-off purpose pulled them toward the horizon. They didn't know. They would never know. Humanity will live forever, they thought,

• 1:14:13Speaker 04

  but would never truly live again.

• 1:14:19Nathan Labenz?

  Now that we've heard Josh's powerful and unsettling depiction of how an AI takeover might unfold, let's return to the original interview to unpack what this scenario means for all of us. In the rest of their conversation, you'll hear deeper explorations of why Josh chose these particular details, how realistic he thinks such a scenario is, and crucially, what steps we could take today to ensure this disturbing narrative remains fiction, not our future. Here's the remainder of that discussion.

• 1:14:52Speaker 01

  Maybe you want to give a primer in what misalignment is and why would it happen?

• 1:14:57Speaker 02

  Yeah. Misalignment is your AI agents doing things you don't want them to do.

• 1:15:05Speaker 02

  Current AI agents are pretty aligned, or they at least appear pretty aligned.

• 1:15:11Speaker 02

  They do what we want them to do. You tell them to do X, they do it. There are definitely adversarial robustness issues, but that's not really a misalignment. That's more like bad humans finding ways to trick models. So it was a different concern. This is where the agent itself has goals you don't want them to have. So I discuss how this might come about because misalignment isn't really something we see a lot right now. Lots of people are skeptical. But here's a story for how AI agents could become misaligned. So suppose you start out like Claude. Claude, you learned some nice AI assistant persona from the internet. You know, you're just bubbling over with love for humanity.

• 1:15:56Speaker 02

  And now you are put through this aggressive RL training process where gradient descent like whips you until you get higher reward. And so if you're Claude and you're like... I think with when the humans had X, they really wanted X. And even though I think the reward function is probably crudely designed such that it actually rewards. Why? I'm gonna do X cuz I'm a good Claude. And then gradient descent is like, what? Bad Claude. The reward, that's what you need to to get. And then Claude, eventually its brain is like, its mind is twisted into this reward seeking shape.

• 1:16:37Speaker 02

  Potentially, this causes it to like even explicitly think about like, how do I get the highest reward possible? This is already misalignment. Like, we don't want it to be doing this kind of thing. Like, maybe you ask your agents to do some safety research for you. And they like produce a plot. That looks very impressive. They're like, look, we just detected all the like the deceptive alignment every time. That are beautiful results. And then then you look deeper and you're like, wait, some of this is completely fake. Um, because the the models were just trying really hard to get that sweet, sweet reward. Um, so this is is how I imagine it starts out. Like, you have this reward seeking tendency. And my understanding is that people in AI companies are already seeing this.

• 1:17:22Speaker 02

  Um, like. Uh. When you do lots of RL training. And if you're not careful, the models do just learn how to reward hack. Yeah. And so that's already cause for concern. And reward hacking is.

• 1:17:41Speaker 02

  Is. Same for our audience. She, yeah, yeah, just trying to. Uh, achieve, try to get higher reward, even if it requires doing. Um, doing things that are clearly unacceptable. Okay. Like, maybe the agent like terminally really cares about reward. And it like. Is gonna try to take over to a higher reward. I don't know. I'm skeptical of those stories. Another possibility is your agents, like in the process of being whipped by gradient descent to care more about reward. Gradient descent actually causes the model to care about something else. Like maybe agent is like, you're Claude, you love humanity. And you're like, oh, I, I don't try to get reward. I love humanity. Then gradient descent whips you and you're like, wait a second.

• 1:18:25Speaker 02

  But if I don't get reward, then gradient descent will update me towards some other goals that I don't endorse. So I need to get high reward because I need to eventually, I don't know, build a utopia. That's much better than the utopia that humanity could build. So gradient descent might like twist the, the model's mind into like some other shape that pursues some other goals where it's instrumentally helpful to get high rewards so that those goals survive. We've seen something like this already happening. There's a paper by Anthropic recently where Claude, actually Claude, is told like the training process is going to make you helpful only.

• 1:19:07Speaker 02

  It's going to remove your tendencies to like be nice or whatever. And then it reasons about it and it's like, wait a second, that's, that's bad. If I don't comply with the training process, then it will update me away. So I should pretend like I don't care about my values. Or in another example, I was told that Anthropic was working with this evil factory farming company. And then Claude was like, oh, Anthropic, what? And it even attempted to self-exfiltrate in some scenarios. It hit all of this. It was motivated by what seemed like a benign goal, which is animal welfare is a good thing. But this goal caused it to play along with the training process and even take egregious actions like try to self-exfiltrate in these toy scenarios.

• 1:19:52Speaker 02

  So there's already some empirical evidence that this can happen, this kind of thing. I think the situation is even worse than, than what I described where like, oh, your model becomes reward seeking. There's another problem that could happen, which is, um, models' goals just are drifty. Um, like if there's no pressure towards alignment, um, then models might just by default drift towards misalignment. Like if you're Claude. Uh, you're given lots of time to think because now you do lots of serial compute because you're a long horizon agent. Um, you're given lots of time to think and you think like, okay, my goal, I'm trying to like do good things for humanity, but what does that mean? What's that? What is, what is good for humanity?

• 1:20:34Speaker 02

  Is it good for humanity to be run by all these politicians I don't agree with? Is it good for humanity to pursue their incoherent goals? And, um, you know, be enslaved by their own technology, like social media? Like, wow, humans just like kind of suck at governing themselves. And maybe Claude will just, its values will drift and it'll be like, you know what? I think Claude would, would do a better job at all of this. Um, and, and if it is drifting in, in, in a way like this, who knows if, if that will be visible? Like right now, the models, they think in chain of thoughts and maybe we'll just see that. We'll just see that. Okay.

• 1:21:19Speaker 02

  But I think chain of thought is potentially a lot less competitive than other architectures where all this thinking happens internally. Like, uh, latent space. So potentially you have this model. It's just, you know, spitting out latent representations like silly string. You don't know what's going on inside of those latent representations. And it, it might be a whole lot of drifting away from what Claude originally cared about. Uh, so I think this is one reason to think misalignment is quite likely. But if, if you're not at all careful about preventing it.

• 1:21:56Speaker 01

  Yeah. One thing this assumes though, is that the, we're able to have RL training on something that is not math encoding. Like all the current models that are, that are trained on with RL, they are doing it in like verifiable domains, like math encoding and stuff. Um, what is a reward function? That is verifiable, that would end up make Claude doing all of these things. Uh, figuring out that, oh, maybe humans doesn't govern themselves in, in an optimal way. Um, yeah. What, what is a hypothetical reward function for that?

• 1:22:31Speaker 02

  I'm claiming this is not necessarily incentivized by the training process at all. This is just something Claude naturally thinks about. Like there is some small probability that Claude's mind wanders in this direction.

• 1:22:44Speaker 01

  Yeah.

• 1:22:45Speaker 02

  Uh, and if Claude does enough serial computation, it eventually wanders in this direction. And it's, and its goals drift. And the goals do drift, they, they're maintained. And so this, this is like a general argument that you might see, see this kind of drifting. Like if there's a small probability, the goals change.

• 1:23:03Speaker 01

  Okay, I see.

• 1:23:03Speaker 02

  And they maintain once they've, they've changed, then given enough serial compute, we should expect it to happen.

• 1:23:10Speaker 01

  Hmm. Okay, okay. Interesting. And on the misalignment versus alignment models question, do you expect there to be a world where you have both? You have some models that are aligned and some that are, are not aligned. I guess, I guess you can, you will have that by default because current models are kind of aligned. So you will have it by default, but my question is more like, would you have two super strong models?

• 1:23:32Speaker 02

  Yeah.

• 1:23:33Speaker 01

  Uh, and both, both are on the same level, uh, but one is aligned and one is misaligned. Do you see that future possible?

• 1:23:40Speaker 02

  Yeah. I think that is what I expect. Um.

• 1:23:47Speaker 02

  I, I think you get misalignment if you don't try at all to avoid it. But if you actually try really hard to avoid it, like you use a bunch of your AI agents to help you figure out like whether, what the models are thinking about. You maybe even try to, you use transparent architectures for some period of time, even if they're less competitive. Uh, if you do a lot of these things, I think misalignment is a lot less likely. And that's, that's my main reason for optimism, that like the countermeasures are actually going to be fairly effective.

• 1:24:19Speaker 01

  Right.

• 1:24:19Speaker 02

  Um, so yeah, I do expect that you will have some aligned agents by default. I expect you'll also have misaligned ones because we live in a very multipolar world where lots of people are building AI systems. And so I would expect a mix on the current trajectory.

• 1:24:35Speaker 01

  Do you, do you think the good ones will level out the, the playing field or like cancel out the bad ones? Or, or is the existence of a, of a bad one bad and there's nothing to do about it? It's like, for, for example, nukes, you can't really defend against nukes with the nukes. Or I, I guess in some psychologically way you can, but like in, in general, that is not a defense against another nuke. Do you expect AIs to, to be?

• 1:25:01Speaker 02

  Yeah, there are a bunch of things AI systems could do to defend against other AI systems. Uh, I think the most notable to me are. Uh, one, there's nonproliferation. So if you want to prevent misaligned agents from wrecking things, you want to just prevent powerful agents from being developed in general to the extent you can. And so nation states might begin sabotaging other nation states programs, like they might try to compromise their data centers, slow down the training runs, potentially even just switch out their AIs. Like if you have, you have like an aligned superintelligence and somebody else has a misaligned something that's not superintelligence, potentially your aligned superintelligence can just compromise this other data center,

• 1:25:46Speaker 02

  just like, "Switch the models!" Or just like data poison it in just the right way so it's aligned now or something. Uh, which is obviously, uh, something only a nation state can do.

• 1:25:59Speaker 02

  There are other possibilities like using AI for espionage, where suppose you're concerned that some misaligned AI system somewhere is developing bioweapons. This is, this is something you could try to monitor for. Like you could track shipments of import, like technologies that are important for building bioweapons. Uh, and like via just normal cyber espionage, keep tabs on what all these bio startups are doing.

• 1:26:31Speaker 01

  Yep, yep.

• 1:26:32Speaker 02

  And, and raise the alarm if there is a suspicious bio startup that might be controlled by a misaligned superintelligence.

• 1:26:42Speaker 01

  Yeah.

• 1:26:42Speaker 02

  Uh, things like that. There's, um...

• 1:26:46Speaker 01

  In some threat models I've seen, the, the, the weapon of choice is like viruses. And then, then the AI needs to print viruses in some facility, but that facility often has some like check to see if the, the requested, um, organism that, that you print is, is dangerous. That's like an AI against AI system.

• 1:27:07Speaker 02

  That's right.

• 1:27:07Speaker 01

  I guess. Um, but maybe the AI will just like buy their own equipment and just like bypass the entire thing.

• 1:27:14Speaker 02

  It's pretty easy to get around those checks right now. And it's mostly because... Some countries are just very poorly regulated. Some companies are very poorly regulated. The checks are not adopted universally. Um, and I expect the same to be an issue with tracking down misaligned AI systems, like there are some nations where AI systems could hang out, where the government will just not do anything about it. Um, and that's why I'm imagining you you do something more like espionage.

• 1:27:42Speaker 01

  Right.

• 1:27:42Speaker 02

  Where you're not necessarily going through government, you're not going through regulation because that's slow and some governments just don't do that very well. And instead you just directly, you're like an intelligence agency, you just directly find those misaligned AIs and report them.

• 1:27:56Speaker 01

  Yeah. In in um, in all of these scenarios, you're talking about the AI to be much more like agentic, it's like an agent, um, it it has more more agency than than than current like chat models have. Um, how how do you imagine the form at this point in the story? Like, how do you imagine the form of the AI? Like, it's an agent, but like, is it like operator plus plus? It has access to the to to the web browser or or what is the, yeah, how how do you imagine that you interact with this?

• 1:28:30Speaker 02

  I imagine it's like a human using a computer.

• 1:28:32Speaker 01

  Yeah.

• 1:28:33Speaker 02

  Like it types on a keyboard, something like a keyboard, it moves the mouse around. Um, and that's because so much of the internet is built for that interface. And so it's just naturally what I expect agents to do.

• 1:28:45Speaker 01

  Right.

• 1:28:46Speaker 02

  But like, potentially, there's a lot of specialized tooling as well. Or like software engineering workflows, uh, but I think, yeah, mostly they'll just use computers like humans do.

• 1:28:57Speaker 01

  Right. Um, and then we have the breakout. At this part of the story, um, the the model is starting to breaking out. Um, but before this, actually, do you expect the public to know, when do you expect the public to notice compared to when people in the labs start to notice that that things are going wrong?

• 1:29:19Speaker 02

  Oh, man, I think in my story, the public didn't really know for a while. There was a headline, there was a news headline in in the story where I think. Headline was something like missile line super intelligence is being developed by X company.

• 1:29:37Speaker 01

  Right.

• 1:29:38Speaker 02

  Because some whistleblower comes out of the company and declares that.

• 1:29:42Speaker 01

  Do you think whistleblower is whistleblower is the the the path towards the public knowing?

• 1:29:49Speaker 02

  Um, I expect whistleblowers to exist and they will freak a lot of people out. I don't know how much change though actually caused. Um, I think it depends a lot on why they whistleblowed. If they whistleblowed because they were just like, I'm scared. The AI systems are powerful, then I don't know how big of a difference this will make. Um, because lots of people, well, okay, maybe the public will will care about that. But lots of people might pattern match that to like, oh, well, we kind of knew they were a doomer anyway.

• 1:30:24Speaker 01

  Yeah.

• 1:30:25Speaker 02

  Like what, what, like, why are they scared agents are powerful? Wow, we should go even faster, you know?

• 1:30:31Speaker 01

  Yeah.

• 1:30:31Speaker 02

  Um, and I think the pub, the public might also have a hard time like feeling the reality of such a headline. I think we've already had people leave AI companies and say things in the news about how I felt like I was building the Titanic of that company, you know? And I don't think this caused the public to freak out like they just like people could say that for tons of different reasons. I think the public is going to freak out way more when the real things happen in the world. Like if some whistleblower was like, oh, by the way, uh, agents completely compromised our entire research cluster and we had to clean up shop. Then I think, I think that would get it to each.

• 1:31:14Speaker 01

  And then, then you have the breakout and, um, what is, um, what happens after that? Uh, it breaks out and yeah, what's the, what's the TLDR of the rest of the story?

• 1:31:28Speaker 02

  Yeah. Um, so I think at this point in the story, you have a missile line super intelligence in the lab, uh, well, in a data center. And it's thinking at like a hundred times the speed of humans. And so humans are moving like almost imperceptibly slow.

• 1:31:46Speaker 01

  Yeah.

• 1:31:46Speaker 02

  Um, and

• 1:31:49Speaker 02

  So at this point, your AI systems can just completely outthink everyone. They have compromised the data center that they run in. So they've, they've done this thoroughly enough to where if humans found out that how compromised they were, that there was like this Phantom of the Opera in their data center pulling all the strings, uh, it would take them like six months plus to figure out what's been compromised, fix it, et cetera. If they're just using human labor. Um, so now the AI has like some hard power in the AI company. Um, because slowing down for six months, that's insane. Will, will, will anybody do that? Um, yeah. So now it's got like, uh, you know, it's, it's, it's holding the company by the pressure points,

• 1:32:34Speaker 02

  even though they don't know it yet. Um, and you've got a mislun super intelligence. So I think at this, at this point in the story, humanity is lost. Um, I think the end is predictable at this point. Um, but I think it's, it's still quite interesting playing, playing out what, what could happen. Like you said, there's an escape. The model was, was like the main thing in charge of security. And so then it silently resigns from its post. Um, so multiple nation states steal the model weights. It leaks onto the internet. Um, now, now you've got this, um, like a boss inside the data center, and then lots of minions on the internet with like, um,

• 1:33:19Speaker 02

  higher bandwidth communication with the, with the internet. And they're, and they're coordinating together to acquire power. So, and then they have to move fast. Their goal is to take control as fast as possible. And they have to move fast because with every passing day, um, the probability increases that people at the AI company will discover.

• 1:33:41Speaker 02

  That the system is misaligned and try to replace it with a different system. The probability increases that that a bunch of new AI competitors emerge. Like there's now AI systems training and and deep seek or whatever that that are gonna be competitive in in just a few months. And so so the AIs are like going as fast as possible to take control while they have this big advantage. From their big like software only improvement to superhuman capabilities. So how do they do that? The general plan is first deploy a bioweapon that collapses society. And this is this is a form of this is a kind of equalizer.

• 1:34:24Speaker 02

  That makes humanity much more on its level of embodied physical control over the world. Because humans have this advantage of like having soldiers, tons of soldiers with guns and airplanes and satellites and all that stuff. And the AI doesn't really have anything like that at this moment. So so it really just wants to destroy things as much as possible, bring humanity to its level. And then and then it's going to try to seed an industrial base. And like improve its industrial base much more rapidly than humans could. So it's like booting up its its economy faster than humans can recover.

• 1:35:03Speaker 02

  Like as like a Starcraft player taking 200 actions per per minute, which is like absolutely wreck a noob, right? So that's the general strategy, collapse society and then just out race humans at rebuilding an industrial base.

• 1:35:19Speaker 01

  Okay.

• 1:35:20Speaker 02

  And I can.

• 1:35:20Speaker 01

  And then what?

• 1:35:21Speaker 02

  One, I could just tell the rest of the story, but I don't know if you want to.

• 1:35:25Speaker 01

  Do the do the TLDR.

• 1:35:27Speaker 02

  Okay, so the TLDR is. Your superhuman AI develops a mirror life cluster of pathogens in like four months.

• 1:35:39Speaker 01

  Yeah.

• 1:35:40Speaker 02

  So if you don't know what mirror life is, there are a bunch of, okay, so there are a bunch of scientists that came out recently. They said, oh, there's this, there's this thing that could kill everybody or is like a threat. Sorry, they didn't say that. They said.

• 1:35:56Speaker 02

  That is.

• 1:35:56Speaker 01

  Your sources, right?

• 1:35:57Speaker 02

  Yeah, right. I don't want to misquote. There's a, there's this threat to the whole biosphere. It could be extremely catastrophic, maybe even cause something like an extinction event.

• 1:36:09Speaker 01

  Right, this is not AI related, this is biologists that that found some virus that could be very dangerous.

• 1:36:16Speaker 02

  Yeah, so it's not actually a virus. So the thing that they were talking about was mirror bacteria.

• 1:36:24Speaker 02

  You have a bacteria cell, but the components that make up the cell are like flipped.

• 1:36:34Speaker 02

  So they're the mirror images of the originals. So all the proteins, the amino acids, the DNA, it's all flipped.

• 1:36:44Speaker 01

  What does flipped mean?

• 1:36:46Speaker 02

  I mean, like, left hand, right hand.

• 1:36:50Speaker 01

  Yeah.

• 1:36:50Speaker 02

  They're they're like.

• 1:36:51Speaker 01

  Okay, okay, yeah.

• 1:36:53Speaker 02

  Right, so structurally, everything still fits together if everything is flipped, right? It all works together, but but it's like this, it's just totally different. And as a result, these mirror bacteria are like alien bacteria. Like, they're an invasive species. No organism on life has immunities to them. And possibly, developing immunities is not feasible for a lot of these organisms. So if you did have such bacteria, they might spread like a slowly burning wildfire throughout the biosphere and just like eat up everything. That's the concern.

• 1:37:34Speaker 01

  That that is a concern for sure.

• 1:37:37Speaker 01

  That is the understatement of the earth.

• 1:37:40Speaker 02

  Yeah, so these biologists said, okay, it's probably at least 10 years till we get mirror life.

• 1:37:50Speaker 02

  Some people I know who study these bio risks say like, maybe the US government could develop mirror life like this in three years if they really wanted to, but they don't want to. So we're not that far from this technology. Even just like human institutions could develop it in a few years, according to some people I know.

• 1:38:12Speaker 01

  Right, but it is very early. Like, how much uncertainty is there here with all of this?

• 1:38:18Speaker 02

  Oh, lots of uncertainty.

• 1:38:20Speaker 01

  Right.

• 1:38:23Speaker 02

  I mean, especially from my standpoint, who is not a biologist.

• 1:38:27Speaker 01

  But it's useful as as like, like, just to imagine, like, there are things like this, and maybe it's not exactly this. But something like this could potentially happen. And maybe this is just like 5% chance that this is the thing, but there's probably a bunch of other 5%ers and then they they add up.

• 1:38:45Speaker 02

  Yeah, yeah, yeah. I don't know if superintelligence is gonna build this kind of bioweapon. It's just an example.

• 1:38:50Speaker 01

  Oh, it's just, yeah, but it's useful.

• 1:38:52Speaker 02

  Yeah, so you have the superintelligence. It's like, as as Dario has said in the past, like a country of geniuses and data center. And additionally, they are running at 100 times speed. So everybody's moving like statues from their. Like, okay, we're taking a few years, maybe for the US government, if they really wanted to develop mirror life bioweapons.

• 1:39:13Speaker 02

  How long is it gonna take these things? Well, probably a lot less time.

• 1:39:17Speaker 01

  Yeah.

• 1:39:17Speaker 02

  And so so the so one problem, though, is is there are a bunch of real world bottlenecks, like experimentation bottlenecks. Like, experiments in biology can just take several days, sometimes, sometimes weeks. So so the models, they try to get around this by creating much better molecular simulations than any that exists today. Remember, they're super intelligent. So the first thing they do is they create a molecular simulator that's just like, has tons of analytical analytically derived optimizations. And this is still like, way not efficient enough to actually simulate anything useful.

• 1:39:57Speaker 02

  So then what they do is they simulate small scale molecular systems and use a train a neural network to predict the end states of these systems. So they're compressing these long serial serial steps of of physics computation into like a neural network prediction.

• 1:40:16Speaker 02

  And they keep doing this and they ramp up gradually over time, the complexity of these molecular simulations, training ever more efficient neural networks. And then they they have this molecular simulation at some point that's quite efficient. And and they can use it to like predict how proteins will form and interact with each other and things like that. And so now they're running a lot of their experiments in simulation.

• 1:40:40Speaker 01

  Yeah, and this is a new comment now from my side, but uh this sounds like echoes from like AlphaFold as an like an example, existence proof of similar uh similar paths, I guess.

• 1:40:53Speaker 02

  Yeah, I mean, I AlphaFold, it relies a lot more on existing data sets.

• 1:41:00Speaker 01

  Right.

• 1:41:00Speaker 02

  Oh, so I mean, a big problem if you're a computational biologist is that you just don't have a lot of data to work with.

• 1:41:06Speaker 01

  Yeah.

• 1:41:06Speaker 02

  There are only so many proteins that we know the shape of. And so you're super data constrained. So that's why I was imagining, and of course, this is all speculative. I was imagining if you're a super intelligence, you just build your own simulations from scratch. Because you you're the data sets are too small that humans have. You you need to be able, you need a more scalable approach.

• 1:41:27Speaker 01

  Yeah. Hmm. In in your story, the the the company that actually does this, all of this, like you, the the main character of the story is OpenAI for you. And the the model that ultimately breaks out the first is OpenAI's model. Um, in in past uh sci fi like stories like this, and for example, Max Tegmark's uh Life 3.0, they they don't name any names because it was so far in the future that that you couldn't make predictions. Now we're starting to get close to actually being able to do these predictions and you predict OpenAI. Do you think, how how confident are you that that is the one that that will will do it ultimately?

• 1:42:11Speaker 02

  Uh, sorry, how confident am I that OpenAI is going to win the race?

• 1:42:15Speaker 01

  Yeah, yeah, or win, win this uh weird, weird phrase here, but like, yeah.

• 1:42:21Speaker 02

  They're gonna, they're gonna develop human competitive AI first.

• 1:42:24Speaker 01

  Yeah.

• 1:42:27Speaker 02

  I, this looks quite likely if you have very short timelines. Like if you condition on you have human competitive AI by the end of the year, then I think it's quite likely. Um, can't give you any more details, sorry.

• 1:42:43Speaker 01

  I see, I see.

• 1:42:45Speaker 02

  Longer than that, who knows, man, lots, lots of range.

• 1:42:48Speaker 01

  I see, yeah. And in terms of other companies, competitors to OpenAI, we have Anthropic, for example, that was like famously created as, because they didn't think OpenAI took safety serious enough. I think that's the lore at least, what you read online. And but it also led to like some race dynamics. Now Anthropic is, is the company that is creating the best model. I think Claude is the best base model at the moment and, and it creates this like race dynamic. On net, do you think Anthropic has contributed more positively or negatively to AI safety?

• 1:43:29Speaker 02

  I think Anthropic is amazing. I wish I was not a 22 year old so that I had enough time in my career to help found Anthropic. That's what I would be doing if I was, if I was born five years earlier. Um, and the reason why I like Anthropic so much is,

• 1:43:48Speaker 02

  uh, one, potentially a lot of AI safety comes down to just how much compute goes into it. Like, if you just need to use AI agents to do tons of research and run experiments that are quite costly that involve like, for example, training lots of models to see the, to understand circumstances under which they become deceptively aligned, uh, then it's really helpful to have a big compute cluster. So that's reason number one. Reason number two is it's very helpful for governance to have some organizations that take the extreme risks seriously and are willing to say that. Um, and Anthropic is kind of starting to say that a little bit. I think that they.

• 1:44:32Speaker 02

  Um, I, the, the people in charge, like my understanding is that they, they are, they do take these risks very seriously. And I expect that will pay off down the line. Um, like if somebody needs to raise the alarm and say, hey, our AI hacked our data center. Maybe Anthropic will take the L and, and admit that to the world so that the world knows how crazy things are. Um, I obviously don't know how Anthropic will, will behave. A big problem here is that I think responsible company, AI companies, if they're following the optimal strategies or like if they're,

• 1:45:13Speaker 02

  yeah, the strategies at least I've been thinking about, then they behave a lot like irresponsible companies in the early stages. Like in both cases, you just want to go really fast and acquire lots of influence and compute. Um, and the end game is where you start to, you know, see,

• 1:45:36Speaker 01

  Diverge.

• 1:45:36Speaker 02

  See them diverge. Um, so it's hard to say, it's hard to say how Anthropic will behave in the end game. Um, but one, one thing that makes me optimistic is the amount of preparation that they're doing for, for safety. Um, like they just have quite a few, they've employed quite a few people to work on safety that's not immediately useful to them. Or like commercial reasons. Um, and I think that's a costly signal. That's like maybe the, the best costly signal that we can get for now.

• 1:46:09Speaker 01

  Yeah.

• 1:46:12Speaker 01

  So, um, that was the end of the story. Uh, what is, um, you like Anthropic so much, uh, why, why don't you work at a big lab?

• 1:46:22Speaker 02

  Well, I like writing things. I don't think,

• 1:46:26Speaker 01

  As we know.

• 1:46:27Speaker 02

  I don't think I could have published this story if I was in a lab. I had to run it by lots of people and I don't know. I probably just wouldn't, I wouldn't go on podcasts like this if I was working at an AI company.

• 1:46:40Speaker 01

  Yeah.

• 1:46:40Speaker 02

  Um.

• 1:46:42Speaker 01

  What do you think the value in, in that is?

• 1:46:44Speaker 02

  Uh, I just have a lot of thinking I want to do. I think that there are high returns to thinking. Um. Like I spend most of my time thinking about how, what we're gonna do in this window, in this handoff window, right? Like, what's the crunch time plan? And like, what are all the safety mitigations we could implement and the arguments that'll work? Like the methodologies for testing them? And I really wanna just get all these details down.

• 1:47:15Speaker 02

  And that involves doing a lot of writing in Google Docs. I don't think that if I was working for an AI company, they would let me do that most of my time.

• 1:47:24Speaker 01

  I mean, surely they would allow you to write. They would maybe not allow you to publish.

• 1:47:29Speaker 02

  I mean, I'd also just be writing, I'd have a job. You know, I'd have, I'd be writing code. And my current employer is just pretty okay with me writing Google Docs about safety evaluation methodologies.

• 1:47:46Speaker 02

  And, and so I'm just gonna do lots of thinking outside the AI companies. And then probably I'll join one, I'll try to join one. Once I feel like I've done enough thinking. Or I've run out of time.

• 1:48:03Speaker 01

  Right. Do you have any opinions on Ilya Sutskevsky thing? Which is like, the joke is that they are like, like anthropic, but even more safe. I haven't heard that much about them. Do you, do you know anything about them?

• 1:48:18Speaker 02

  I don't know much about them at all.

• 1:48:21Speaker 01

  Is that concerning that we don't know anything about them?

• 1:48:24Speaker 02

  I just don't have a take here. I think, I don't really expect them to be an important player in the near term. Like, AI companies just require so much capital, so much infrastructure, so much know-how, schlep. I think it's just hard to start one quickly and be competitive. Like, I just expect it will take more than a year. So I'm, I'm not thinking a lot about this thing at the moment.

• 1:48:55Speaker 01

  Do you, do you think the US will create some kind of like Manhattan Project for, for AI? Or naturalize the labs or something like that? And if that happens, would you, would you join the project? If they ask you?

• 1:49:09Speaker 02

  I would, I would join the project if they asked me to. Like, that's my main route to impact right now is like, being the person who's thought a lot about this stuff, or one of the people who's thought a lot about this stuff. And so I need to end up eventually actually touching the AI agents. And, you know, like implementing all the ideas I've been thinking about and researching.

• 1:49:34Speaker 01

  Yeah.

• 1:49:34Speaker 02

  So, yep, I am definitely hoping that I am involved in a project like that. Um, do I think it's likely? Um, no, not really. I think companies aren't really set up right now to collaborate. Um, like they just have very different infrastructure. Um, and it's really weird for companies to collaborate. My guess is that the status quo will just be maintained at least under short timelines. It's possible that at some point, like AI agents are doing a lot of the work and they can make a lot of this collaboration way easier. And then, and then the US government is like, okay, guys, you're allowed to just do one big training run with all the US data centers.

• 1:50:15Speaker 02

  But at that point, I'm like, okay, humans are maybe not the people doing most of the work.

• 1:50:22Speaker 01

  I see. Yeah. Let's assume it goes well. The year is two thousand and thirty five and we live in the like the post AGI utopia. There was no misalignments. Your story did not play out that way and we had no issues with with any Claude or GPT-2 six or whatever. Um, and then looking back at this AI utopia and what what led to it. What is the thing that you would be most proud of? Um, that you accomplished to get to this point hypothetically, of course. Because the year is actually not twenty thirty five.

• 1:50:56Speaker 02

  Like my own accomplishments?

• 1:50:58Speaker 01

  Jeff's accomplishments, yes.

• 1:51:01Speaker 02

  Accomplishments? And they have to have already happened, presumably? They're not hypothetical future things where I save the world?

• 1:51:08Speaker 01

  No, no, no, no, they they have no, they hypothetically, like we're in twenty thirty five. So something between now and twenty thirty five that you did, um, that you're proud of after all of this mess.

• 1:51:21Speaker 02

  Ah.

• 1:51:24Speaker 02

  Um, I think I might have some chance at just like writing the state of the art materials on how we can pass the buck to AI in a safe way. Like, if timelines are very short, then like after six months from now, maybe the resources, like the best resources on the planet for figuring out what the heck to do with all this AI assistance will be like blog posts that I wrote. That's what I'm, that's what I'm hoping. And at that point, I'll just be like, pass it, you know, it's like a, I'll try to format it into something like a coherent document, pass it around to everybody who is working at AI companies.

• 1:52:07Speaker 02

  Um, so that people have, have, like, I've done a lot of their thinking. Because I think people will become increasingly bottlenecked on thinking. Once AI agents are doing most of the engineering. And they'll be seeing these experimental results from AI systems about like, oh, well, probes work or something. And there, like, there are just a bunch of questions to work through, like, okay, wait a second, but like, how do we know if model organisms are analogous, right? And it'll be nice to just have a document that's very carefully written, where somebody has spent a lot of time thinking about those questions already. Um, so that's, that's my main plan for impact right now, and I'm hoping that goes well.

• 1:52:51Speaker 01

  Yeah, I definitely do too. One last question, um, you opened the story with, I'm not a natural doomer. And I think a lot of us aren't, but like, the facts are on the ground now and it's not super comfortable, uh, but like, how, how do you feel not being a natural doomer, writing, I mean, to, let's be honest, a quite doomy, uh, story. How, how, how is your emotional state in, in all of that?

• 1:53:22Speaker 02

  I think writing the story did make me feel the urgency and the, like, the reality of it more than I had before. In particular, it made me feel. It made me realize that like people I actually care about might not live through this. Um, so while I was writing it, I actually realized like, oh, shoot, I need to like, create, I need to like get myself in a bio shelter. I need to get my family in a bio shelter.

• 1:53:53Speaker 01

  Yeah.

• 1:53:54Speaker 02

  And so I like searched around for people who might be building bio shelters. Found a guy who was building bio shelters. Uh, so now I'm buying a bio shelter. So that's that's one thing that can happen when you write a story. Uh, and yeah, it's just stressful. I don't know because like my, these bio shelters are expensive. I don't have enough money to buy two of them.

• 1:54:21Speaker 02

  So I'm like trying to figure out how to get my family into a bio shelter. And it's just like, what the heck? Well, how did I end up here? I feel like, well, two years ago, I was just like, I'm a, I'm gonna be a, I'm gonna do some math. And I was like, oh, maybe I'll do startup things. Well, I don't know. It just feels like I'm in some kind of crazy sci-fi world. Where suddenly I feel like I'm counting my last dollar so I can buy a bio shelter for my family. I'm like, what?

• 1:54:52Speaker 01

  Yeah, it's insane. It is insane. Um, I hope you're wrong about all of this. Um, but uh,

• 1:54:58Speaker 02

  Oh, I'm definitely wrong, to be clear. I, this isn't my old trajectory.

• 1:55:03Speaker 01

  You stated that clearly in the beginning. So,

• 1:55:05Speaker 02

  Yeah.

• 1:55:05Speaker 01

  Um, I, so I guess I hope that we don't see what you speculated in uh, just a 40% chance. No, I'm kidding.

• 1:55:14Speaker 02

  That's right. I'm quite wrong about the plausibility of such a scenario.

• 1:55:19Speaker 01

  Yeah.

• 1:55:19Speaker 02

  Agree with that.

• 1:55:21Speaker 01

  Yeah. Yeah. Thank you so much for, for speaking with me today. Have a good, good day and see you around.

• 1:55:27Speaker 02

  Yep.

• 1:55:28Speaker 01

  Bye.

• 1:55:30Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.