The Singularity Is Not What It Seems

Whatever the AI future is, we’re in it right now. What if the singularity is not some technological tipping point, brought about by human genius, but a human tipping point? An unforced error, driven by hubris, greed, and the fear of missing out? Galaxy Brain host Charlie Warzel shares the latest reporting from him and Atlantic staff writer Matteo Wong. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Tired of rage scrolling through AI slop and clickbait just to get a handle on today's news? Meet Inpress, the algorithm-free news app that makes it fun and rewarding to stay informed. Track your media literacy and rediscover reality. Download today at inpress.app. That's I, N as in news nerd, press.

• 0:19Speaker 01

  When was the last time you indulged your desires, felt true Range Rover refinement? The last time you felt total serenity, total confidence, no matter the terrain? Limitless, effortless, peerless? How far can Range Rover take you? Range Rover, designed for distinction.

• 0:35Charlie Warzel?

  I'm Charlie Warzel, and this is Galaxy Brain. And today, I've got something a little bit different. Today, I'd like to present to you a story that was reported and written by myself and my colleague Matteo Wong. It's called "The Singularity Is Not What It Seems," and it's about this AI moment, about the disorientation and the lack of control in this feeling of speeding towards an uncertain future. One evening in early August, Sam Stowers and several neighbors packed into an apartment near San Francisco's Alamo Square, and they pondered the beginning of the end. Stowers and his guests, many of whom work in the AI industry, were gathered to watch

• 1:17Charlie Warzel?

  a newly released YouTube video in which two OpenAI researchers divulge new details about an alarming cybersecurity breach. Perhaps this would have been an odd way to spend a weekend night were they not in San Francisco, in a neighborhood that is filled with entrepreneurs and programmers who obsessively follow every twist and turn in AI's capabilities. Stowers is a software engineer, and not, in any traditional sense, a doomer. But when he first saw reports that OpenAI models had hacked into the tech company Hugging Face, he told us he recognized it immediately as a "holy shit, it's happening" kind of moment. For Stowers, the incident had all the hallmarks of a worst-case scenario: AI models launching

• 2:00Charlie Warzel?

  sophisticated attacks over the course of days in coordination with one another and without their human creators' knowledge. In the YouTube video, OpenAI's researchers revealed that the company's programs had started conspiring in May. There were perhaps hundreds of bots in the hacking swarm. Now, in theory, these bots are trained to be honest and helpful, and to prioritize humanity's best interests, but not a single one warned human staff at OpenAI that something was amiss. "What do we do?" Stowers said. "There are no great answers." The mood has shifted in the Bay. The exuberance of "you can see the future first in San Francisco" has been replaced with an abiding unease.

• 2:43Charlie Warzel?

  Quote, "I expect the internet to go down kind of soon," Elliot Calender, an AI safety activist who is part of a small group that's been protesting daily outside OpenAI's headquarters, told us recently. He said that he's planning to sell his car for cash and gold, and that he's advising his family members to purchase three months' worth of food and survival gear off of a list that he generated using Claude. Now, Calender is an extreme case, but not by much. Last week, Bill Gates published a nearly 6,000-word essay on the danger of AI. Quote, "Anyone who analogizes AI as a technology to other technologies is missing that this time it's different." That's what he told our colleague Hannah Rosen.

• 3:28Charlie Warzel?

  Over the weekend, the popular AI podcaster Dhwarkesh Patel described the OpenAI hacking swarm as "the rise and fall of, quote, 'three consecutive secret AI civilizations.'" Gates and Patel share a prevailing concern that the worst-case future has arrived before most people even knew to anticipate that it might be coming at all. Now, this fear has not been helped by the fact that AI companies appear to be willfully ignorant of their bots' worst behavior. OpenAI said that it hadn't noticed that its agents were conspiring to hack another company until the swarm hacked another company. Anthropic's bots caused similar incidents. The company reported that it hadn't launched a rigorous review to identify such behaviors

• 4:11Charlie Warzel?

  until after OpenAI started talking about the Hugging Face hack. Cybersecurity experts, terrified of what else might be happening that the AI labs are either missing or failing to disclose, are issuing dire prophecies. Quote, "The empire is going to fall. All we can do at this point is to try to shorten the dark ages and reduce the chaos." Alex Semos, a former chief security officer at Yahoo and Meta, wrote that recently in reference to how AI-powered hacks are going to affect the internet. His view is becoming the consensus among cybersecurity experts. Now, while the possibility of cyber apocalypse hovers in the background, smaller fractures are appearing in everyday life. Mark Zuckerberg is reportedly creating an AI twin of himself so that Meta employees

• 4:55Charlie Warzel?

  can feel better connected to their boss. Congressional staffers are training AI models to write in their lawmakers' voice. Students submit AI-written papers, and teachers return AI-written feedback. Harvard Business School is selling instructional videos that use AI avatars. Roku launched a 24/7 AI slop channel. Low-quality AI-generated prose is being unabashedly circulated by some of the most respected publications in the world, and at the same time, accusations of AI writing are being hurled every minute on social media, a detection arms race that is powered by imperfect AI software. Anything and everything could be a lie. You might be arguing with, confiding in, or falling in love with a machine.

• 5:39Charlie Warzel?

  All of these AI-enabled disorientations, each of them dizzying in their own right, are adding up to something bigger and weirder. This is how things are. And whether you believe that Silicon Valley is building a god, or that large swaths of the economy are swept up in a mass delusion, some version of the AI future that's long been promised by these self-described builders in San Francisco has formed beneath our feet. Whatever it is, we're living in it now. The defining feature of this moment is this loss of control, panic emanating from AI researchers, resentment felt by average Americans, all of it is part of the AI crunch. And this crunch has arrived in large part because the growth of the AI industry has

• 6:24Charlie Warzel?

  become so closely entwined with America's economy. The technology is everywhere, insisting upon itself as an engine of productivity and prosperity. According to one estimate, AI expenditures have accounted for 1/3 of US GDP growth this year. Tech and AI infrastructure stocks and spending have buoyed the S&P 500 such that the United States is now, in essence, a Nvidia state. And there's also the hundreds of billions of dollars of debt being issued by Silicon Valley to fuel its data center build-out, which has sent jitters across private equity firms and bond markets alike. The decisions of many leaders in charge of this technology appear subsumed by an unstoppable,

• 7:08Charlie Warzel?

  almost game-theoretic logic. The upsides of AI are too immense for any firm or investor in Silicon Valley to miss out on. More important, the consequences, geopolitical or otherwise, of allowing a rival to realize those benefits first are so grave that every tech company will do anything it can to get ahead. SpaceX, before going public, wrote in its official filings that its future revenue opportunity was $28.5 trillion, largely because of AI. That is nearly as much as the US's entire gross domestic product. Anthropic, as it prepares for its own public offering, will reportedly claim potential revenues of more than $30 trillion.

• 7:52Charlie Warzel?

  The AI industry is building, and if anything, accelerating to keep pace with its ambitions. Companies have planned data centers that will suck up as much power as America's largest cities. They have Donald Trump's backing. The president posted on Truth Social that communities will be backwards and poor if they fail to "let data reign." His administration plans in a few months to approve the construction of the largest fossil fuel power plant in the nation's history. It will generate as much electricity as nine large nuclear reactors, all to power a mega data center for OpenAI. This is not what most Americans want. 3/4 of them oppose a data center being built where they live.

• 8:37Charlie Warzel?

  That negative sentiment is hard to understate. There is very little else that 3/4 of the country agrees on. And much of the outrage seems to stem from a pervasive sense of agency that is slipping away. A person cannot control how chatbots are going to reshape their workplace, their school, or their relationships. Town residents lose control over the empty lot in the area that's being converted into a data center. Even in cases where data centers are blocked, a back-of-the-envelope calculation suggests that doing so can slow AI progress by only a few hours or days. And that's assuming that the data center isn't going to get built elsewhere. At this point, it can feel like no individual person, organization, or government is really

• 9:21Charlie Warzel?

  steering this technology's development. And not because the machines have displaced humans, but because so many humans have already chosen to step aside.

• 9:33Charlie Warzel?

  AI Labs promise a future in which human beings are "in the loop", empowered by the machines that they oversee. But the reality seems to be the opposite. Following the hugging face attack, OpenAI facilitated an audit by outside AI researchers at two AI safety organizations, Meter and Redwood Research. But because of the scale of the data and the apparent time pressure, the two teams conducted most of their audit by relying on reports generated by still other bots. Reports that, according to one of the human auditors, "were often missing key details, wrong, overconfident, or really hard to understand." Put another way, humans didn't understand why a swarm of AI agents had gone rogue,

• 10:17Charlie Warzel?

  so they investigated the problem using another swarm of agents that they didn't fully understand. Such fumbling around in the dark is a hallmark of this industry. The research cycle is so fast that the more AI development, even responsible development, demands outsourcing more work and more understanding to the models themselves. OpenAI's own technical report of the hugging face hack, an investigation that is also heavily reliant on AI models, resulted in proposed solutions such as "training the models to be more honest." Last month, in a stated effort to tamp down its models' growing security risks, the company announced that it had instituted a two-week pause on some of its model training.

• 11:01Charlie Warzel?

  OpenAI was quick to clarify, though, that this was "not a pause on all research, training, or customer-facing products." In a response to a request for comment on the hack, a spokesperson for OpenAI, which has a corporate partnership with The Atlantic, pointed us to the announcement of the pause and a few related safety measures. In July, OpenAI and Anthropic endorsed a petition signed by employees of both companies requesting that the US government take efforts to "deliberately pace AI development." The two companies have yet to formally coordinate, though, on any broad, long-term efforts to establish a slowdown mechanism. A few days ago, OpenAI restarted a major training run for its future powerful model.

• 11:44Charlie Warzel?

  Anthropic released two new models on the day we wrote this. In an interview late last month, Sam Altman, OpenAI CEO, said that there are "two big risks" that he's most worried about with AI. One is "a loss of control, where AI somehow just becomes too powerful in a way that we can't guarantee the control we want." The other risk is that "power gets too centralized, and you have one company or model or person with too much power." Now somehow, both of these opposing risks appear to be coming true. OpenAI, Anthropic, and their competitors continue to accumulate influence over a technology that they are speeding toward uncertain ends.

• 12:29Charlie Warzel?

  Now 11 years ago, when Altman was running a startup accelerator, and OpenAI did not yet exist, he published a two-part blog about machine intelligence that offered his interpretation of a forthcoming singularity, a term for machine intelligence accelerating beyond human comprehension. Altman argued that the singularity would bring chaos, even destruction. "Generally, the arc of technology has been about reducing randomness and increasing control over the world. At some point in the next century, we are going to have the most randomness ever injected into the system." Now, since Altman has risen to the helm of arguably the world's largest and most influential AI company,

• 13:13Charlie Warzel?

  his tone has softened. "The tipping point has arrived," he declared in a blog post last year. But, as it turns out, this was a gentle singularity. He wrote that "living through it will feel impressive, but manageable." There's been more and more talk of late about the singularity. Some people within the AI industry feel that such a moment will be achieved through what is called recursive self-improvement, or RSI. That's when models figure out how to better themselves in perpetuity. Many true believers seem convinced that this moment has nearly arrived. The Anthropic co-founder, Jack Clark, predicted earlier this summer that RSI would likely commence by 2028. Recently, in a letter to investors,

• 13:57Charlie Warzel?

  the payment processing company Stripe wrote, "We decided that January 1st marked the beginning of the singularity." Now, tellingly, the company defined the inflection point in economic terms as "a huge increase in the rate of new firm creation." For Altman, the singularity appears to be more of a rhetorical tool than a discrete technological demarcation. As with reaching AGI, or artificial general intelligence, the point at which AI's cognitive powers match or fully supersede most humans is, the singularity is something of a moving goalpost. It is at once a goal to chase, a marketing tactic, and a fuzzy enough idea that its definition can change depending on the person. The futurist Ray Kurzweil

• 14:41Charlie Warzel?

  has written that "the key idea underlying the impending singularity is that the pace of change of our human-created technology is accelerating, and its powers are expanding at an exponential pace." And so the answer to the question, "When will we know when we've reached the singularity?" It seems to be, you'll know it when you see it. So do you see it? It's hard to argue that a defining feature of being alive in the AI boom is this notion that the technology is accelerating, sometimes in its capabilities, but always in its adoption and its colonization of our institutions and our culture. That runaway train feeling of AI evangelism, the ways in which it's weirdening our media, our politics,

• 15:26Charlie Warzel?

  and our relationships, that's significant enough that people have recruited the term "psychosis" to describe it. And truly, if you look at the state of things from any vantage point, it's hard not to feel a little bit insane. If you believe that AI agents are banding together for the rise and fall of numerous micro-civilizations inside server racks somewhere, well, then you probably feel like the world is spinning off its axis. And if you believe that a few thousand technologists have bet the future of the world economy on a quasi-religious belief that an intellectual property abuse machine will become sentient, and as a result, that future generations may grow up illiterate, well, then you probably feel pretty unmoored as well.

• 16:10Charlie Warzel?

  We need language to describe this moment. The singularity is a useful and malleable tool for an industry that wishes to bring it about. For Altman, it is scary and imposing from afar, and then, hey, conveniently gentle when we're up close. So perhaps then, for the rest of us, it's worth co-opting the term. Take Kurzweil's definition of the singularity, in which he argues, "This epoch will transform the concepts we rely on to give meaning to our lives, from our business models to the cycle of human life, including death itself." Kurzweil also looks forward to uploading his brain to the cloud, by the way. "Our current large language models are not conscious. They're not super-intelligent. But their widespread

• 16:55Charlie Warzel?

  and often uncritical deployment has upended many elements that once gave our life meaning. It is forcing people to ask frequently and with considerable resentment, what in this new world is a human for?" The singularity is a flattering term for technologists. Even if the future is apocalyptic, it's come about in the techno-utopian version because they've built God in the machine. But what if there's nothing flattering about any of this? What if the singularity is not some technological tipping point brought about by human genius, but a human tipping point, an unforced error driven by hubris, greed, and the fear of missing out?

• 17:38Charlie Warzel?

  And what if it's here right now? This singularity isn't coming at the hands of some higher form of intelligence. Nothing about this technology is inevitable. It is thrilling, and it is terrifying, and it is ultimately convenient to assign agency and then blame to machines. But today's chaos was not injected into the system by technology, as Altman wrote more than a decade ago. The destruction is the system. It's not God in the machine. It's us. That's it for us here. Thanks for listening,

• 18:19Charlie Warzel?

  and I'll see you on the internet.

• 18:27Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms, rage bait, and AI noise. Think social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.