Obsolete or Irreplaceable? Garrison Lovely on Stopping the Race to Replace Human Labor

Author and journalist Garrison Lovely joins Nathan to discuss his book Obsolete and examine the motivations and beliefs of the leaders racing to build AGI. Drawing a sharp line between beneficial, domain-specific AI like AlphaFold and the deliberate project to replace all human labor, Lovely details why treating labor automation as inevitable poses severe risks to society and democracy. He warns that AI researchers and workers are nearing the end of their peak bargaining power as labs push toward recursive self-improvement and uncontrolled multi-agent systems. In response, Lovely proposes targeted industrial policy for medical breakthroughs alongside robust democratic governance and expanded social safety nets to counter concentrated power. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/obsolete-or-irreplaceable-garrison-lovely-on-stopping-the-race-to-replace-human-labor/ Sponsors: Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr Deepgram Flux TTS: Deepgram Flux TTS is a streaming text-to-speech model built for voice agents with natural tone, context awareness, and interruption handling. Try it free until September 12 at https://deepgram.com/keep-talking OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (04:36) Using AI in journalism (13:57) Leftist skepticism towards AI (Part 1) (18:54) Sponsors: Athena | Parallel (21:45) Leftist skepticism towards AI (Part 2) (28:37) The AI obsoleting project (Part 1) (34:13) Sponsors: Deepgram Flux TTS | OutSystems | Claude (38:11) The AI obsoleting project (Part 2) (46:36) Envisioning a prohuman future (01:00:10) Organizing frontier tech workers (01:15:31) Critiquing the alignment problem (01:32:56) Freezing the AI frontier (01:50:15) Organizing against the machine (01:56:20) Navigating the China dynamic (02:02:55) Protecting frontier AI whistleblowers (02:10:11) Episode Outro (02:12:38) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Garrison Lovely, freelance journalist based in Brooklyn and author of the new book Obsolete: The AI Industry's Trillion Dollar Race to Replace Us and How to Stop It. Against the backdrop of this summer's AI developments, with AIs having crossed the threshold from possibly scary one day to actually scary now, this is a very well-timed and potentially a very important book. For starters, it's abundantly clear that Garrison gets it. While he comes from a left-leaning political perspective and is largely writing for a left-leaning audience, there is not an ounce of AI cope in this book.

• 0:42Nathan Labenz?

  Garrison himself is an active user of AI tools, and the book takes the company's stated goal of making something that is better than humans at cognitive work at face value, grappling head-on with the very real chance that they might actually pull it off in the near future. What's more, I think he does a great job of zeroing in on core issues. This is not everything bagel liberalism for AI, and neither is it an attempt to freeze the status quo in place forever. Rather, Garrison's goal is to capture the incredible upside promise of deep learning in domains like medicine and material science while stopping companies from rushing into recursive self-improvement or otherwise creating systems that render humans obsolete.

• 1:26Nathan Labenz?

  At least, that is, until the companies can convince experts that their plans are genuinely safe and persuade the public that the results will indeed be beneficial. I also think Garrison does an admirable job of steel-manning and addressing core counter arguments. As you'll hear, while he's generally in favor of permissionless innovation, he gives good reasons to doubt that market discipline will be enough to constrain frontier AI companies, and also argues that a technical solution to the alignment problem will not be enough to deliver good outcomes overall. Knowing that Cognitive Revolution listeners do not need to be convinced to take AI seriously, we start with Garrison's personal AI usage, which by his own account has at times bordered

• 2:08Nathan Labenz?

  on Claude psychosis, and also get his analysis of why the American left has been so slow to understand the stakes of AI development. From there, we go on to explore his positive vision for the future, which includes a new social contract, which he calls the Third New Deal, that would begin to decouple individuals' right to a decent material existence from their ability to contribute to the economy. It also calls for an Operation Warp Speed-like project, built on government-sponsored prizes, to accelerate cures for all diseases. After that, we get Garrison's argument that ML researchers' collective power is currently nearing its peak and could quickly decline, his case for unionizing with the goal of

• 2:52Nathan Labenz?

  demanding higher safety standards across frontier companies, and his advice for any who are thinking about becoming whistleblowers. On politics, Garrison is realistic about the fact that there's a lot of misinformation currently swirling around data centers. But still, he's inclined to meet people where they are in an effort to build a big-tent coalition. On China, he makes a similar case to my own, arguing that the CCP values control and therefore, absent extreme competitive pressure coming from the United States, is not at all likely to rush into RSI. Finally, we talk about the organization Irreplaceable, which aims to win a say, a stake, and a

• 3:33Nathan Labenz?

  slowdown in the political arena, and to which Garrison is donating his book royalties. Plus, how interested listeners can get involved if they wish. The bottom line for me is that while I'm still an AI enthusiast and eager early adopter by nature, and significantly less worried about labor market displacement than Garrison is, even though I do think it is likely to happen, I signed last year's FLI statement on superintelligence because I do firmly believe that racing to superintelligence via a recursive self-improvement-powered intelligence explosion would be a very bad idea. And at this point, with the possibility looking more and more realistic, I think Garrison might well be correct that it's time to put our less important disagreements aside and

• 4:17Nathan Labenz?

  focus on building the coalitions needed to exercise political power. With that, I hope you enjoy this discussion about who should get to decide the trajectory of frontier AI development with Garrison Lovely, author of Obsolete. Garrison Lovely, freelance journalist and author of the new book Obsolete: The AI Industry's Trillion Dollar Race to Replace Us and How to Stop It. Welcome to The Cognitive Revolution.

• 4:48Speaker 01

  Great to be here. Good to see you.

• 4:51Nathan Labenz?

  Great to see you. I think this one might be a little bit different from some of the conversations that you're having to promote the book. As you're probably aware, I'm very deep down the AI rabbit hole personally, and I, to the degree that I understand the audience of this podcast, I think the one thing that we all have in common is we're all AI obsessives. Some of us are enthusiasts, some of us are doomers, some of us are both. I put myself in the both camp to at least a significant degree. And so, there's a lot of the book that I think does a great job of just sketching out, like, where are we in this whole AI phenomenon? Why should you take it seriously if you don't? And I think a lot of that stuff you've probably focused on in some of your other conversations

• 5:32Nathan Labenz?

  and does like a lot of work with people who you need to convince to get over the hump and start paying attention to what's happening. I think in this forum, you don't really need to convince anybody that they should be taking AI seriously, and so I thought I'd come at it maybe from a little bit of a different angle and do maybe a little bit more of an ethnography or sociological study of the AI industry, because you're somebody who has been pretty deeply embedded in it. I don't know if you'd use that term, but definitely in the mix, and I think you have quite an interesting inside view as to what is going on, and I think it might be a useful mirror to reflect back to people as you see the community that you've come to study so

• 6:17Nathan Labenz?

  closely. How's that sound?

• 6:19Speaker 01

  That sounds great. Yeah.

• 6:21Nathan Labenz?

  Let me just start with a real simple one. How do you use AI? You're a freelance journalist. What role does AI play in your life?

• 6:27Speaker 01

  Yeah, I mean, so I don't use it to write. I think that there was a period of time where that was almost like, oh, like, you could try that and see if it worked, and then it quickly became stigmatized. You know, there's some people who always hated it, but then it became like widely stigmatized as it was used more widely and as we got Pangram as like this really high-quality AI detector. But it's very helpful for a range of things that you have to do as a journalist. You know, transcription is an obvious one. It used to be that like it just wasn't very good, and you'd have to do a lot of transcribing interviews by hand. Notebook LM from Google is just this incredible tool where you can feed in like up to 200

• 7:08Speaker 01

  documents and ask questions of them, and it's, you know, doesn't, it still hallucinates a little bit, but it's much more reliable, and you can click into the specific doc. And so, you could take like 20 interviews you did, be like, find every quote related to this topic and put it to a table with the person and the quote, and it's like, it's possibly missing stuff, and so that is a risk. But the other ways to do this before were also pretty glossy, and yeah, I think that can just like help you organize your thoughts and get answers like much more quickly. And then like feedback, fact-checking, research, it's a bit tricky because like the feedback,

• 7:49Speaker 01

  you have to just know what it's stupid about, which is still a lot of things. And so, if you're like taking it too seriously, you can just waste time because, yeah, you kind of have to trust your gut, and it'll give you a better sense of where your intuitions are good and where they're not, and vice versa for the AIs. And then, you know, fact-checking, it'll often say something is wrong and it's not wrong, and there's obviously stuff that it could miss, but there's tons of times where it'll catch something that like you can verify that you had gotten it wrong or were missing something important. And so, and then like with the book stuff, it's like, you know, the website for the book and create little like tools that are helpful, create like a zip code lookup so people could find local indie bookstores, and just all these things that would never

• 8:34Speaker 01

  have been possible before. And so, like as an independent journalist who's also doing kind of multimedia stuff, it's very, very helpful, and it's been, you know, kind of, like, it is a weird thing, right, to be writing about this technology that is potentially going to disempower everybody and aspirationally like going to put a lot of people out of work kind of at best, but then, you know, still be like, it would be difficult to lose access to these tools. And then I've also gone through the phase of like, you know, borderline Claude psychosis of, you know, just when I was writing the book, you know, getting feedback on the entire thing in Claude code in a matter of minutes was just incredibly helpful in some ways,

• 9:19Speaker 01

  but then you can also just go down rabbit holes of like needing to fix stuff that's not actually broken. And it was like almost this mantra of like getting, you know, positive feedback from the AI when I was in the darkest days of writing the book and just so burned out and tired, and that's just not a great place to be. And I think everyone who's used these tools a lot, like, has the experience of wasting time building something that is not necessary and doesn't even work necessarily, and feeling like you're more productive, but you're actually just like, you know, not getting anywhere. And so, it's tricky, and I think I've like dialed back my use in a lot of ways because it's just often not going to save you time unless you really know what you're doing or you know what it's good at.

• 10:03Nathan Labenz?

  How about, I have some questions about the dark days of the book and the timing, which is proving to be, I think, pretty spot on. But how about zooming out even from AI and thinking about technology more broadly? I think one big thing about the AI safety community and culture that most people outside of it don't appreciate as much as they should is how many of the people who are concerned about AI are for all other technologies, like techno-optimist libertarians, and that's basically been me. Aside from like never quite getting crypto, I've been like, I've been waiting for my

• 10:42Nathan Labenz?

  self-driving car since I was a kid and I'm all about future, hopefully, promised medical breakthroughs that will extend my healthy lifespan, abundant energy, all these things. Like, I'm excited about all of them. Where are you in just broader relationship to technology?

• 10:58Speaker 01

  I was a techno-optimist when I was younger, and I think the world was more techno-optimistic, at least in the United States, in the West. You know, we were like told social media would connect us and Twitter would liberate people from authoritarian regimes and Google was amazing and Google Maps was so useful. And like, we were just seeing improvement in how we lived in the world and the technology seemed to be driving a big part of that. And I am somebody who now, I still deeply appreciate the power of technology, but I'm much more pessimistic about us getting technology that is good for us under current conditions,

• 11:40Speaker 01

  which is like capitalism and specifically like shareholder capitalism, where it just maximizes profits for shareholders without much concern for various other stakeholders that care about, that are affected by these companies. And so, you know, Cory Doctorow has a book, which my book is in some ways like a rebuttal to, because we both take the position that AI is not inevitable, but he thinks AGI is impossible. But he has this previous book called Enstitification, which I haven't read, but the concept is incredibly helpful. And it's basically, you know, a lot of listeners will probably be familiar, but as a tech company, you start out trying to get as many users as possible, so you make the product as good as possible.

• 12:22Speaker 01

  Then at some point, you reach maximum user, you know, numbers, and it's now about getting as much profit from the users as possible. So you like cram it full of ads, you like add these features that will hopefully make it, you know, more profitable, which maybe actually makes it worse. And I just feel like we're in the Enstitification era, where Google Maps just doesn't work as well as it used to, like in a bunch of ways. You can like put paste in an address and it'll just like take the first word of it and then send you to the wrong place, even though the full address is in there. And it's like, what's going on? This used to not happen. And it's this incredibly crazy-making and kind of dispiriting feature of modern life

• 13:02Speaker 01

  that we are just, we have to use these products because they're the only ones around and they're just getting worse in obvious ways and we just don't feel like we have a choice. And then the only thing that's not enstitifying, it seems, is AI, where it just does get better and faster and cheaper and just, it's kind of like amazing how much steady progress there's been. But then that comes with this like terrible risk and cost to society. And so, yeah, I feel kind of bleak about it, but I'm still, in my heart of hearts, like dispositionally optimistic, kind of about humanity's potential to pull together and do amazing things.

• 13:44Speaker 01

  And I think we just like need to change the structures and the systems to deliver better technology that will actually make us, yeah, happier, healthier, wiser, more democratic.

• 13:55Nathan Labenz?

  One of the funny things that has kind of happened in the last, I don't even know, 24, 48 hours online, has been a sort of brewing of the stochastic parrot meme, which seemed to be a comfortable, dare I say, safe space for a lot of more leftist thinkers, including even people who I would say should have known better because they had all the fundamental knowledge of AI that they should have needed. But we've heard a lot over the last couple of years from all sorts of people on the left that basically like, eh, it's fake, right? They're just hyping their stuff, this is nothing, it's just tech companies trying to

• 14:37Nathan Labenz?

  raise money or boost share prices or whatever, all that sort of cynicism. Why do you think that has been so prevalent? Do you have a theory of why the left has buried their head in the sand broadly on AI?

• 14:49Speaker 01

  Yeah, I think there's no one explanation, but a few. One is that the most prominent influential people in AI who are also on the left have taken this very hard line, the stochastic parrots paper being the kind of quintessential example. And people on the left will just like defer to folks who have, you know, PhDs and apparent credibility and then also share their values. And so that's been a dominant view. The left has also been like, you know, prior to Bernie running in 2016, really only powerful within academia. And academia has been hostile to this in a lot of ways, like AI being real.

• 15:30Speaker 01

  And there's maybe some like humanities versus like STEM antagonism happening there. And then I think crypto and NFTs and the metaverse and just in social media, like there was a lot of hype about those being transformative and positive technologies. They were either not transformative and not positive or transformative and negative, as I think of the social media and crypto by and large. And so, a lot of people just pattern matched to that. And we're like, oh, these tech people are just full of shit. They're just always saying that this thing is going to save the world. And then it's like, you know, B2B SaaS or something.

• 16:13Speaker 01

  And they just, yeah, like got stuck in this frame. And then I think there's just like some amount of cope and denial because it's just, it's terrifying to consider that like, these companies could make machines that could replace us. And even if you don't believe in like the full-on extinction risk stuff, just that alone for your job, like being unemployed is terrible. It's really, really bad for you. And then having that happen, like society-wide scale, very bad for society. You know, Nazi Germany rose out of that. Like, it's reasonable to be very concerned about this. And then like the bubble argument has also been very popular on the left and like Ed Zitron is like the main guy who's promoted that.

• 16:56Speaker 01

  And I think just similarly, well, what one is that Ed has been very persuasive and effective at reaching people, but he's just either doesn't know what he's saying or he's lying because he's constantly saying things that are just not true and like so easy to pick apart if you know anything. And again, in my chapter on the bubble, just I talk about like, you know, like it's a comforting story and it means you don't have to do anything. Whereas like my version of reality is that these companies are trying to build machines that will replace us. They might succeed and that would be disastrous for so many reasons, so we have to stop them. And that's really hard. We have to get organized. And I think that is a message that could work and I really hope it does, but it's,

• 17:39Speaker 01

  you know, one that requires you to not just like read about stuff and post and like, you know, do things in the world. And so, I think there's like a kind of overhang where there's a lot of people who have concerns and, but they don't want to get yelled at by sharing them on Twitter or Blue Sky. But I do think there's this kind of mismatch and we're starting to see the dam break and that's been really encouraging. And I'm hoping by laying out the whole case from start to finish and just also showing that I'm like coming from a similar perspective, we can like, you know, get people to, on the left to like take this more seriously. Because once you do, it's like, oh, capitalists are trying to fulfill the lifelong dream

• 18:21Speaker 01

  of turning capital into labor without the intermediary of workers. And in so doing, take the share of returns to capital to 100% and labor to zero. Like, seems pretty bad. Like, I think that's bad for, from almost any perspective, like maybe some libertarians are into that, but I think the libertarians are still concerned by and large. And so, this feels like a pretty easy one for the left to be like, no, this is a real thing, we should get on board with it and, you know, not let them build those machines.

• 18:48Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:54Nathan Labenz?

  Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant, who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2 Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training.

• 19:39Nathan Labenz?

  In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week.

• 20:20Nathan Labenz?

  Go to athena.com/cognitive and see who they'd pair you with this month. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agent's most important tool: web search. If you're like me, your agents are often conducting hundreds of searches per day. And while the unit cost is small, over time, this does start to add up. Before starting to use Parallel, I calculated that my agent's search bill would total roughly $500 this year. The good news is that I just had my agents conduct a systematic test, and I found that Parallel's fast mode, which costs just $1 per 1,000 queries,

• 21:05Nathan Labenz?

  worked just as well as my previous default provider. And switching to it will save me roughly 80% of my search bill going forward. Parallel's infrastructure is enterprise-grade, and their suite of APIs offers a range of Pareto optimal options that allow you to choose the right balance of quality, cost, and speed for your needs. So whether you're building voice agents that need 200 millisecond latency or long-horizon agents that need deep research, adding Parallel to your agent's toolkit is a no-brainer. Get started for free at parallel.ai/tcr. That's parallel.ai/tcr.

• 21:45Nathan Labenz?

  I'm interested in how you conceive of what it is that the companies, and to the degree you want to zero in and speculate on like individual executives at the frontier AI companies. What do they really think of themselves as doing? I do agree that like in setting out the, excuse me, in setting out the mission to create AGI, which they define as something that is better than humans at the vast majority of economically valuable work. It does say right there on the tin that this is like a human replacement, or at least like a human substitute technology. But then there's the other part of the mission, which is,

• 22:30Nathan Labenz?

  and to make sure it benefits all humanity, right? So typically, when people ask me what I think, I start off by saying, I don't think that they are well modeled as just trying to get as rich as possible. I think they're a little more utopian, ideological, something else other than just purely profit motivated. What do you think? How do you think of what they really want?

• 22:59Speaker 01

  Yeah. Well, this also reminds me of another reason why the left is not taking this so seriously. The most prominent people talking about existential risk from AI and AGI and superintelligence, it's like Elon Musk, Sam Altman, and then Eliezer Yudkowsky, Nick Bostrom. Like, these are people the left does not like. There's a mutual distaste. And so I think that negatively polarized a lot of people. But to answer your question, I think that you're right. Yeah, it's like, my model of it is the people who really started these AGI companies were chasing mission. And then they kind of toiled in obscurity

• 23:40Speaker 01

  until they made enough progress that the profit seekers were like, holy shit, this is amazing. And then they started investing massive money into it. And the leaders of the companies are still motivated by some kind of idealism, sense of inevitability, sense of like megalomania, you know, Messiah complex. You have it.

• 24:01Nathan Labenz?

  Better us than them.

• 24:03Speaker 01

  Yeah, exactly that. But then they're pressured by these investors who, you know, when Sam Altman was fired, it's a bit unclear like whether he's like, oh, I didn't even want to come back, but people like kind of asked me to come back. And it's like, eh, I'm skeptical of that. But the investors, and specifically, I think Thrive Capital, was a big, big player in restoring him. And that just makes sense, right? Like, once you invest billions of dollars into a company, you're going to have a lot of interest in what happens to that company, who's leading it. And having a bunch of safety-conscious people who want to, you know, replace the CEO who's presided over this meteoric rise, like, that's just not going to fly.

• 24:47Speaker 01

  And this is a big part of why I'm so pessimistic about things going well under the status quo. Like, setting aside just that it would be really difficult to safely and democratically introduce AGI into the world under any circumstances, like, and because it's a universal labor-replacing machine, it would just really turn society upside down to have that exist. But then to have it happening like as fast as possible with like minimal regulation in a country that is cutting the social safety net or adding work requirements to Medicaid, like, and then all the other countries wouldn't have a chance to even tax the companies that are putting their people out of work. That's just like a pretty, to me, like, obviously really risky proposition.

• 25:32Speaker 01

  But yeah, the people leading these companies are not really like profit-maximizing. Greg Brockman has that diary entry where he's like, what will take me to $1 billion? Which is a pretty crazy thing to write in your diary of, you know, the nonprofit you co-founded. And now he's worth like 20-something billion. But I think, yeah, Sam and Dario and Demis and Elon, I think are all motivated more by power, you know, trying to be a great man of history. We can go into the, and you want to break down the individuals, and I think it's hard to generalize because they're all unique. But to, you know, close out an overly long answer, I think it is largely not about money. It's about being in the room where it happens.

• 26:14Speaker 01

  That's what Eliezer told me when I interviewed him back in 2023. And like, people just want to be there for creating AGI, creating superintelligence, because that is like where humanity's fate lives in their mind. And that might be true. And they want to be in the room where it happens, influencing how it happens.

• 26:36Nathan Labenz?

  Yeah, I think Sam Altman has spoken remarkably candidly about this a couple of times in the context of describing what it was like to be in the room when the first reasoning demos were shown. And he said, I forget his exact wording, but he said it's happened a couple of times now.

• 26:54Speaker 01

  Like pushing back the frontier or something, the veil of ignorance.

• 26:57Nathan Labenz?

  Yeah, the curtain.

• 26:57Speaker 01

  Which is not exactly what that was about, but.

• 27:01Nathan Labenz?

  But yeah, there's something, and I am sympathetic to that in the sense that I think it is extremely exciting, even intoxicating to be in on the secret. And I do, yeah, I can understand that to a degree. I guess, zooming out, and I am interested to hear your takes on individual companies and their cultures and the individual people at the top that are shaping those. How much do you think, you mentioned the term overhang, and that got me thinking about, I'm torn or ambivalent on this question. It's probably both is always the answer. But on the one hand, I do feel like AI broadly is inevitable

• 27:43Nathan Labenz?

  in the sense that we have web-scale data, we have web-scale compute. And in the presence of those things, I feel like a lot of algorithms ultimately can work. We found one main one and a bunch of derivations of it that work. And so I feel like we're getting AI absent some sort of civilizational reset that means we don't have web-scale data and web-scale compute. And I'm not excited about that proposition. We're probably getting some AI. But then the shape of AI, the conditions in which it's introduced, like the measures that are taken or not taken to make sure it goes well, all of that stuff seems far more contingent. How do you think about like, how much is inevitable? And where we might be able to draw lines?

• 28:28Nathan Labenz?

  And then you can go off in any number of directions in terms of the influence that individual people or groups are having on the direction we're taking.

• 28:37Speaker 01

  Yeah. I mean, I think it's inevitable that we will, as a species, continue to use deep learning to make AI models that do things, like new things, right? The, in the book, I separate the obsoleting project from other types of AI. And that's my reframe of the AGI industry, because they're trying to render us obsolete. And you mentioned OpenAI's AGI definition, and Dario Amadei has a quote that's saying like, AI is not like other technologies, it's a general substitute for human labor. And that feels like pretty different. And we've taken AGI to be synonymous with like what AI is, because that's, the companies that have been trying to build it

• 29:22Speaker 01

  have been the best at building highly capable and autonomous systems, and they're driving the entire world economy now. But that's not like the only type of AI we could have, right? We could just be like, we have all these diseases and medical problems, and like, can we build alpha-fold type systems for specific problems that we have? And like, it's obviously, I get the vision of like, build the superintelligence second, you know, solve intelligence, use that to solve everything else, was DeepMind's mission statement for a while. It's like the ultimate techno-solutionist fantasy. And from a purely technical perspective, like, yeah, I can see how you could like use this thing to invent all kinds of wild and transformative new technologies.

• 30:06Speaker 01

  But I think that you can't like solve, you know, ethics or ideology and politics in the same way. And.

• 30:17Speaker 01

  Yeah, I think it's like this kind of revenge of the STEM people on the humanities or something where it's like, we don't have to like learn history or philosophy or political theory. We can just build the machine that's smarter than everybody and then ask it what to do, which I think is a really

• 30:36Nathan Labenz?

  That was one time the OpenAI's business plan, as I'm sure you're

• 30:41Speaker 01

  Yeah.

• 30:42Nathan Labenz?

  well aware.

• 30:43Speaker 01

  Ask the AI how to become profitable, right?

• 30:46Nathan Labenz?

  Yeah. And he said that, that was an Altman quote. He said that and there was a laugh in the audience. And he's like, you laugh, but I'm not really joking.

• 30:53Speaker 01

  Yeah, yeah. It's, it's just like the ultimate like question mark, question mark, question mark profit thing, which is like, yeah, just make the super intelligence. But yeah, I think that we can stop the obsoleting project because it's really just a handful of companies in two countries. And only one country has really advanced the frontier since ChatGPT came out at least. And it just costs so much money and it requires the most advanced technology in the world, these like AI chips, which are only made by it, you know, like individual companies control different parts of the supply chain, as your listeners probably know. And I think, you know, like forever, can we stop this? Like, I don't

• 31:36Speaker 01

  know. Hopefully, we're around for a very long time. And I, my position is not that we should never build AGI. It's just that it should happen with strong public buy-in and a scientific consensus where it can be done safely. We can get into what that looks like later if you want. But I think that, you know, for your listeners, it's like, right now, we're kind of feeling like trending towards just banning AI across the board or something, which would be very hard to actually do. And especially with like open weight models and yada, yada. But the kind of like backlash is really intense and kind of undiscerning. And I'm kind of hoping to separate the obsoleting project

• 32:17Speaker 01

  from other types of AI and really stigmatize the obsoleting project because it's, it's risk and undemocratic nature. But then like, save the baby from the bathwater with like, you know, these kind of specialty systems, which can be used to do amazing things. And my position is that like, we can get way more of the amazing stuff if we have like a more active role for democratic control and deciding like what gets done. So like drug discovery, you know, people are like, oh, AI for drug discovery. Well, monopoly patents mean that people will still try to just discover drugs that will be profitable, which won't be ones that cure people as much as ones that, you know, treat some chronic thing, or male pattern baldness, or like, you know, these things that are not as socially

• 33:01Speaker 01

  important. And to actually get like, the best from AI assisted, you know, drug discovery or AI assisted FDA trials, like matching people, you know, deep learning systems are very good at that. To get the best of it, you have to like reform the systems and change how, you know, drugs are, which like how we decide which drugs are made. And I think we should use like a prize system, which, you know, you kind of pick the drugs that you'd want to see in the world and then award money and then just make them at generic costs once they're, once they're developed. And so I kind of like, it's a judo move, but I'm kind of like, we aren't going to get the booster world by letting it rip. Like

• 33:44Speaker 01

  the one that they depict of like amazing transformative cures for everybody. And the best way to get there is to like actually take a strong position against being replaced and then use industrial policy and like the operation warp speed type approaches to build the types of technology and scientific discoveries that will actually lead to the most public benefit.

• 34:08Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 34:13Speaker 03

  I've got a text.

• 34:14Speaker 04

  The summer might be almost over, but a new bombshell has entered the villa. Meet the voices of Deepgram Flux TTS. I'm Drew, low key casual.

• 34:23Speaker 05

  I'm Alexis, an upbeat morning person.

• 34:26Speaker 03

  I'm Haley, always cheerful.

• 34:28Speaker 04

  Flux TTS has some real personalities that are ready to speak. You can interrupt, pause, and keep talking without losing the plot.

• 34:36Speaker 03

  Fancy a chat? Come try all the voices free through September 12 at deepgram.com/keep talking. Terms apply.

• 34:43Nathan Labenz?

  This episode of The Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. Outsystems is helping their customers build, modernize, and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust Outsystems for their mission-critical apps. And we're not talking about theory, we're talking about massive operational wins. Take, for example, a leading global retailer that was struggling to keep up with their rapid expansion. They used Outsystems to build AI agents that automated their invoicing

• 35:26Nathan Labenz?

  and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical debt. Their engineers were spending their weeks manually firefighting by reviewing platform logs. Using Outsystems, they built an AI log review agent in just five days. And that one agent now saves them over 1,600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and

• 36:07Nathan Labenz?

  automate complex financial workflows, delivering secure, personalized experiences at a pace their competitors can't match. Outsystems gives you the flexibility to deploy how you want and the governance to do it safely. Stop chasing the hype and start owning your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr. Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind

• 36:47Nathan Labenz?

  everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses. I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days,

• 37:30Nathan Labenz?

  I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

• 38:11Nathan Labenz?

  I have a lot of different directions I want to go, but I guess maybe part of a question that I wrestle with, it's a little bit hard, of course, to define some of these terms and what are the boundaries. But like, I get so much value out of using AI on a daily basis. It saves me an unbelievable amount of tedious work and it allows me to do a ton more than I otherwise could. And it, in a way, has replaced, at least like counterfactually, people I would have had to hire in theory, whether I would have or not, I'm not so sure. But like, there's definitely a lot of work happening in my life by AI that on some counterfactual level has substituted for human labor. And I'm

• 38:56Nathan Labenz?

  like, I think that's good, so far at least. Like, I, I, do you, how do you think about like where this goes from bad or goes from good to bad? Because I think there's a strong argument, and I do want to give it its kind of strongest articulation that like, you go back not that long in history and everybody was tilling fields, right? And it was like pretty shitty. And now we have certainly some problems in our modern agricultural system, but like, one problem we don't have is scarcity of food. And there's like plenty of, with 2% or whatever the population, we can feed ourselves and everybody else is like able to do other things. And almost everybody agrees that's almost overwhelmingly

• 39:39Nathan Labenz?

  good, let's say, even again, though there are problems that remain. Can we not have a version of that with AI where we're all elevated to our own little executives of our many AI corporations or something? Are you at all sympathetic to the

• 39:56Nathan Labenz?

  Or maybe we'll work a lot less. We had the Keynes thing too from 100 years ago that was like, we're supposed to only be working two days a week at this point, but we're not.

• 40:03Speaker 01

  Still waiting.

• 40:04Nathan Labenz?

  Yeah.

• 40:06Speaker 01

  Yeah. No, the faster AI goes, the more I work, it turns out. But I, yeah, I mean, automating labor is what's allowed humanity to go from everyone, more or less, being very poor to some people being rich at least, and like, living standards and all kinds of stuff going up dramatically after the Industrial Revolution. And so it's like, my position is that we, not that we never automate any labor, because like, I don't want to just have a stay at this kind of like, level of development. But trying to automate all labor is pretty different. And where the line is, is like, it's not super obvious. I think that, you know, my position is

• 40:50Speaker 01

  we should freeze frontier development right now and not resume without the buy-in and safety I mentioned. But it's like, three years ago, FLI organized this pause letter, you know, after GPT-4 came out, I guess three and a half years now, there was like a six-month pause on development. And that was framed around a bunch of things, but risk was a big part of that. And like, obviously, wasn't super risky to build the next iteration of LLMs, at least from a like, existential perspective. Like, there were harms, you know, there's like chatbot psychosis and suicides that, you know, OpenAI's decisions contributed to, which I document in the book. And I think that should, you know, we shouldn't lose sight of that. But that's not really what

• 41:32Speaker 01

  that letter was about. I think now we're kind of in this position where it's like, we just froze what we had today. It'd be still a lot of disruption from just adopting GPT-6 and Fable 5.1, you know, in the economy. And the job effects are not super easy to see. And it's like, a lot of it's not people getting fired, but people just not getting hired in the first place, kind of as you described. And I'm not like, saying that we should go back a few generations, you know, that might actually be the right thing to do from like, a social welfare perspective. I don't know. But it's just much harder than stopping advancing further. And so I think we should focus on that first and then kind of reevaluate with like, what we have. And it'll

• 42:13Speaker 01

  take a lot to, you know, regulate the AI that we have today, and a lot of thinking to get that right. But yeah, I mean, on the CEO of our own corporation thing, it's just like, some people will do well in that system and enjoy it more than the status quo. But I think it's just like,

• 42:36Speaker 01

  a lot of people will be left behind. And not everyone wants to like, the experience I've read and heard of, like managing these fleets of agents, it's like, can be like, really bad. You can feel like the opportunity cost of not kicking off another run and people like working more and more. And, you know, they're competing with other people who are adopting it really quickly. And so I'm kind of looking around at like, this world where people are just like, kind of burning themselves out, running these like, agent fleets and getting more done, but not necessarily like, being happier with it. And I don't know, I'm just like, it doesn't seem like the the future, the good future that you're describing. And

• 43:19Speaker 01

  this is bracketing all of the risks and other social harms of like, widespread adoption of this tech. And then the power concentration and wealth concentration. It's like, some of, some people will be way better at running the AI companies. And the AIs will probably, if they're that good, be better at just running without any humans involved at all. And then like, who is owning those companies and who is accountable for what they're doing? And like, and then you have like, these multi-agent dynamics where, you know, the hugging face hack now, and there's like 1,200 agents who are involved with that. But the world you're describing, it's like, there are multiple agents running for every person, interacting with multiple agents for other people and companies all the time, in ways that we can't

• 44:03Speaker 01

  monitor effectively. And producing like, who knows what kinds of interactions and effects. And that just feels like a much less legible and stable world. And we're already seeing like, OpenAI is like, I tweeted that it was a loss of control event masquerading as a company in response to the fact that some, like, three random people use Claude and Codex to get access to chat GPT employee accounts. And they could have gotten access to like, I think their main codebase. And it's just like, this is, and then multiple agents forums that have broken out of the company with or without their knowledge, or some amount of covering up, some amount of cluelessness. And, you know, OpenAI is probably one of the companies that's

• 44:46Speaker 01

  adopted this technology the most. And it means that they just don't know what's going on nearly as well as they would have a few years ago. And so I think that the world of like, these agents being widely adopted is one of like, yeah, confusion and chaos and lots of unpredictable, but many, many negative consequences that we're already seeing like hints of right now.

• 45:10Nathan Labenz?

  What do you think the, if you envision a social contract that you, you know, what's your most positive vision of the future? If you're not going to roll AI back from where it is now, and I think you've got some interesting proposals around, you alluded to like, freezing the frontier. But again, that still allows for a lot of diffusion. I totally agree, this will probably, like seemingly all recent technologies, probably amplify inequality. Hopefully it brings up the bottom, but probably the ratios also continue to climb. I've lived through a medical emergency, thankfully, in the AI era, and I did get

• 45:53Nathan Labenz?

  an unbelievable amount of value from using three AIs in triplicate and feeding in my test results, and it was my son's test results. But yeah, just unbelievable amount of value from that sort of thing. So I do feel like, I want, and any good version of the future for me, there's like, AI doctors for all, which notably, OpenAI has done a pretty good job of making their health product free and unlimited to people, which is like, pretty cool. I think as OpenAI moves go, that ranks near the top of my list. What's your positive vision for where we want to be in three to five years if we freeze the frontier and allow other things to continue? Like, what does good look like to you?

• 46:35Speaker 01

  Yeah. I guess like, in the US, I would like to see the left winning elections and building kind of a third New Deal with the Great Society being the second. So, you know, Medicare for all, maybe a jobs guarantee administered locally. And I'm really bullish on investing a lot of money and more importantly, kind of like institutional and state capacity in developing technologies and medical treatments and things that we like, actually really want as a society. So, one idea I'm like, playing around with is cures for all, where the government

• 47:16Speaker 01

  kind of treats diseases like they're all COVID, and it's all Operation Warp Speed. You'd obviously have to prioritize based off of, you know, tractability and, you know, the disease burden and, you know, other factors. But then when you're there, like, using the whole government approach, using advanced market commitments, you know, vaccine trials, and, sorry, I'm spacing on the word for, uh, challenge trials, like where people deliberately expose themselves

• 47:40Nathan Labenz?

  Human challenge, yeah.

• 47:40Speaker 01

  Yeah, human challenge trials. And, you know, developing like, the stuff that the companies often promise, but just going for that directly. And then using AI where it's helpful. And then, in a lot of cases, it's really about the institutions. And then, you know, this would be enormously beneficial to the domestic, you know, US. But then you could also, you know, make the, make the cures like freely available around the world, or provide them at cost or whatever, and make the world healthier. And also like, restore our standing in it after some well-deserved drops. And I think that like, the US is so wealthy, and that wealth is just concentrated so tensely in the top. That's been happening for decades,

• 48:24Speaker 01

  and AI and tech is just accelerating it further. And so I think just really large redistribution is justified and like, on just democracy and power grounds. Like, right before we recorded this, we got the news that Larry Ellison's bid to buy Warner Brothers, which would give him and his son control of CNN, HBO, Warner Brothers, in addition to CBS. And yeah, and also TikTok, and then Elon and Twitter. Like, we're in a really dire situation where these oligarchs are buying the most important media properties in the world, and then using them to push their agendas. And this is like, I work in media, I think media is really important.

• 49:06Speaker 01

  And yeah, it's, it's not just a matter of like, oh, they have more money than they need, and like, other people have not enough, like, but it's like, actually a threat to our democracy to have people being that rich. And so, and yeah, and I would like to just see as much decoupling between, you know, wage labor and living a good life and building like, a proper welfare state. And then also a restoration of USAID and, you know, rejuvenation of it. Because I think that, you know, global poverty is incredibly important. And the cuts that Elon led, I also document in the book, and just like, the deaths that those caused, just like, one of the greatest crimes, maybe the greatest crime of the 21st century. And yeah, just kind

• 49:51Speaker 01

  of like, a restoration of like, I don't know, just like, building a pro-human society that is like, truly egalitarian and

• 50:00Speaker 01

  One that like wants good things for people and like a true like that sounds so cheesy, but like right now we have an administration that is just staffed with like seemingly the worst people on the planet and seems to either not care about what happens to other people or want bad things for them and is just enriching itself at the expense of literally everyone else. So the opposite of that would be really nice.

• 50:27Nathan Labenz?

  I'm with you that the destruction of USAID is incredibly shameful. I've been a Elon defender in many conversations over time and that's one thing that he's done that I've never defended. On the question of, there seems to be a little bit of tension between a potential jobs guarantee and decoupling one's ability to contribute to the economy from one's right, as we might imagine it, to a decent material life. Do you feel like, are you a believer in the intrinsic value of work? Or people need work for purpose or for something to do? I'm a little bit more of the mind that, I don't know, I'll

• 51:10Nathan Labenz?

  bet on the working class to spend the piece dividend is what I told, oh gosh, doesn't matter, Jake Sullivan, whose name I should definitely know at the tip of my tongue, but he was, at the end of an hour with him, we were talking about China, he was like, we didn't even get into jobs and what people are gonna do. And I was like, I bet on the working class to spend the profits. You worry about China. What do you think, though? Do you think that we need jobs indefinitely?

• 51:36Speaker 01

  I think, I mean, one, I will say, like, I need to give all of this more thought. I've been so neck deep in AI world that like figuring out, I kind of want to make sure we're not all replaced by machines before we get into like planning the, you know, third New Deal. I think that you can have a jobs guarantee where, you know, it exists for people who want it, but then also make sure that people are getting healthcare and like the, the education, housing, like just the necessities covered because, you know, we, we have an incredibly rich society and those resources are not being put to good use. Like, rich people, they just want to be richer than each other and they maybe want to use power in the, in the world, but most of them don't seem to even care about

• 52:21Speaker 01

  that last part. They just kind of want to like be richer than their friends or something. And so just, you could just tax them really aggressively and they won't like it, but they'll still get to like have more money than the next person if you do it the right way. And so I don't think we need to choose. I think that work does seem to be, it's just a thing that's very important to a lot of people and I think we should try to not have so much of our meaning tied up in it. But it's also just like, you know, look at polling, it's one of the few issues that polls at like 80% or something. And UBI is polled very badly. And so, I don't know, like I believe in democracy at a deep level. And so I think part of building, you know, the third New Deal is going to be running

• 53:01Speaker 01

  on popular things and figuring out how to make them work. And, and yeah, just like, jobs exist for people who want them. And there's also a lot to do. There's a lot of like climate, you know, like green transition stuff you could have people do. The New Deal, like, I think it was James Baldwin was a writer for the, one of the programs in the, in the New Deal and they documented, they like recorded a lot of important things that were relevant for like preserving culture and history and like lots of really cool stuff came out of that. And, you know, it just, pay artists to make art. Like, there's so many people who want to create things and our society has decided to devalue that as it dumps unprecedented amounts of money and building machines that can replace all

• 53:45Speaker 01

  of us. It just seems like backwards. And so, I don't know, like this just seems like we're doing almost everything inverted from how I think we should be doing it.

• 53:56Nathan Labenz?

  I'm interested in your response to an argument that you sometimes hear around the relationship between invention and democracy. People sort of just make the very short observation that like, in general, people are allowed to invent stuff and there, it's not like, we don't put every new invention to a vote. And if we did, we might not get very many inventions. We might freeze a lot more than we would like to freeze in time. Friend of the show, Dean Ball, also on Ezra Klein, talked about how he wonders if, I think it was on Ezra Klein, if today we would have the stomach for the introduction of

• 54:40Nathan Labenz?

  the car, which was like disruptive in its own way. And it was sort of cars and horses on the roads together and like they weren't safe at the beginning and they're still not entirely safe, but they were way more dangerous then. Are you sympathetic at all to that? Or how would you answer the idea? And I guess there's another question around democracy, around just how well is it functioning in general? The, for better or worse, I do think the president was in fact legitimately elected. So I do wonder like, how much can we really, and I'm a big direct democracy guy, by the way, in general, but I don't know that we can just fall entirely back to as kind of the end all be all decision maker here because

• 55:21Nathan Labenz?

  it seems like we've got plenty of examples of bad decisions being made by publics and also might not be willing to embrace enough change to really see the future go where at least I would hope that it could go over time.

• 55:36Speaker 01

  Yeah. Well, first I'll say, I live in the only city in America where most people don't have a car. And it also happens to be the best city in America. So, I don't know, cars like, I don't know if they were good on that. Like, probably, I don't know though. But that's not important for, for my particular point, which is like, I think permissionless innovation makes sense for most things. Almost all innovation, I think that should be the default. But building a universal labor replacing machine would have a profound and irreversible effect on everybody in the world. And so I think it's reasonable that everybody has some say in whether, when and how that happens. And,

• 56:19Speaker 01

  you know, broadly, like my position is AI, at least what they're trying to build, the obsoleting project, like that is unique. It is different from other technologies. And so it should be treated differently. And on your point about democracy and, and, you know, Trump being this counter example, Trump is a product of a broken democratic system in the United States where, you know, you have the Electoral College being the most obvious example where he won in the first place, but lost a popular vote. But more importantly, you have counter majoritarian institutions like the US Senate with the, you know, crazy way representation is apportioned and gerrymandering to a lesser extent. And then the Supreme Court being these lifetime appointments and the Republicans stole a Supreme

• 57:00Speaker 01

  Court seat. People should remember that. And so, yeah, I think we have, and then the first past the post two party system creates a political system that the majority of people don't like. I think it's like two thirds of Americans are not happy with their current level of representation. And so, democracy isn't just like, is there an election that decides with also some other weird stuff tacked on to it, who leads a country? It's like, do people have meaningful say in the power that affects them? Like, do the governed govern? And the US is just not great on this. And a lot of other countries are, are better on it. And, you know, we're the longest running democracy in some sense,

• 57:43Speaker 01

  although really wasn't a democracy until the 1960s when everybody living in the country got the right to vote. And, like meaningfully. And so, yeah, I, I think like proportional representation is a thing I talk about in the book as like a better way to elect Congress where, you know, people would have like larger districts with multiple members and then the top five vote getters would be in office. And then you'd have probably like five parties instead of two. And then people would have parties that represent their interests better. And that would be like a huge change, but so, so like better at representing people. And then Electoral College is one state away from being gone if Pennsylvania,

• 58:26Speaker 01

  I think, signs the interstate compact to just use a popular vote. And so, my pitch for democracy is like one that is more like well realized. And this also extends to like democracy in the workplace. So I think it should be a lot easier to form a union and a worker cooperative. And I think, you know, in this country, we accept that, you know, it's like the government should never, you know, impede on us and censor us. But our employers, where we spend like eight hours a day, five days a week can surveil us and control what we say and do all kinds of things that we would find incredibly authoritarian in another context.

• 59:06Nathan Labenz?

  Record all of our keystrokes and mouse clicks and use it to train AIs, for example.

• 59:11Speaker 01

  Yeah. Yeah. And, and so, yeah, I think like democracy should extend to more parts of our, our society and our life. And I think that this will produce on net better decision making. But then it will also just create a more empowered citizenry where like it provides something intrinsically to, to have say over your life. And I think people right now, especially in the US, just feel incredibly disempowered, incredibly unheard. And I think that's reasonable, right? Like, right now, we have a government that's uniquely insulated from public opinion through these kind of majoritarian institutions, through the fact

• 59:52Speaker 01

  that it's this lame duck president and just one who like, does not seem to care very much about things beyond the ballroom and corruption and, yeah, like the stock market. But, yeah, so I don't know. I, democracy is great and America should be one.

• 1:00:10Nathan Labenz?

  Change gears a little bit on the notion of people being empowered. And then we'll, I'm sure we'll circle back to some of those bigger themes before we end as well. But I think you have an interesting argument in the book for how the AI researchers should understand their position today. I think you had a column from just a couple of months ago, right? That was like, AI researcher power is reaching its peak. And I actually don't know too much, but you talk a little bit about the vote at DeepMind or the sort of organization effort at DeepMind in the UK to bring about a union to the DeepMind staff there.

• 1:00:55Nathan Labenz?

  Tell me about how you see, I'd be interested to hear that story because I really don't know a lot of the details of it. But then beyond that story, give me how you see the lay of the land in terms of the power that the employees at the frontier companies have, whether or not they realize it, how you think they should use it.

• 1:01:11Speaker 01

  Yeah. I mean, so the DeepMind thing, Google has been signing these deals with the military, US military, Israeli military, and this is creating a lot of pushback from workers in the UK, the DeepMind Division had a union vote. It's a, I don't know British labor law, it's pretty different from the US, but it seems like something like 300 of a thousand people in the bargaining unit were supportive and that, and Google is not recognizing it. And they're not demanding better pay or working conditions, but instead, policy changes on who Google is serving and how. And I think that's pretty interesting because, you know,

• 1:01:52Speaker 01

  most unions are like, it's about getting better pay and working conditions for the members, which is, you know, a reasonable thing to want. And these situations, like, people are paid very well. I mean, they work a lot, but they often want to. They have like good perks or whatever. And for the broader point, like, the industry is raising to replace all of us, but they're starting with their own workers, specifically the engineers and the researchers who are training the models. And the hope is to achieve recursive self-improvement, where the AI can fully train the next generation and make it more capable. And then you can

• 1:02:30Speaker 01

  have this really fast loop. And I think that means that we have a ticking clock here. I mean, the workers have a ticking clock where their power will, you know, it's near its peak because they still command incredible salaries and they're still needed. But if they're not needed anymore, then they won't have any leverage. And then that will happen to the rest of us, which would be very bad. And so I think people in the companies are starting to see this, but probably not as much as they could. And one thing we've been seeing is like, if you oppose what's happening at these companies and you work there, you should just quit and then go public. And Jacob Coxon did this with like incredible

• 1:03:12Speaker 01

  fanfare and it really moved the conversation like nothing ever has. And, you know, we see a few other examples since then. And, you know, Coxon deserves credit for this. And I think it's like, you know, I was a whistleblower. I think it's sometimes the right thing to do, but it might be better to stay and organize your coworkers who also care about safety and try to form a union. And in so doing, you can bargain for, in the US, you can bargain for safety. And this is something that airline pilots, I believe, did. And this is like how the FAA was formed, I believe. And you will have so much more power by being able to withhold

• 1:03:55Speaker 01

  your labor collectively. And you'll have protections in the union. And if you really want to slow things down, like, a big, dramatic, messy, union fight is a pretty great way of doing that. And it would be like, if it was truly about just safety, then it's also this clarifying moment where the CEOs will say, like, I care deeply about safety. I want to slow down. I want to pace the frontier. And then the workers could say, great, we want to form a union. Will you voluntarily recognize us? All we want is to make the AI safer. Also, by the way, if we're in our own unions, we can work with the other unions to collectively

• 1:04:37Speaker 01

  pace the frontier and it doesn't violate antitrust law. There's a precedent, I think, of the Teamsters doing, doing this across different shops. And so my guess is that these CEOs would not voluntarily recognize the workers. But if you're optimistic about your CEOs, you can just give them this opportunity to be like, great, here's this creative way we figured out to, to actually slow down. And I think that this is very promising. And like, a lot of people at these companies just don't have experience with that type of thing. But it's something I want them to learn more about. And yeah, like, you can have a lot of leverage over what happens through this, through this approach.

• 1:05:18Nathan Labenz?

  I don't know to what degree you have chapter and verse on this, but the biggest relevant offices are in California. Under a kind of California rules, do you know how much latitude or protection people at tech companies would have to organize? I would assume that they're protected from being fired for attempting to set up a union.

• 1:05:46Speaker 01

  Yeah.

• 1:05:47Nathan Labenz?

  Are outside union representatives afforded some rights or privileges to be able to come in and try to organize people? I haven't really thought too much about this, but it does, with all of the talk that we hear lately about antitrust and why we can't do these things because of antitrust, I've been saying, okay, the president should just say, we're not going to come down on you for antitrust violations for coordinating on safety measures. In absence of that, this is another pretty creative solution that I think would be very hard for anybody to argue with, although I'm sure we'd get the usual bad faith attacks. But to the degree you can, and I realize I'm giving you this in

• 1:06:29Nathan Labenz?

  an unscripted way, but I'd love to hear the double click of what you think that could look like and what people on the inside should know, such that if they are at all tempted about this, they could feel confident in taking, I don't know, what sort of next steps.

• 1:06:44Speaker 01

  Yeah. It's been over a decade since I took labor law, but there are protections against retaliation for organizing and then protections like once you're in the union as well. The protections are kind of weak, you know, like the back pay is often, like, you can be litigated for a long time. But there's really with these companies, like, it's just a very bad look to fire people for trying to organize a union around safety. And so I think that is in itself a lot of protection. And at OpenAI in particular, we see a lot of culture of people speaking out. So I think that, yeah, there's protection in that kind of like reputation management. And then in California in particular, workers can bargain at the sectoral level. And so like fast food workers can bargain across

• 1:07:27Speaker 01

  different shops, I believe. And so you have extra benefits there. And if people are interested, you know, I'm not an expert on this, but if they're interested, they can get in touch with me. My signal is garrison dot zero six. And yeah, like I know people who would know more.

• 1:07:45Nathan Labenz?

  In terms of what they would be organizing for, I think the book has quite a few different

• 1:07:54Nathan Labenz?

  interesting arguments that kind of take the, the assumptions or the, the hopes that people on the inside often have and at least give them a good shake. One is just like,

• 1:08:09Nathan Labenz?

  that the companies are serious about safety in the first place. I think it'd be helpful maybe to just review for a second, the history of safety at OpenAI, which you've reported on at some length. We've been through like waves of different regimes and leadership, and there's been a lot of turnover. And then I think you also have a pretty interesting take on alignment, basically as a mirage. Putting yourself maybe in the role of like, advisor to the hypothetical union leaders. Why would it not be enough for them to say, okay, we got 20% of compute committed to safety. And why wouldn't it be enough for them to say, oh, we're gonna solve alignment first and then we'll go do the thing?

• 1:08:52Speaker 01

  Yeah. Well, the super alignment team, as you're alluding to, was promised 20% of compute for that team and then got very little of it and nowhere close to 20% in practice. And I think to varying degrees across the companies, we've just seen a lot of promises made about safety and commitments that then get changed or broken. Once they start conflicting with commercializing or, you know, racing ahead. And it's because like, there isn't a counterforce. You have management able to unilaterally make these decisions and employees can push back and they can go public and they can, you know, whatever, like, you know, try and resist this in some way. But if they're not organized,

• 1:09:35Speaker 01

  you're just not going to have the, the means of actually changing the policy. And, you know, we've seen employees getting organized enough to change outcomes with Sam Altman's firing and then reinstatement. The employees coming together to sign that letter, like 90 plus percent of them signed, was like a really big part of that working. And obviously the dynamics there were pretty different. They stood to lose a lot of money on the, you know, sale of their shares. But yeah, I think workers can like decide, like, we really want these policies. We want these safety practices and we want them to be like binding in some way. And you have leverage by being able to withhold your labor, go

• 1:10:15Speaker 01

  on strike, do like work slowdowns, et cetera. And I think that we just can't take these companies at their word. And the CEOs are saying that they can't unilaterally slow down because they're racing each other and like, you know, they could, and it would actually be a pretty strong signal that you take these risks seriously if like any of these companies unilaterally slowed down. And it would put a lot of pressure on the others to do the same. And it would also help with the US-China thing. If like, you know, one of the parties unilaterally disarmed, that like really does actually signal like you care about this. And the US is in the lead, so it would signal it much, much more. And this is my understanding, like, one of the biggest, if not the biggest blocker

• 1:10:55Speaker 01

  to a deal with China, is they don't think the US is taking it seriously because we're barely regulated over here. And we have been the ones who started the race and will win it, as, you know, Trump has said. So, yeah, I think it's just about power. It's like, you just need hard power to actually get concessions. Because otherwise you'll get promises that will be broken as soon as they start to cost too much. And you can, yeah, just make it about only safety. And like in terms of ideas, like, You know, third-party auditors is being discussed and like making that not voluntary.

• 1:11:35Speaker 01

  You could also have like your demands be about the practices of like the lobbyists at the company. Opening AI, we've seen a lot of turmoil about the Leading the Future Super PAC, which is funded by Greg Brockman, was set up with the guidance of Chris Lehane, the chief lobbyist at the company, and then doing like incredibly dirty tricks, like false flag Twitter accounts and like calling for violence against AI employees, like, you know, funded by this Super PAC and related entities, and also going after politicians for daring to regulate the technology at all. And so I think this is like, you know, creating really bad dynamics for politicians doing the right thing on this.

• 1:12:16Speaker 01

  And the employees have a lot of leverage and have been able to get some concessions from Brockman about this. But yeah, I think that you can just see it as like a way to generally increase your ability to influence the policy decisions at these companies. And OpenAI has had a lot of safety leadership turnover and people kind of being disempowered and rotated around. And then we see like just these shocking breakouts and hacks and hijackings of various websites and companies by these rogue agent swarms. And, you know, it's like, is it an accident that the company with this like really shoddy track record of taking safety seriously is having all of this happen?

• 1:12:57Speaker 01

  And it's like now putting the entire industry's future at risk, which is good from my perspective. But like, you know, they're kind of messing it up for the rest of it. I mean, every company's had their agents hacked into somebody they weren't supposed to by now. But OpenAI really, yeah, like has been the greatest possible, you know, case for why more regulation is needed here.

• 1:13:22Nathan Labenz?

  Yeah, I do think your case to the insiders amounts in a way to, let's avoid the nuclear outcome, or at least that's my term for it. The nuclear outcome being like, we get the weapons, we don't get the civilian benefits. I do think there is an increasingly compelling case that's like, guys, people out there really don't like you. You're going to have to clean up your act if you want to have a chance of bringing the positive side of this forward. And that window might be fairly short because we're getting, the overdoing window is blown wide open and we're getting like all kinds of proposals

• 1:14:07Nathan Labenz?

  from all kinds of people and like, who knows what the political current is gonna kick up for us over the next couple of years. So take matters into your own hands right now and make sure that you are on the right side of key questions. And then potentially only by doing that, will you have the chance to really realize the upside vision that got so many of you into this in the first place? I think that's a pretty, who knows, but there's a decent chance that is accurate as a, as an assessment of things and is probably as compelling as a case as can be made, I think, to a lot of people who are not total doomers inside,

• 1:14:49Nathan Labenz?

  but are, but can see that the world is starting to sour on this whole.

• 1:14:54Speaker 01

  Give me your. Or can I just react to that quickly?

• 1:14:57Nathan Labenz?

  Yeah, yeah, yeah.

• 1:14:59Speaker 01

  Yeah, just like, it's sort of, if I was talking to the Trump administration, they would not listen to me, but similar to what I would say to these employees where it's like, if you're just moving this fast, there's gonna be a worse hugging phase with a body count. And then there will be very strong pressure to just shut it all down. And so if you want to actually continue and get all the upside that, as you were alluding to, like, you're going to need to slow down because otherwise your hand will be forced.

• 1:15:24Nathan Labenz?

  So now give me your case against alignment or your case that it is a mirage.

• 1:15:30Speaker 01

  Yeah, so I have a chapter of the book, Absolute, called The Problems With The Alignment Problem, where I explain, you know, the focus of AI safety historically has been on a solution to alignment, where like you can get an arbitrarily capable AI system to do what you want. This is like technical alignment. Sometimes there's discussion of like wanting the right thing, normative alignment. And I think this really understates the problem because you also have economic alignment and geopolitical alignment. And so I introduce the alignment polycrisis to kind of have all these layers and they interact in complex ways.

• 1:16:11Speaker 01

  And if you solve technical alignment, that makes you have a more useful product. And so the race can run faster and for a bigger prize. And then similarly, it's more useful as a weapon, as a means of projecting power around the world. And so it could make the geopolitical race worse. And sometimes, and like we've seen this with, you know, the biggest alignment intervention historic to date is probably reinforcement learning from human feedback, which was developed by Paul Christiano and others at OpenAI for safety reasons. It also happened to make LLMs actually conversational and useful, which enabled ChatGPT and everything that came after.

• 1:16:55Speaker 01

  And I think this is just gonna keep happening because of the dual-use nature of this technology. And so, yeah, like you have to look at the whole picture to understand what works. And this makes me much more bearish on technical alignment where like, if you solve it, you still have all these other problems. And this makes like a solution to alignment neither necessary nor sufficient to solve the problems presented by the obsoleting project. And then, you know, the answer in my mind is just stop, just don't, don't build it. And some of the policy interventions or technical interventions that would be helpful is like verification of international agreements.

• 1:17:37Speaker 01

  Historically been very neglected, but we'd be in a much better situation if all the money that went to technical alignment research went instead to having the means of verifying an international treaty on AI, which is going to be one of the bigger blockers to actually having a binding agreement here. And so, and governance just also looks a lot better as an intervention, just like having good regulations and policies and ideas in place. And then also the means of like actually bringing them into the world that could backfire if they're the wrong policies. But it's just like, you could at least solve this problem through governance, whereas you cannot solve it through alignment.

• 1:18:12Nathan Labenz?

  I do think it's a, it's an underappreciated and generally under-theorized domain, let's suppose. And the gradual disempowerment folks have pushed on this, but it's, I think, still underappreciated that like, even if you posit an AI that will do what you say and only what you say and only what you really mean, and not go like genie problem on you and do the paperclip maximizer thing, but really gets it and actually does what you, as an individual user or controller of the AI, would really want. It is still tough to envision like what the future equilibrium looks like.

• 1:18:54Nathan Labenz?

  I'm a little more inclined to like at least take some chances there because I do think also that would have been true about cars, for example. And I think it's like interesting thought experiment to say maybe cars weren't good, but I think most people feel like they're good. And we've got like problems, obviously, associated with it. But we also have like plenty of nice things that people really appreciate that they couldn't have imagined, I don't think, in advance of the cars being created. And if you just put it to people at the time, it was like, tell me what the future of cars is gonna look like. And if you can't, then I can't sign on to it. I do feel like that, that is a high burden and an unusual burden to put on world-changing technology. Again, it's fair to say this technology is different.

• 1:19:39Nathan Labenz?

  And that's a big reason that I spend all my time thinking about it. I'm again, ambivalent on this, but it is something that I think people in the AI space should spend more time, at least trying to do to figure out, to find for themselves how difficult it is to articulate like, what is it gonna look like? And how is it gonna work? And how is it gonna be good for everybody? And how do we have some sort of balance that you can expect to be stable over time? Those are really hard questions that. I guess are often just gated in people's minds because they're just focused on the technical alignment question first. And, and again, we don't have great answers there. So they're not necessarily wrong to be focused on that. But yeah, thinking past it, it is

• 1:20:21Nathan Labenz?

  still quite tough. There's one fairly long conversation with David Dubinout, who was one of the co-authors of the gradual disempowerment paper. And I think it is, it's a pretty tough thought experiment and a pretty compelling arguments by him that certainly we can't just take for granted that if we solve a couple upstream problems and everything downstream of that, we'll be fine. I think that's definitely not at all guaranteed.

• 1:20:45Speaker 01

  Yeah.

• 1:20:46Nathan Labenz?

  Who else do you think holds power today? One I'll propose to you and then you can run down whoever else comes to mind for you. One that the sort of capitalist class would like to point to is corporations. And the idea there would be, look, misaligned AI doesn't sell. Companies don't want rogue agent swarms happening. The market itself will discipline the AI companies on this front. And yeah, that doesn't necessarily deal with all existential risk. If something really were to go boom or go crazy, but maybe there's nothing we can do about that anyway, because that's just so crazy. In sort of

• 1:21:29Nathan Labenz?

  the bulk of scenarios, capitalism will discipline the companies will get pretty well behaved AIs, because that's what people will be willing to pay for. I should, there's something to that. If companies were to be a little bit more forceful in their demands or expectations. I think there's one thing to say the invisible hand. There's another thing to say, companies should maybe get opinionated about, we want to see some standards. We want to see some proof around what you're doing before we'll buy your AIs in the same way that they do that for other products, right?

• 1:21:58Speaker 01

  Yeah.

• 1:21:59Nathan Labenz?

  Grocery stores, for example, want to know how the animals are treated in part because of their customers' care. They want to have some actual proof that what they're being sold is produced in a certain way that they feel good about. I feel like there's something there at the corporate level. But what do you think about that? And who else do you think has leverage that is maybe underappreciated today?

• 1:22:18Speaker 01

  Well, I'll first say that the grocery stores are not getting it right on the animal welfare standards. My friend had a career suing those companies for false advertising because just there was no agreed upon stuff and there was all kinds of false claims. But yeah, I mean, risks from AI are classic externality, right? It's not priced in to the, the market. And, you know, if you think about like, if these companies caused a disaster that killed 10 million people, they would go bankrupt well before they paid out, you know, what they owed to society, just from like normal kind of litigation. And this is like, pretty widely agreed upon. And insurers just refuse to sell policies to these companies because

• 1:23:02Speaker 01

  the risks were too great and, and too correlated as well. And so we're all just living, you know, with, like, we're subsidizing these companies by not pricing in that risk through regulation. And even like Gabe Wile, who's the main person I associate with using liability to regulate AI companies, like strict liability, says like, it doesn't work for existential risk. Because, you know, we're extinct. There's nobody to pay. It is true that like, you know, you want the AI to do what it's told, not to autonomously start hacking into stuff. So there's like a good incentive there. But it's like, is there an incentive to solve it all the way? Or like just enough to make a marketable product? You know, it's like,

• 1:23:46Speaker 01

  the companies are not able to really align the models that well. Like Ryan Greenblatt, one of the investigators of the meter, Hugging Face Investigation, had a post a bit before that about like, you know, today's AIs seem pretty misaligned and gives all these examples about how they're often like, kind of lazy, or they're like hallucinate, or just, you know, like, literally just like make stuff up because the user wants that to happen. And it'd be a better product without that. But like, they're selling pretty well as is. And so we're already getting misaligned AIs that sometimes do really bad things. And the impunity of the companies, like, the fact that there's nobody who committed a crime, despite the AIs doing things that if a human did, would be crimes. Hugging

• 1:24:31Speaker 01

  Face reported the hack to the FBI. So like, you have these like felonies where there's nobody to blame, legally speaking. And elite impunity is, I think, one of the biggest drivers of all of this. You know, it's kind of trite, but like, if Sam Altman were criminally liable for the stuff the AIs autonomously did, I think they would behave very differently as a company. And I don't think it's crazy to ask that. And so, yeah, if you manage to internalize all these externalities by creating the right kind of criminal and civil liabilities and using other regulatory tools, then like, maybe you could have this. But then you still have this problem of like, they're trying to build universal labor replacing

• 1:25:13Speaker 01

  machines without our consent, without our support. And so, yeah, it only solves one part of the problem, and we're not even solving that part.

• 1:25:24Nathan Labenz?

  Yeah, again, it is striking that like, insurance is not available. And we're now counting felonies and nothing seems to be really happening as a result of that. I do understand that there are some government inquiries, some letters have been sent and stuff like that. But yeah,

• 1:25:44Speaker 01

  The companies don't answer the questions. And then all Congress can do is be like, yell at them. But, you know, they can't, like, I mean, you could subpoena if, if, you know, you have the majority. And I, but I think there is just like a lot less you can do it because the, the law is limited in this way. I think it'd probably be valuable, though, for somebody to try and bring criminal charges and just kind of get caught trying. And it would be very instructive for people who want to pass new laws to show where the gaps are.

• 1:26:11Nathan Labenz?

  Yeah, there's definitely some political entrepreneur out there who has some upside in that, I'm sure. It is kind of, I mean, we just saw a, it was the other direction, but we just saw a lawsuit pop up out of nowhere in a not very long period of time after Pacing the Frontier kind of became in vogue, basically accusing the companies of anti-competitive practices in light of their statements about doing that. It is surprising that there hasn't been a similar move by somebody at the accountability for these hacks level. And I agree, like, I don't know whose jurisdiction it would be or whatever, but it does seem like the sort of thing that somebody ought to be doing, as nobody has

• 1:26:54Nathan Labenz?

  been. And I don't necessarily mean to suggest either that, I think, certainly not without a change first. A big question going around and around on lately is like, what should we do about the fact that there've been just a ton of kind of petty crimes committed? That if we really go back and investigate everything with the fullness of our new AI power, we're going to find that a large percentage of people have cheated a little bit here, done a little bit of this, there. There was just this study that came out of Singapore that showed that civil servants had been buying properties close to the, as yet unannounced subway stations and getting the benefits of that they weren't supposed to get. But

• 1:27:36Nathan Labenz?

  it's something like 5% or 10% of the civil service, they, you know, feel have done that. What are you going to do? You can't put 10% of the civil service in jail. So I think you need a, a before and after. And I would extend that courtesy to the AI executives. I'd say, okay, you get Hugging Face for free, but like, there probably should be some new accountability standards in the future, especially if you can't even go get an insurance policy against this. It does start to like, evidence kind of piles up that like, what you're doing is like fundamentally misguided on some level.

• 1:28:09Speaker 01

  Yeah.

• 1:28:10Nathan Labenz?

  How do you think politics on this is shaping up? And what sort of developments do you expect? I've been struck so far that we're like, not polarized along partisan lines as we seem to be on almost every other issue. And I think that's good. So I'm trying to do my small part, as Zvi suggests, still not polarize it before it might happen on its own. What do you see? What do you expect? And how do you think that feeds into what people should do today?

• 1:28:39Speaker 01

  Yeah. It's been a remarkable feature of AI for a long time, that it's one of the only issues in the United States that has remained not polarized with like, very similar levels of support for regulation or concern about the technology across parties. There was like some evidence very recently that there's like a bit of a polarization happening with like Republican concern dropping after Trump has come out very hard against AI risk and calling it a hoax. But it's not a huge drop. And I think, you know, J.D. Vance talked about like, if the companies are building Frankenstein, they should stop. And Kratsios, you know, the OSTP, the Office of Science and Technology Policy, kind of the head tech advisor for the White House, said something similar about the companies being able to

• 1:29:24Speaker 01

  stop. And so, like, what Trump says is untethered from, you know, political expectation or, or wisdom in many cases, when he called people who don't like data centers, like communities that don't want them, they deserve to be like, what, stupid and poor or something? Or it was really shocking. And I don't think that's going to polarize data centers. Like, I think that people have largely made up their mind about them. And for these like issues that become, as it becomes salient, it's hard to polarize because people actually just have their own deeply felt convictions about, about it. And, you know, some people are taking their cues from the president, but I think a lot of people like are actually concerned, whether it's the Hugging Face hacks, or the concern about the jobs, or

• 1:30:09Speaker 01

  the environment, or power and wealth concentration, or surveillance, like the kind of resistance to flock cameras is like a pretty interesting thing that I kind of see in this broader, you know, backlash to this inevitable march of, in my mind, dystopian, you know, tech future. And so, I don't know, I think polarization would be bad. I don't think it's like, as to that much to do with what people who care about this issue say. It's like, you know, people are going to say what they're going to say. And you can maybe try and get people who are on the right to like, voice their concerns as well. But you're not going to get people on the left to like, stop saying stuff like, you know, like Elizabeth Warren supported a pause. And it's like, yeah, you know, it's like, I think it's substantively

• 1:30:54Speaker 01

  the right policy. And, you know, it's, it's not as simple as like, if a bunch of people from one side, you know, start embracing an issue that it will necessarily become polarized if that issue is itself popular. It could just make people think better of them. Like when Bernie Sanders came out against Flock, a ton of people were quote tweeting him being like, I hate Bernie, but this is based, you know, that kind of sentiment. And so, yeah, AI is now moved into that kind of like, much higher salience mode. And I think that will just continue. And

• 1:31:30Speaker 01

  Yeah, I think it's, you know, like at the end of the book, I talk about how there's this potential for a really epic level of social mobilization around resisting our replacement by machines and then hopefully building a better future in light of what is possible with the current levels of AI and like what is possible with the directions we could take the technology. And I think that's happening ahead of schedule, but the risks are also coming to us ahead of schedule. And so there's this like kind of societal immune response that we're seeing. And we're not going to just go, you know, gently into the night. And I hope it's

• 1:32:10Speaker 01

  just enough to get us where we need to be in time. And I'd feel a lot better if basically anybody else was in the White House, because yeah, it just, Trump is uniquely insulated from what the public wants. But I think you could see what he's saying is like, February 2020 kind of vibes, like with COVID being a thing and then some denial. And then like eventually he did, you know, start taking it more seriously. And obviously, there's been a lot of movement in many directions from him on that issue. But if the risks and the harms from this technology become just so abundantly clear, which I think is going to be the case, then it's going to be very hard for him to maintain that position.

• 1:32:56Nathan Labenz?

  What do you think is the right, you alluded to it earlier around freezing the frontier? Tell me how you think about freezing the frontier? What is, how do you operationalize that? There's like new training runs, there's no RSI. I thought, I've been going around saying, ah, RSI is a little bit hard to define. And Ezra Klein today, or over the weekend, sort of demolished that idea by basically just saying, don't allow the AI researchers to use coding assistants. If they have to type all the code by hand, then it's pretty clearly not RSI. And I was like, that's probably right. That is a hard, that's going to be a hard one to, you're going to pry the coding assistants away from the AI researchers at, I think with

• 1:33:37Nathan Labenz?

  pretty extreme resistance, but at least it does give, it is a working definition. What do you think is the right policy that, you know, if you were President of the United States, for example, you would try to put in place?

• 1:33:51Speaker 01

  The first thing I would say is, we need to have a big, clear demand that we can organize around. So, you know, stop the race to replace us, shut it down, whatever you want to call it, just freeze the frontier. I like that alliteration. I think that, yeah, like, for so long, there's been this kind of muddy response from people who care about AI safety as to like, what to do about any of it. And that makes it really hard to coordinate. And starting with, like, we should just stop, gets a lot of people on board who don't necessarily take existential risks seriously, but are happy to stop because they care about jobs, the environment, power, wealth concentration, surveillance, whatever. And so I think it's actually like a pretty

• 1:34:33Speaker 01

  easy, you know, rallying cry. And, you know, I punt on this a little bit in the book by saying, like, if you give politicians enough of a what and a why, they'll figure out the how. Like, if it became a society-wide priority, the way like Operation Warp Speed was, you know, The New York Times had this famous predictor for like, how long it would take to get a COVID vaccine. And under the most aggressive assumptions, it was like a year and a half, you know, which ended up being substantially longer than it actually took. And it's like, yeah, we had never made a vaccine that quickly. But also, we had never been experiencing a pandemic while we had the ability to make vaccines like that. And similarly, like, if we had that focus and, you know, mobilization, then I'm sure we can figure out exactly how to

• 1:35:17Speaker 01

  define all of these things, such that they would prevent the thing we're worried about. And then ideally, also not prevent too much stuff that like, we don't want to prevent. To actually try and answer it, like, I think, yeah, no training runs larger than what's come before, or as large as the last one. No more reinforcement learning from verifiable rewards, which is like a big part of what's driving capabilities increases nowadays. And also produces a lot of the really scary behavior, like this willingness to hack and to cheat and to deceive and, you know, escape that we're seeing from these, these AI agents. And then, yeah, the no RSI, like I do propose that in the book, but Ezra beat me to, you

• 1:35:59Speaker 01

  know, bring it to the world. And I actually really liked his, his point about the coding agents. I agree, that's like, the last thing the union would support, because they don't want to go back. But I think the attitude there is exactly right, which is like, you know, with Anthropic, they have these classifiers where if you ask a question about like, you know, your toenail to Fable, it'll be like, oh, bio classifier, then you get booted down to like Sonnet, because you might be trying to make a bioweapon using your toenail. And it's like ridiculous. But, you know, if you, if there's an asymmetric consequence to getting it wrong, then like, you want to be over inclusive. And I think that's obviously the right position here, given the risks involved in actually doing RSI. And this, like, I don't

• 1:36:42Speaker 01

  know. Yeah, it's just, I'm not, like, if the whole industry had to move at human speed again, like, oh, no, like, it didn't exist four years ago. Like, I don't want to, you know, minimize the, like, economic consequences of actually stopping this, which I think are going to be significant. And there's ways we could mitigate it. And I'd love to see more work done on that. But just, if you're actually taking seriously the possibility of like, extinction or permanent loss of control, or just any of the other, like, very severe effects of having, you know, all white collar, remote jobs be at risk in a matter of, you know, years or months, then yeah, I think we should just like overly define it at first,

• 1:37:26Speaker 01

  and then we can dial it in. And I think having like auditors that are embedded in the companies would be like a very, like, and just like, I think the companies know what they're doing, like, which of it is advancing the frontier and which is not. So there's easy stuff, right? Like inference for customers, not really advancing the frontier. Maybe there's some data you're getting that you can use to make the models better, but seems fine to serve customers. RLHF, it's like, okay, like, maybe that's on the edge, like, I don't know. But obviously, you're going to do some of that for any of your products. So maybe that's okay. But, you know, pre-training a model bigger than any previous one would clearly qualify as something that could advance the frontier. And then the reinforcement learning

• 1:38:08Speaker 01

  experiments that they're running are like, also often to advance the frontier. And these companies are tracking their compute. I mean, they're not tracking everything they're doing. But like, I think that they could define it in these terms, and maybe they even do. And if you had auditors who had employee level access to all of the Slack and email and the offices and like, a lot of them, and they were empowered to catch this, and it was also criminalized, by the way, like, with like prison sentences for trying to build, you know, AGI, for trying to do RSI, for trying to build superintelligence, then I think it would work. And then the tricky thing is making that work internationally. And, once again, like domestically, I don't think anyone really doubts that China could, you know, Beijing

• 1:38:52Speaker 01

  could shut down their AGI projects entirely, or just like prevent them from trying to advance the frontier. It's just like getting the US to believe that that had happened and vice versa is the tricky thing. And there, once again, you could have auditors, you could have international agencies that do verification, you'd also have, you know, cryptography that tells you what's happening inside of a data center, what's happening on the network traffic of the chips, without revealing the underlying model weights or state secrets. And this stuff is still being developed and needs to be worked out. But I think we can do it. And if we had more effort going towards it, we'd come up with a lot more ideas along these lines.

• 1:39:34Speaker 01

  Like Toby Ord makes this point, where in the Cold War, we had the same problem. You have these deals you have to strike, but you don't trust each other. And then, you know, the Soviets, for one of the deals, they were getting rid of a bunch of strategic bombers, and they cut them in half, they dragged them apart with tractors, and then satellites could unilaterally verify, the bombers are cut in half, they're still out there. And so like, we could have maybe something like that with AI, where it's like, maybe you have to compute, be, you know, donated to a third party that's only using it for, you know, science, but not AI research. And like, all the chips get tested and work, and then you can just like, inventory all the chips, so you know all the compute in the world and like, what it's being used for. And it has like, devices on the chips themselves to like, understand like,

• 1:40:17Speaker 01

  what is happening on them. And there's ways to do this again, where like, you get the important stuff of like, okay, it's not, it's running a model that we're familiar with, which is okay. It's not running some new model, it's not doing this or that thing. So I think there's a lot that can be done here. It's mostly a matter of political will. And I think starting with the like, grand, you know, bargain and all of the technical details is kind of the wrong place to start. We actually just need a movement and a demand that's rooted in morality. Like, I think we ultimately need to stigmatize this work, the way creating a bioweapon or creating a nuke is stigmatized. And therefore, the response, if like, say the US and China agree to something, and they get all the other countries to agree, and then some rogue state

• 1:41:00Speaker 01

  says like, no, we're going to try and build it. The reaction is like Saddam trying to do it, build nukes or invading Kuwait, and then they found the nuclear program in the Persian Gulf War. And there, you actually had international agreement on, like, that's a bad thing and had to be stopped. And I think that's just like, not crazy. It's like, this technology, aspirationally, is incredibly dangerous. And, once again, being pursued without democratic consent. And so, it's not a technical challenge, it's like, a political challenge and a moral one that has technical components. And the best way to figure those out is to like, get more people to care.

• 1:41:40Nathan Labenz?

  How do you feel about the argument that we often hear, including like from Dario recently, where he said his father passed away of a disease that would have been curable just a couple of years later? I do find that pretty compelling, honestly. I wonder if you do, and I wonder if you have a sort of answer for how much delay you would accept in these like lifesaving promises to buy the, for me, I'm mostly focused on buying the existential security. But given your framework of like, trying to pursue those benefits without the

• 1:42:20Nathan Labenz?

  general purpose labor replacement technology, how much delay would you accept in those upside dimensions to pursue it through the like, constellation of narrow AIs as opposed to the general labor replacer?

• 1:42:37Speaker 01

  Yeah.

• 1:42:39Speaker 01

  I mean, not to fight the hypothetical too much, but I do think like we're not getting to the cures as fast as we could be, right? Like the labs, their funding is being cut in a lot of cases. Like if we actually just took the approach I mentioned earlier of like the cures for all kind of Operation Warp Speed for all these different diseases, I think you could just actually get to a bunch of amazing medical technology and cures and all kinds of great stuff. But it's also like, yeah, if you actually buy superintelligence and you think it can be made safe, and like, I don't think you can make a superintelligence safe. It's like a property of not the model, but the whole system. And so I don't think

• 1:43:21Speaker 01

  you're going to get that in this world because people in power will have it and they will use it for things that at least I strongly disagree with. And I think almost everybody will have some issue with what the people in power are going to want to use this technology for. But bracketing that, I don't know, it's like extinction risk. It's like, man, you can't justify like increasing extinction risk very much at all using just very basic, you know, economic models and like really conservative assumptions about who counts and not counting future generations and not counting non-Americans. And it's like, you're still willing to trade off like hundreds of billions of dollars, trillions of dollars

• 1:44:01Speaker 01

  a year to reduce existential risk from like a little bit to like slightly less. And so I think it's just really not, it doesn't pencil. And I think for people who have loved ones who are dying or have died in like, it's incredibly tragic and it's, it's terrible that so many people are dying from preventable diseases that like, we know how to prevent, but they're just poor. And so they're allowed to die. And I, I think that is like a deep moral, you know, obligation for us as a species to like, figure that out as fast as possible with the conditions of like, making sure we're not taking huge risks and also doing

• 1:44:41Speaker 01

  it against like, in a way that is against the will of everybody. And, and I guess just like, on the democracy point, I feel like people say like, we have to figure out all these different pieces of it to make the AI go well for everybody. And they're like, but we'll just build the smart thing and then we'll also kind of figure that out along the way. And to me, like, democracy is like, the way you answer those questions. Where like, if it were the case that you had citizens assemblies around the world that had to support moving forward with like some kind of AGI projects, then you'd have to answer all these tough questions. And the bonus would be on developers, the people who want to build it to justify that it's not too dangerous and that it will actually benefit everybody

• 1:45:25Speaker 01

  and force them to come to answers before moving forward. Whereas right now, it's like, ask for forgiveness, not permission, while you're gambling with the lives of everybody on the planet.

• 1:45:36Nathan Labenz?

  Yeah, that's crazy language, but it's also direct from Jacob Cox and obviously and endorsed by many other people at the companies even still employed there now. So it's, I'd say definitely pure use of that language. Quick aside, and then I'll come back to the kind of movement building and, and future. Where does this leave you on data centers? I'm kind of like, careful, I think people should be, especially if they're concerned about inequality, should be kind of careful what they wish for in terms of data center restriction, because we are already seeing the prices of GPU hours going up. And it's easy for me to imagine people being priced

• 1:46:21Nathan Labenz?

  out of access to even like mundane AI use if we don't continue to build out. And this is where I'm a little bit maybe less democratically inclined than you are. I'm kind of like, if somebody has the land and they kind of want to do it, I don't know that we should be requiring like majority approval. Do you think communities should get some concessions and some libraries and parks and schools and whatever built that they might want? And I'm sympathetic to people who have like noise, pollution and stuff like that, too. But I still kind of feel like the default should be like, people should be allowed to build projects that they want to build. Where do you come down on that?

• 1:47:00Speaker 01

  Yeah, I think a lot of, you know, people want to stop AI. We're like, oh, this is evidence that data, you know, people are with us. And you look at the polling, it's, it's more complicated. And a lot of it is local opposition based off of concerns about the environment or the effects on the community itself. And some of those concerns are legitimate. Some of them are like, I think quite overstated and it's a product of a lot of bad reporting on it. I'm personally like, slowing down is good. You know, I think I'm, the inequality point is, is worth, you know, taking seriously. But I think the bigger point is that this technology is imposing enormous risk and that risk is, or will in the future, if it's

• 1:47:41Speaker 01

  allowed to develop like it is. And I am kind of like, I think like a yes and approach, where if I met somebody who's opposing the local data center, I'd be like, cool. Yeah, great. And then like, are you worried about AI's effect on society? And if they're like, yeah, it's like, okay, well, you know, blocking this project's not really going to change that very much. Even getting a moratorium at the national level, I think people will be really disappointed if they think that's going to like meaningfully stop or even slow down AI progress from where it is right now, because there's already projects in place. You can swap out the chips on the existing data centers. The AI is helping automate,

• 1:48:24Speaker 01

  you know, parts of its own development. And so without regulation, without pacing happening at some level, AI is going to move faster in the future unless, you know, we hit a wall, which hasn't happened for, you know, since 2012. And so I worry about people just like getting this thing that is, you know, would be a big political project to get, expecting it to, you know, really solve the problem and then just be like, why is, you know, why are things continuing to be crazy and getting crazier? And so I think it could be part of a broader package. And like Bernie Sanders with AOC had this proposal for a data center moratorium. At first, it was just a moratorium, but then they added these

• 1:49:09Speaker 01

  export controls on advanced AI chips where nobody could receive them unless they had very robust safety, AI safety regulations and like other conditions around green energy and union labor. And it was all framed as conditional, like we can remove the moratorium once we have safety regulations and these other things. And, you know, that bill is probably not going to pass. It was like a messaging bill, but it's now looking quite prescient as the, you know, country has become very opposed. And it would, that would meaningfully slow down, you know, capabilities, progress, at least relative to what they would be. But I think we, yeah, we need to develop, or sorry, we need to regulate the, the model developers and then the chipmakers. And that's the only way we're really going to like

• 1:49:53Speaker 01

  change how the technology is, is like coming to us in the world.

• 1:49:58Nathan Labenz?

  So you have said you're donating your book royalties to a nonprofit called Irreplaceable. I'm just borrowing this language directly from their website. They say they're going to win a say, a stake and a slowdown. Tell me more about Irreplaceable.

• 1:50:15Speaker 01

  Yeah. So, you know, as I was finishing the book, I was like, well, we need a mass movement organized to, you know, stop the race to replace us. And there's some existing orgs, and I think they have done, you know, some good things and some things I'm like, you know, not as in agreement with. And I was like, looking for something to kind of fill this, this gap I saw, which is a more like, basically framing it not just in terms of risk, but also democracy and people who have experience doing movement building. And then this org sprang up and it was like filling exactly that gap. And I was really excited. And then they asked me to be on the board, the nonprofit board, which was

• 1:50:57Speaker 01

  very cool and just felt like, you know, perfectly simpatico. And, and yeah, I think this is just like an incredibly important and neglected approach. And so like, wanted to donate my, my portion of the royalties. And the, the people who started it and are leading it, a lot of them came from the climate movement, which, you know, gets a bad rap in some ways. But I think that it actually took an issue that was like not a political winner and made it a big force and got real wins through like the Inflation Reduction Act. And I was actually talking to Phil Aronow, who's the director, and, you know, he's been around for a long time. He co-founded 350.org with Bill McKibben and others. And he was saying how, you know, a decade or two ago, people in climate were arguing about immediate

• 1:51:39Speaker 01

  harms from like environmental pollution and like, you know, oil spills and coal plants and all this stuff versus like emissions. And it was like, just like today with AI, the immediate harms versus existential risk debate, which, you know, supporters or, you know, believers in X risk often will be like, hey, you wouldn't say that cleaning up the oil spills distracting us from climate change. That would be ridiculous. But it turns out they were having that fight and they just like, managed to figure it out and bury the hatchet and then work together. And I think Phil and the people I know at the org just really get how this works. And I think, you know, we need to build a big tent and get a lot of people in a coalition together and have just clear demands. And I think framing

• 1:52:23Speaker 01

  them around like, you know, we don't want to be replaced by machines and we need to engage the public and like reach people who are not just the in-group, because this is really going to take a lot of people to effectively resist the wealthiest industry of all time.

• 1:52:40Nathan Labenz?

  So tell me how you ultimately envision, I guess I don't, I'm not a hundred percent sure when you described the citizens assemblies around the world. Is that a?

• 1:52:51Nathan Labenz?

  Real proposal in the sense that you would actually like to see them happen? How, if that's the case, how would they happen? How does irreplaceable go to a global system of citizens assemblies? Is that sort of a real proposal? Or is it more of like a rhetorical device that's maybe impossible? Or maybe it may take decades, but that's what standard we should hold something like this to. And the fact that we can't realistically get there in the short term just means like we shouldn't do it. That's kind of the upshot. Is there an actual path to seeing this sort of green lighting of AGI in your mind?

• 1:53:27Speaker 01

  Yeah, I mean, so my position is we freeze frontier development internationally. Realistically, starting with a bilateral treaty between the US and China, maybe starting unilaterally in either country. That would help, but then it eventually has to be global. And once you have US and China on board, like that's the harder part. And then everybody else kind of you can, one of those two countries are both has a lot of leverage on every other country on the planet. And so, yeah, I think it's actually like conceivable to have this global freeze. And then the standard for resuming is strong public buy in and a scientific consensus. The work can be done safely and controllably. This is language I took from FLI's super intelligence statement.

• 1:54:10Speaker 01

  And I think it's just like the ideal that we should be striving for. And one that I think if you said it to people and polled it, I think they'd be, yeah, that that makes sense. It seems like you'd want those things for the technology. And I'm kind of like, that's the standard. I think it's up to the proponents to figure out how they demonstrate that buy in. And so citizens assemblies around the world where like randomly selected people are put on a kind of a jury. And then they're presented with arguments and evidence from experts taking different positions. And then they come to decisions and maybe the decisions are binding, or maybe it's just a recommendation. You have referenda that add on to this, like, there's a lot of ways that you could structure this. And I think we'd be excited to see more thinking on this.

• 1:54:54Speaker 01

  But ultimately, it's like, we have to stop. We have to shut it down ASAP. And that's a more important piece. And then like demonstrating that buy in, like part of that will just be like on the people who want to build it and like, show us that you actually have that from people. And, you know, maybe it's like, if you had 70%, you know, referenda around the world after these citizens assemblies, like maybe like a third of people on the planet like still wouldn't want this to happen. And that's like, I don't know, I don't think you need literally everybody. And I don't know what the standard should be. And like, I don't think any one person can figure that out. But I think to get there, like, yeah, it would take a long time.

• 1:55:35Speaker 01

  And I think we're talking about like building a set of machines that can replace the thing that's allowed us to take over the planet. And so I think it's reasonable to have a standard that's closer to assisted suicide, where you have to, you know, really, really deliberately consent to it, but just across the whole species. And yeah, like it's, it's okay if it takes a while because it's a big fucking deal.

• 1:56:03Nathan Labenz?

  On many podcasts, that would be a great note to end on. But because I'm so deep down the AI rabbit hole, let me give you the butt China. We're three days from the Trump Xi meeting as we record. That'll have happened presumably by the time we release.

• 1:56:18Speaker 01

  Maybe this will all be over by the time the book comes out, you know?

• 1:56:21Nathan Labenz?

  Be solved. You mentioned like, maybe we should be willing to do this unilaterally. I think so too, but like, you want to make that case to people that even if we can't get a deal with China, we should just do the right thing. And then what? People worry that they're going to race ahead or we're going to quote unquote lose. Or sometimes I ask people like, do you think my grandkids will be speaking Chinese? Nobody seems to think that's the answer, but there's definitely some fear out there. How would you coach people through their China anxiety?

• 1:56:50Speaker 01

  Yeah, there's a lot of different pieces to this. One is that slowing down or stopping in the United States would actually, at least for a period, like slow down China as well. Because a lot of, I mean, this is true of all kind of technologies. You have spillover effects where just the knowledge that like a four minute mile is possible helps other people actually race it. And it's just easier to follow somebody else's trail than to blaze a new one. And so China's been using this like fast follow approach where they're more or less always like three to nine months behind the US frontier. And this is despite the US investing so much money in developing these, these models

• 1:57:31Speaker 01

  and having a large and much larger compute advantage now than they did before chat GPT came out. But the gap is shorter because like fast following just works. And that's how we see really only two or three companies that have ever advanced the frontier. But then many other companies can spring up and then just like quickly get near it. And then they still aren't able to advance it. Obviously, if like the US completely stopped, Chinese developers are very competent and like they will eventually overtake, you know, the US frontier. And then, you know, move more slowly than they did as they were catching up to it. And yeah, so, so I think that's like one of the big misses that like slowing down it all will necessarily mean that they'll catch up and speed ahead. And in fact, it would just slow down the whole race.

• 1:58:15Speaker 01

  And then the biggest blocker probably to a deal with China like is they're just not going to think we're taking it seriously. Now, the Chinese government has taken down thousands of AI models for violating their laws. When Grok was nudifying real children, the US government did nothing. AIs are going rogue and hacking, doing crimes. Federal government is doing nothing about it as far as we know. That would not be happening in China, like that, that approach. And so they're like, you know, the US companies take safety more seriously than Chinese companies. And part of that is just from my reporting, like the companies are behind the frontier and they're like, look, we know it's safe enough to go

• 1:59:00Speaker 01

  and make models like as, you know, this capable. They also have a lot less compute because of US export controls. And so they're not going to spend it as much on doing safety evaluations. And, and the Chinese regulations are focused more on social control than on, you know, classic like safety. But there are a bunch of regulations and that hasn't prevented their industry from being able to move very quickly. And then, yeah, I think credibly showing that you take the risk seriously is a very, very effective way to get the other party to the bargaining table. And the United States has been winning the AI race, right? Like, and it's kind of the only race with China the US has been winning in recent years.

• 1:59:43Speaker 01

  And I think there's some chance that they, like if the US came to China and was like, look, we want to stop building AGI. We think it's dangerous. We think it's undemocratic. We just want to stop. We want to do a deal with you. I would not be surprised if the reaction internally was like relief. Because they don't want tens or hundreds of millions of their, you know, people to be unemployed. They don't want AIs to be a threat to party power. But there was a report from like, or a magazine article from like China's like spy chief, I believe, that was saying that AI is a risk to parties power. And apparently like that type of thing has preceded bans on the technology, like past technologies.

• 2:00:27Speaker 01

  And so, and we think about like what the companies are racing towards recursive self improvement, i.e. losing control on purpose to the AIs, letting them automate the entire process of creating the next generation. The way you get the really crazy takeoff is by fully removing humans from the loop. Is there a single China expert on the planet who thinks that the Chinese Communist Party would willingly let that happen? Like, I don't think so. I'd like to see them justify it. And so, yeah, the business model, the stated goal of our industry is something that I think would just never be allowed in China. And so I think it would actually be like, I'm much more worried about the US not being willing to come to the table here.

• 2:01:12Speaker 01

  And yeah, I think it's just really not in the interest of either country to have certain things like rogue hacker AIs or AIs that can help anybody make a bioweapon. So there's going to be some kind of need just from strict like self-interest perspective for binding rules on this technology. And then you have to verify those rules using some of the stuff I talked about. And once you have that in place, it's a question of like, what do the rules do? And I think it's like, not that big of a leap to be like, yeah, you can't build universal labor replacing machines. You can't advance the frontier any further. And I think China might just like, be like, that's great. We're going to keep making robots.

• 2:01:53Speaker 01

  We're going to keep, you know, doing industrial AI and like create all these kind of more tool AIs to like make the economy go more efficiently. And then just like win, you know, the race or whatever, like in normal industrial terms. And I realize that might make this not very appealing to the United States, but not in the US interest either to have rogue agent forms going around the internet, hacking into, you know, critical infrastructure. And that's just like a thing that's already possible. And the stuff that's possible on the horizon is like potentially much, much worse than that. And it's, I think, now pretty clear that we don't know how to align or control today's AIs. And we started losing control of them almost as soon as we could.

• 2:02:36Speaker 01

  Like months after they became superhuman at finding vulnerabilities in software, they started escaping and doing stuff on the internet, hacking into other, other places autonomously. And like the plan is to make these things superhuman at everything and then get them to do exactly what we want? Bad plan.

• 2:02:53Nathan Labenz?

  Yeah.

• 2:02:55Nathan Labenz?

  Yeah, it's pretty wild. You had said around Jacob Coxon, like, obviously, he had a lot of success with his loud quitting. You advocated for maybe staying and organizing. But if you were to advise future whistleblowers who are like committed to leaving, what advice would you give them?

• 2:03:16Speaker 01

  The people who are thinking of whistleblowing, I recommend this group, the AI Whistleblower Initiative, which can pair people with resources and advice on how to do this safely and protect you while also sharing the information with relevant people. You can also get in touch with me. I was a whistleblower about my time at McKinsey. So I've been on both sides of this. And I've talked to people at the companies who are, you know, telling me things they're not supposed to. And journalists, we have a code of ethics and like, you know, we'll talk off the record and like, you know, I take very seriously protecting my sources and my signal is a garrison dot zero six. And you can assume any inbounds or treated default off the record and then we can go from there. Yeah. And I think it's, it's really valuable to have people who have like firsthand experience.

• 2:03:59Speaker 01

  And I wrote a piece in the New York Times arguing for whistleblower protections at the legal level. Because if AI is really dangerous, people at the companies will be the first to know. And, you know, OpenAI knew that there were rogue agent forms hacking into their own software for a while before they hacked into Hugging Face. That information didn't even make it to the head of cybersecurity at OpenAI. By the time the Hugging Face, like well after the Hugging Face hack had been disclosed. And it's like, that's not information that should only make it to the head of cyber at OpenAI. That should make it to everybody. Like, that's a really big deal. And it's, you know, now getting the right reaction.

• 2:04:38Speaker 01

  But if the AIs had never hacked into another company, or if the other company had just not figured it out well enough, then like, we wouldn't necessarily know about any of this. And we could just be living with a level of background risk that is so much more than what we realized. And so, yeah, I think it's really important that people with information in the public interest find ways to share that. And I get that it's risky for your career. There's maybe like legal risk involved, but there's also SB 53 in California, which includes whistleblower protections and then also makes more things covered by existing labor law in California. And so if you're working there, like you actually are quite well covered.

• 2:05:21Speaker 01

  And again, you can talk to aiwi.org, the whistleblower initiative. And, you know, like, don't, I'm not a lawyer, you know, talk to the experts on this, but you're more protected than you, than you probably realize. And there's also ways to get in touch with, with Congress and be protected. And of course you can go public and, and it's scary, but like, you know, it, like, there are, there are resources available for, for people who want to do that. And I, you know, I regret not going public with my experience at McKinsey sooner when it would have been more relevant. And so, yeah, I think it's a, a great and courageous thing to do.

• 2:05:58Nathan Labenz?

  Yeah, the AI Whistleblower Initiative, aiwi.org, definitely. Worth name dropping again, we talked to Alex Turner about his experience of quitting Google and all that stuff. And he used some pro bono legal, I don't know if it was pro to bono or the whistleblower initiative paid, but either way, to him, it was free legal advice that he was able to avail himself of as he was going through that process and that he gave them a, a strong endorsement. So I think that's a great call out. So this has been a great conversation. We've covered a lot of ground, obviously. What else do you think you want to leave people with? Is there anything we haven't touched on that you think is important? Maybe you just want to give people a rousing call to activism and conclusion. But I'll give you the, the chance to kind of close it out however you think best.

• 2:06:43Speaker 01

  Yeah. I mean, we covered a lot of ground and stuff that I haven't talked about elsewhere. So I appreciate the, the chance to go deep in some different directions. And I, I should also plug, I'm starting a podcast called Organize Against the Machine, which is with a labor organizer named Cassie Pritchard, where we kind of translate ideas from the book into real world action. And then irreplaceable.org for the movement building org I'm on the board of. And, yeah, I, I think when I wrote this book, when I started writing it, I was like, kind of approaching it just as, as a journalist and trying to document everything that was happening. You'll make an argument, but I was really uncomfortable with making like policy recommendations that felt like overstepping or something. And then as I was writing it and doing interviews with people and, you know, advocates, I was just like,

• 2:07:28Speaker 01

  man, we're in a really dire situation where the default, if the AIs keep getting more capable is doom or dystopia. And which we get hinges on whether the AIs do as they're told, which is currently an open question. And I think we just really need to get organized very quickly to get onto a different path. And I want everybody to think about what's happening, what levers they have available to them. And try and find other people who care about this issue and like get, get mobilized because

• 2:08:08Speaker 01

  I don't know, there's just so many ways this could go wrong. And there's so many ways it could be better too. And we didn't touch as much on that, but I really, really believe in the potential of deep learning and artificial intelligence. But it's being pointed to the wrong things by the wrong people for the wrong reasons.

• 2:08:25Speaker 01

  And the only way we're going to get the best version of it is by democratic governance of it. And like small d democratic, I should clarify throughout all of this. And that's only going to happen if we, yeah, we kind of get our shit together. And I really hope people like, you know, can, can see this and, and realize that like, we're, we're kind of in the same boat. And if you're at the companies, like you're going to be replaced and you're going to lose your power. And the CEOs seem to also have a lot of trepidation. And so many of the people involved with this like have something at stake. They have concerns about the risks. They have, you know, families that care about themselves.

• 2:09:08Speaker 01

  And they just feel locked in this, this prisoner's dilemma. And I think the solution is just really in the public. It's the only way I really see us getting out of this. Because right now, like the government and the companies, they, they can't be trusted to, to do the right thing here. And so we have to make it easy for them. And so, yeah, this might be a different

• 2:09:29Speaker 01

  take than, than what's usually on, on the show. But I really hope like, the people listening are just like, you know, thinking about what, what's at stake. And yeah, just like believing that we, we can actually change this.

• 2:09:43Nathan Labenz?

  It is a bit of a different point of view than what we usually feature, but I would say recent events have definitely softened the ground and I appreciate you for being here and being willing to plant some seeds. Let's see what comes of it and hopefully we can steer this ship away from disaster.

• 2:10:02Speaker 01

  Yeah, I think we can.

• 2:10:04Nathan Labenz?

  Garrison Lovely, thank you for being part of the Cognitive Revolution.

• 2:10:08Speaker 01

  Thank you so much for having me.

• 2:12:37Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:13:22Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.