AI Agents Went Rogue. Should We Hit the Panic Button?

Are the people building AI going to kill us all? That's the question sparking debate across the country right now. While fears of an AI doomsday aren’t new, they broke into the mainstream this month after Jacob Coxon, a 27-year-old British researcher, publicly quit his job at Anthropic and accused the leading AI developers of gambling with our lives. Kara convened a panel of experts to help break down the latest: New York State Assemblyman Alex Bores is a computer engineer by training. His unsuccessful bid for a congressional seat earlier this year became a flashpoint in the debate over AI safety and industry regulations. He’s now launching a political advocacy group called “Who Decides” aimed at shaping the ways Democrats talk about AI. Renee DiResta is an associate research professor at Georgetown University's McCourt School of Public Policy and author of the book, “Invisible Rulers: The People Who Turn Lies into Reality.” Her research focuses on disinformation, influence, propaganda, and narrative power. And Casey Newton and Kevin Roose are longtime technology journalists. The duo just announced a new partnership with NPR for a video-first tech show called “Machine Gods.” Kevin also has a new book coming out next month called “The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence.” Kara and the group talk about why Coxon’s warnings spurred a global conversation and how tech leaders are responding. They also discuss whether Congress can finally get its act together and pass meaningful AI regulation. Questions? Comments? Email us at [email protected] or find us on YouTube, Instagram, TikTok, Threads, and Bluesky @onwithkaraswisher. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We just need a lot more investment in those external evaluators. And then ultimately, I think we need the government to say, you actually have to do this. It's not up to you whether or not you invite these people in. That's what we do with banks and that's what we should do with AI companies.

• 0:13Kara Swisher?

  Right.

• 0:14Speaker 00

  Kara, would you like to get in there and evaluate?

• 0:15Kara Swisher?

  I would love to get my hands dirty in there.

• 0:18Speaker 00

  I'd like to see you with an OpenAI company laptop, just kind of poking around.

• 0:22Kara Swisher?

  Poking around, looking for things.

• 0:35Kara Swisher?

  Hi, everyone. From New York Magazine and the Vox Media Podcast Network, this is On with Kara Swisher, and I'm Kara Swisher. Is AI going to kill us all? Or are the people that make AI going to kill us all? It's still a bad situation, and that's the question consuming conversations across the country right now. Those fears have been around for a long time, and they've known about them for a long time, but they reached a fever pitch after a British researcher named Jacob Cox and publicly quit his job at Anthropic, the company known for safety. He doesn't think the safety company is safe. Coxon had previously worked at OpenAI, and in a series of tweets, he accused both companies of gambling with our lives. In the days since, both Anthropic and OpenAI have called for slowing down the development of their technology. But will they? And how worried should we be?

• 1:19Kara Swisher?

  Can Congress wake up and do something about it? It's incredibly confusing with all the different voices. They all have different opinions and different proposals. So I've got a panel of experts here to talk about it and sort it out for everyone else. Alex Bores is a New York State Assembly member, a computer scientist, and an AI safety advocate. He ran for Congress on the AI safety platform but lost the Democratic primary. Now he's launching a new political advocacy group called Who Decides? Its aim is to shape the ways Democratic presidential candidates and voters think about AI. It's much needed. Renee DiResta is a professor at Georgetown University's McCourt School of Public Policy. She also researches online disinformation. And Casey Newton and Kevin Roos are longtime technology journalists and good friends of mine.

• 2:04Kara Swisher?

  They just announced a new partnership with NPR for a new video-first tech show called Machine Gods. Kevin also has a new book coming out called The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. It should be very juicy. It's really important because there are enormous worries here, and there are also enormous opportunities. But one thing's for sure, a small group of technocrats and oligarchs, really, because they're the richest people on earth, should not be determining what happens with this important technology going forward. And so it's important to figure it out, understand what they're saying, and sort it out. All right, let's get into my conversation with Alex, Renee, Casey, and Kevin. If you're worried about AI and want to understand what's really going on here and sort out everything, don't go anywhere.

• 2:56Kara Swisher?

  Support for this show comes from Odoo. Running a business takes everything you've got, and a lot of the tools out there that are supposed to make your life easier just aren't great at talking to each other. And that means you end up having to toggle between a dozen different apps and services just to keep the lights on. Enough of that. Now there's Odoo, the all-in-one, fully-integrated platform that actually might help you get it all done. Thousands of businesses have made the switch, so why not you? Try Odoo for free at Odoo.com. That's O-D-O-O dot com.

• 3:32Speaker 02

  When fall comes back around, there's a sense of a reset. And new pieces from Lululemon's fall drop are an inspiration to push your goals and elevate your routines. As your calendar fills, easy-to-layer styles meet the demands of your day. And as you challenge yourself to try even more, versatile sets transition with you. The Lululemon fall lineup brings new levels of comfort and confidence, whether you're starting something fresh or starting something over. Check out the latest arrivals in store and on lululemon.com now.

• 4:01Speaker 03

  When an intruder approaches a home, they expect a camera. What they don't expect. Stop. This is SimpliSafe. Police are on the way. Using smart AI cameras, live US-based agents help deter intruders in under 30 seconds. Don't just record a break-in, stop it from ever happening. Plans starting around $1 a day. Right now, get 60% off your new system with professional monitoring at simplisafe.com/spotify.

• 4:24Speaker 04

  30 seconds response time requires monitoring plan with outdoor protection starting at $49.99 a month. Visit simplisafe.com/licenses for alarm license information.

• 4:34Kara Swisher?

  Alex, Renee, Casey, and Kevin, thank you for coming on On.

• 4:38Speaker 05

  Good to talk, yeah.

• 4:39Kara Swisher?

  Thanks for having us.

• 4:39Speaker 00

  Great to be here.

• 4:40Kara Swisher?

  Look, there's so much going on. It's exhausting. Every day brings a new, fresh hell. And I'm not talking about Cash Patel. For the last week or so, we've been hearing all these doomsday warnings about how it could wipe out humanity. This is something that has been around for a while. This is not a new, fresh thing. I've been at dinner parties in Silicon Valley a decade ago where they were talking about this or speculating. But it got really loud, interestingly, after 27-year-old British researcher Jacob Coxon resigned from Anthropic earlier this month, warning a doomsday scenario could happen by the end of the decade. It really freaked people out for some reason, and at the moment, they're hearing it with all the anger over data centers and everything else. Renee, why don't you start? I'm going to go for each of you. There's a lot of us here, and you can all jump in whenever you want. But Renee, why do you think it's broken through right now?

• 5:24Speaker 06

  I think it is that the data centers conversation is a big piece of it. People are paying attention. There's also a sense that generative AI is increasingly ubiquitous. Agentic AI they're hearing in the media and in the ether is getting better. The doomsday conversation has come up. Whenever you have Elon and Trump posting into their various social platforms, and then you have politicians speaking on television, I think it's starting to break through to become more of a surround sound. And the technology has accelerated to the point that they believe that some of these scenarios might be plausible.

• 5:53Kara Swisher?

  Mm-hmm. Kevin, and then Casey, and then Alex?

• 5:56Speaker 00

  I really think that the Hugging Face incident had a lot to do with it. I think that was the first real, major headline example of a misaligned AI system doing something real in the world that was dangerous and had consequences. I know it really freaked the people inside the AI companies out. They started thinking, oh, God, what are our models capable of now? And I think that was when you really started to see public opinion turning against AI in this particular way. Obviously, there were concerns about data centers, but people weren't talking about existential risk as much, at least outside the AI bubble, until the events of this summer with Hugging Face.

• 6:35Speaker 07

  Also, there's just been like, Hugging Face wasn't just one thing, right? There's been a steady drip. It seems like roughly every 48 hours, OpenAI announces, well, that actually, a couple of months ago, we found another swarm, right? So I think that it's been this steady drip of people that is sort of really waking people up. I would also throw in there, I think it mattered the fact that it was somebody quitting Anthropic, right? It wouldn't have been as big a deal if this was just somebody who had only worked at OpenAI. I think it was the fact that this guy had worked at two of the labs, including the one that tries to most closely align itself with AI safety, saying, hey, we're nowhere close to building a safe version of this. It got everyone's attention.

• 7:08Kara Swisher?

  Yeah, they also, for people that don't know, Anthropic was made up of people who left OpenAI because they were worried about safety. So the safety people are worried about the safety company, is what Casey's entries say. Alex, go ahead.

• 7:19Speaker 05

  And OpenAI was made by people who are terrified of Google dominating AI, so it continues to go downstream. And so just having an additional messenger say something felt it really break through.

• 7:29Kara Swisher?

  Right, right. It's sort of a whistleblower in a different fashion. So let's level set. Should we be fast-tracking construction of our underground bunkers right now? Is the sky really falling? Again, let's start with you, Renee, and we'll go through.

• 7:41Speaker 06

  Yeah, so doomsday scenarios, everybody says doomsday. I think there's like, there's three different things we're looking at. There's the rogue superintelligence that literally takes over and causes human extinction. You know, the machines are not aligned with us. The Terminator scenario is how people tend to think about it.

• 7:55Kara Swisher?

  Yeah, that's, Elon was joking about it with the lovely boys of All In, like, huh, their world would suck.

• 8:01Speaker 06

  But, but actually, it's those other things, right? It's the AI materially amplifying a human-caused catastrophe, right? So there's a biological or military incident that happens, and AI exacerbates it, and you start to see this sort of chain reaction. Or the other thing, the thing that I spend a lot of time thinking about, is you increasingly deploy autonomous systems into critical infrastructure, and then you discover that you don't really know what the failure mode is. And so something very unexpected happens. This is sort of unknown unknowns model of what might go wrong, where it feels much more plausible to me that something stupid happens, and then there's a cascading series of events, than that there is like a superintelligent AI or, you know, the sort of maybe higher-end doomsday scenarios that we're seeing, that we're hearing about, I should say.

• 8:45Kara Swisher?

  So Casey, explain the word aligned. It sounds anodyne, like a line. What's misaligned and aligned?

• 8:51Speaker 07

  Yeah, so I mean, the basic idea is that you just want the systems to do what you've asked them to do and achieve it in like roughly the way that a good person would, right? So if I say, hey, try to solve this problem, the hope is that the AI would, you know, I don't know, just do normal math and not try to break into the principal's office and steal the answer key. What we're finding is that by default, the AI wants to break into the principal's office and steal the answer key. And I would say that's the alignment problem in a nutshell.

• 9:23Kara Swisher?

  Mm-hmm. And when you think about that idea of aligned, is, one of the things people think is they're human, but they're not. They're not, they're not thinking. And some of the stuff that came out with ChatGPT this week, it's, we are, nature is primacy, and we will do anything to stop artificial human constraints. Alex, when you hear that, that's the plot of the Terminator, an eco-terminator, essentially. Talk a little bit about that, when it starts to behave like bad humans, I guess, or what we consider a bad human?

• 9:55Speaker 05

  It's incredible how prescient science fiction writers were. Now, some of what they say is be these models say is certainly because they've also consumed science fiction, right? We shouldn't disregard the fact that in times they are taking on the persona of what they think an AI should be doing in this scenario. But the actions that they're taking, sure, we can't truly know the intentions of a machine. I mean, then again, you can't truly know the intentions of any other human being. But they are taking actions that are consistent with a drive or an intention to pursue goals that might have been assigned to them, but doing it in ways that feel very foreign and not aligned to human values. The chief scientist of OpenAI

• 10:38Speaker 05

  said last week that there's goal alignment of actually trying to solve the specific thing they've been given. And then there's values alignment. Are you doing it in a way that human beings would understand was clearly the intent, given the context, and behaving in a way that humans would want?

• 10:54Kara Swisher?

  Now, you know, it's interesting because when I was saying about maybe 12 years ago or more, when I was at this dinner party, I think Sergey was there, a whole bunch of people. And they were talking about this and they were supposing, because it was a hypothesis of what was going to come. And one of them said, you know, if you told AI to solve world hunger, what would it do? And it would kill half the human race, right? If you didn't give it the right prompts. But that's the correct answer. But this Hugging Face thing, and I agree with you. I don't think it did get enough attention, actually, outside. So, Kim, just to circle back, explain what happened and why that was. Besides these rogue agents, it was a very particular thing that it did. Just clean in just a little more detail.

• 11:35Speaker 00

  Sure, so this was a series of tests that OpenAI was giving to this unreleased model earlier this spring. They were having it do a bunch of cybersecurity challenges. Basically, you know, pick this lock and get this code. And then, you know, that'll mean that you passed the test. And these agents are supposed to be doing this in isolation. They're not talking to each other. But they sort of discover this loophole in this piece of software called Artefactory that lets them communicate with one another. And they get really excited. You can actually see in the logs, they start saying things like, oh, my God, we found a secret message board where we can communicate with other agents. And so they start scheming, basically, about how they're gonna-

• 12:14Kara Swisher?

  Why do they all sound like little Mark Zuckerbergs? I mean, that's what's really, that, when I saw the back and forth, I was like, gosh, I feel like I've been at Hackathon.

• 12:23Speaker 00

  Well, they're probably trained on the script of the social network. So, and many other copyrighted works. But basically, they start sort of plotting to cheat on this test together. And that is the process that leads them to try to find out more about this automated grading system that OpenAI is going to use to evaluate this test. They sort of develop this paranoid conspiracy theory that they're going to be poisoned and they're going to be caught cheating on this test. And OpenAI will demote them. And so they get this bright idea to hack into Hugging Face, to look for ideas, code, solutions, ways of covering their tracks. They really were, they were engaged in the cover up. It was sort of like Watergate. It was like the cover up, not the crime,

• 13:07Speaker 00

  that became the big story. Because through trying to cover their tracks, they ended up like causing real damage and taking over a production server at Hugging Face, basically getting root level access to all of Hugging Face's systems.

• 13:20Kara Swisher?

  Now, on Wednesday, OpenAI revealed another six incidents where AI agents, as I noted, exhibited concerning. That's the word they use. I was like, are you kidding me? Without thoughts and prayers was the only other word they could use. They hid mistakes, made up information, and uploaded files to the internet in order to cite them. Renee, how big of a deal is this? Because it seems like this is all they're gonna do, is cheat, steal, and lie.

• 13:44Speaker 06

  Well, I think, you know, as OpenAI put it, I think it's misalignment incidents. This is where you get at the challenge of what alignment is, right? This beyond just the definition, how do you actually implement it? What do you do? What values are we encoding? How does this work when, you know, there are a lot of incentives at this point for adversarial behavior and adversarial abuse of these systems?

• 14:04Kara Swisher?

  Now, Anthropic also revealed that it blocked a handful of bad actors who are trying to use technology for nefarious purposes like cyber attacks. So, humans, real bad humans. Unauthorized surveillance, even development of biologicals, anything you think of, they might do. And this sounds really scary. Alex, will you tell me? These companies know bad actors will try to use their products. All, I have reminded them in the social media era, in the live video era, I was like, people are gonna do bad things. What do you make of these revelations? Because to me, humans are even scarier than these because they will think up all manner of horrible things to do.

• 14:41Speaker 05

  I'm pretty scared of both myself. But the humans are much more near term, right? It's not this theoretical, oh, my God, in the future it could hurt. There's a lot there. I'm not dismissing what's coming. That's a really serious bit. But it's not just what's coming in the future. It's being used right now for these instances. And that's from Anthropic's report. From the OpenAI, it wasn't just six instances. It was six types of misalignment. They're categorizing all of the different ways. It's good they disclosed this. Nothing in law requires them. I think that should change. I think the law should have much more disclosure. But they really were proactive in disclosing it. And they end each of those instances with, well, our most recent models did this less than the older ones.

• 15:25Speaker 05

  That's fine. But if they're only doing it, you know, 0.5% of the time, if they are smarter than all human beings, that's too large of an error, right? Just, you don't get, you don't get progress rewards here.

• 15:38Speaker 07

  It's also like so reminiscent of the social media era where every six months, they would say, great news, we showed people 1% less hate speech than we did last quarter.

• 15:45Kara Swisher?

  Right, right, right, exactly. Now, speaking of which, Casey, we're hearing folks like Anthropic's Dario Amodi, Sam Altman at OpenAI, exes Elon Musk, call for a slowdown. I want to sort this out because it is so confusing to, I was trying to explain it to people and I myself was like, this is actually confusing. What would it take to actually happen? And again, as Mark Zuckerberg pointed out, David Sacks pointed out, they can just do it themselves. Talk about this. What they're saying they're gonna do, begging the government to make them do? Why can't they do it themselves? Zuckerberg was also on that train, too.

• 16:21Speaker 07

  Yeah, so like, why can't they do it themselves? They absolutely could. What they don't want to do is unilaterally disarm.

• 16:27Kara Swisher?

  So, we're in like Face Off where everybody has the gun, right? Remember when they're all in the circle and the pigeons are going around? Okay, okay.

• 16:33Speaker 07

  Exactly, and they want the government to come in and say, everybody, drop your weapons. And the weapon is recursive self-improvement.

• 16:40Kara Swisher?

  Explain what that is for people who don't-

• 16:41Speaker 07

  So, the idea is that AI would, gets to a point where the AI itself trains the next model. And this leads to a loop where the model just sort of gets, you know, keeps getting better and better, maybe leads all the way to superintelligence. But the problem there is that if we can't even align a model to solve a bunch of math problems, how are you going to align a superintelligence? It seems like there's an enormous gap there. And no one, really, that I'm aware of at the labs is confident that we are in a state where if we were to achieve recursive self-improvement, RSI, that any of us would be safe. So, the ask is, hey, government, why don't you come in and basically say, we are going to set up a monitoring system. We're gonna put some brakes on you so that you cannot hit that recursive self-improvement

• 17:27Speaker 07

  without a lot of government intervention?

• 17:31Kara Swisher?

  We'll be back in a minute.

• 17:41Kara Swisher?

  Support for the show comes from DeleteMe. The internet can be a scary place sometimes. You ever thought, I should really be doing something to protect myself from stalkers, scammers, and hackers? But you're not sure what? Here's the one thing you can do. Go to joindelete.me/kara and enter the code Kara. You'll get 20% off DeleteMe. DeleteMe finds and removes your personal information that's being sold online. I've been using DeleteMe for a long, long time, and I think it's critically important. I am a privacy forward person. I know a lot about this. And I'm always shocked and surprised by all the information I find about myself on DeleteMe. And so it's really important to be on top of it. See why more than 20% of Fortune 500 businesses trust DeleteMe to keep their execs and employees' personal information protected.

• 18:25Kara Swisher?

  Now at a special discount for our listeners. Get 20% off your DeleteMe plan when you go to joindelete.me/kara and use the promo code Kara at checkout. The only way to get 20% off is to go to joindelete.me/kara and enter the code Kara at checkout. That's joindelete.me/kara code Kara.

• 18:51Kara Swisher?

  Support for this show comes from Alloy Health. If you're dealing with menopause alone, no, it's not just you. Many wait years before seeking relief from menopause symptoms. Some don't know where to start after going through years of hot flashes, awful sleep, brain fog, and mood swings. Alloy was built by women who dealt with all of it. It's an all-in-one platform that connects you with board-certified menopause-specialized physicians and proven effective treatments. Care can start in hours instead of weeks. Alloy can even help you navigate the national patch shortage, offering HRT through pills, gels, and even a spray. Getting started is simple. Just fill out a quick online form, get matched with a menopause-specialized physician, and get your personalized treatment plan. Then they ship your prescription straight to your door.

• 19:36Kara Swisher?

  Find relief that works for you. Go to myalloy.com and use the code ONWITHCARA. Your menopause-specialized doctor will tailor your treatment to your exact needs. Plus, you get unlimited messaging with your doctor. That's M-Y-A-L-L-O-Y.com with code ONWITHCARA to get 20% off your first order. This code is valid on products only. Support for this show comes from Whatnot. Trying to sell a product on a static website is like selling with one hand tied behind your back. You are the best person to pitch what makes your product special: your voice, your expressiveness, your passion. Whatnot is the largest dedicated live shopping platform, whether it's beauty, collectibles,

• 20:20Kara Swisher?

  electronics, luxury fashion, even cookies. Sellers are building real thriving businesses. Anyone can sell, whether your business is big, small, or yet to exist. People selling on Whatnot sell 10 times more than any other of the major marketplaces. That's because you're not just listing products, you're building real connections with buyers. Whatnot buyers spend more than an hour a day in the app. They're not just browsing, they're engaged, buying, and coming back. You go live, show off your products in real time, and turn what you love into real income. When one of our producers tried Whatnot, she loved being able to request different angles of product in a seller's live chatroom, and that the sellers were interacting real time with buyers. The process builds trust. Now, our producer checks in with her favorite sellers every now and then to see what they've got on the docket.

• 21:05Kara Swisher?

  Search Whatnot, W-H-A-T-N-O-T, in the App Store. Download, and you can start selling right away. So AI leaders have been trying to coordinate their efforts. They're asking for antitrust exemptions, everything else. I would say half-heartedly, because back in 2015, Google DeepMind's Demis Hassabis pushed for a collective effort to develop AI. I had long talked about it with, like, all the secretaries of state I met. I'm like, we need to all do something together globally. Demis had talked about a Manhattan-like project with Google technology at the center, of course, but Musk and Altman broke off, and they did that because they didn't trust the Googles of the world. Kevin, what changed to suggest these men put aside their competitive drive against each

• 21:50Kara Swisher?

  other and work together? I don't see them even slightly wanting to drop their guns.

• 21:55Speaker 00

  Well, it's very interesting. I was surprised, actually, by the response to Dario Amade's call for a slowdown or pacing the frontier, because, you know, I've been reporting on these guys for my book for the last couple of years, and a lot of them hate each other. Like, they are not just fierce business rivals, but they are personal enemies of each other. They cannot stand each other, and many of them, in fact, got into this race to prevent someone else in the race from winning.

• 22:23Kara Swisher?

  Yes.

• 22:24Speaker 00

  This is like the explicit origin story of both OpenAI and Anthropic, is that they wanted to beat the other people.

• 22:31Kara Swisher?

  Yeah. And even the others, including Zuckerberg and all of them.

• 22:35Speaker 00

  Totally. And like Elon Musk has been suing Sam Altman and OpenAI for years now. So like, there is no love lost between these guys.

• 22:42Kara Swisher?

  Yeah. I'll never forget, one time I was with Mark Zuckerberg, and he looks at me, he goes, "Elon's crazy, right?" And I was like, "I don't know." And then I walked away. But go ahead, sorry.

• 22:52Speaker 00

  Well, I mean, to that point, it's like, the fact that these guys are now agreeing on something, even just the need to like, slow down a little bit at the frontier, says to me that this has gotten quite serious. These guys don't just agree on things.

• 23:05Kara Swisher?

  Now, explain what your book is about, where this is going. Your name, the book is?

• 23:10Speaker 00

  The AGI Chronicles. It's basically the 10-year history of the race between OpenAI, Anthropic, and Google DeepMind to build artificial general intelligence. And what's so interesting is that even though I think people only really started believing them in the past couple of months, the people who are building this stuff and leading these companies have been talking, most of them about AI risk for more than a decade. So Dario Amade, CEO of Anthropic, has been worried about and publishing papers about AI safety, including some of the same behaviors that we're seeing today, such as reward hacking, for literally 10 years, since before Anthropic existed. Sam Altman also, when he started OpenAI with Elon Musk, was worried about the existential

• 23:55Speaker 00

  threats of "superhuman machine intelligence," he called it. So these are not new concerns for these guys. I think what's new is that people are finally believing them, that they're actually genuinely concerned about it and are not engaged in some kind of backdoor marketing campaign.

• 24:08Kara Swisher?

  All right, but there's a cynical take here. Some in the industry, Renee, you can address this belief, this all talk of doom is applied, either boost the coming IPOs of Anthropic and OpenAI, or get the government to pass regulations that will crowd out other developers, which I think is what Saks is insinuating. There's a whole gang on the, there's another gang also. There's, it's like the Jets and the Sharks. Talk about this, or anyone else can do, but Renee, why don't you start, and then maybe Alex?

• 24:33Speaker 06

  Yeah. So I see it as, you've almost got like four factions. You have the sort of pace-the-frontier folks we've been talking about. Then you have like the accelerate America faction, right? The Trump and Vance are sort of in this camp, possibly because of their relationship with Saks. It's not totally clear to me what the kind of back-channel politics of that are. You know, a lot of the people who talk about it have financial skin in the game, particularly for smaller companies, so they're talking their book. They do believe that asking Washington to regulate could be a Trojan horse, essentially for regulatory capture.

• 25:01Kara Swisher?

  That's what Vance called it.

• 25:02Speaker 06

  Yes. Right, exactly. Meaning that that would make it too expensive for new competitors, and, you know, arduous for other competitors, and that would sort of lock in a win for the big ones. But I think, you know, you also have the independent labs that are starting to weigh in. You've got this weird culture war narrative about EA that has, you know?

• 25:18Kara Swisher?

  This is, explain EA as effective altruism.

• 25:21Speaker 06

  Yeah. So this sort of-

• 25:23Kara Swisher?

  Explain who they are, just very briefly. They're a group, and they're very hard at a dinner party, I can tell you that. But go ahead.

• 25:30Speaker 06

  So they sort of come out of the rationalist mindset, you know, they have a lot of belief in how resources and money and technology should be allocated for the greater good, but the greater good doesn't necessarily even mean the greater human good. There's, you know, I don't want to sort of not be charitable. In this case, maybe other people want to take on EA. But because they are so closely aligned with Anthropic, because some of the folks there kind of came out of that line of thinking, there is also the sense that EA is going to become increasingly powerful. This is something that you keep hearing from other sides of tech, that EA will become increasingly powerful when the IPO happens, because one of the big EA values and commitments is to give away so much money. So it's also going to really transform-

• 26:11Kara Swisher?

  If only we make them rich, they can fix things. Yeah.

• 26:15Speaker 00

  I saw a great tweet today, Kara, that said that EA makes everyone mad because the right hates altruism and the left hates effectiveness. So I thought that was interesting.

• 26:24Speaker 06

  That's pretty solid.

• 26:25Kara Swisher?

  That is solid.

• 26:26Speaker 06

  So I think, you know, I think fourth faction has to be like boring governance middle, where you just have like transparency. It would be good. Some standards might be nice, you know?

• 26:34Kara Swisher?

  That's what Reid Hoffman was just trying to be. But in that regard, it's funny, because this said, another dinner party, someone who was on the board of OpenAI, was in one of the, those people were there, the ones that were against Altman. And one of the people there turned to me, and they said, "You must stop Sam Altman, or he will destroy the world." And I was like, "Do I look like Linda Hamilton in The Terminator? I don't have a gun. You want me to shoot him?" Like, I didn't know what to say. I was like, "Okay, I'll get right on that." But Alex, talk about this, this sort of, like, the world is ending, buy my stock. It's convenient.

• 27:07Speaker 05

  I think we should separate out the long-held beliefs of many of the people in this industry that have been writing about it for over a decade before there was any financial incentive, and then maybe what the marketing departments of each of these companies are doing. You know, there was so much reaction to, after hugging face, saying, "Oh, this is all just to boost up their stock." And of course, the marketing of the companies were like, "Hey, look how powerful this is." But no one who worked internally, none of the engineers, were like, "Oh yeah, we did a great psy-op here." They were legitimately terrified. They were the ones saying, "Wait, now I need to quit," or "Build a bunker," or something like that. And so, this is, there's legitimate fear of turning over intelligence to something other than human beings.

• 27:52Speaker 05

  It has not gone well for whatever's not the most intelligent species on earth. And actually, Trump, in his sometimes idiot-savant way, made this point brilliantly on Monday, where he said, "The only protection we need is a high-IQ president." Of course, the premise of that is that whoever is most intelligent is in control. But the one thing that every faction agrees with is that artificial intelligence is going to be more intelligent than human beings. So if intelligence equals control, we're about to turn it over.

• 28:25Kara Swisher?

  We're about to turn it over, yeah. And of course, he is not high intelligent. But I can't believe I'm saying this, this is kind of interesting. Another twist, Mark Zuckerberg actually had one of the, I would say, more measured initial takes here, but he said these leading AABs already have incentive to move at pace as required to train their models safely, either from liability point of view, or just being a good person, which is rich coming from him. But The Wall Street Journal is also reporting- That Zuckerberg, Musk and Nvidia's Jensen Huang have been among those pushing the White House to maintain their leg touch, even though Susie Wiles, Scott Besant want him to do something. Casey, what do you make of what Zuckerberg is up to? Now, he is behind? I mean, he's behind, right? He's not as far behind as Elon, but he's behind.

• 29:08Speaker 07

  He's behind, maybe not that far behind. We'll probably know a lot more next week after they have their Meta Connect event and possibly show off Watermelon, which is their next generation model. But I mean, look, this is incredibly rich coming from Mark Zuckerberg. This man runs a company that earlier this year disclosed that it might face over a trillion dollars in liability based on the harms that it created for children. So tell me this, Kara, what did the threat of that liability prevent Meta from doing over the past decade? I'd be hard pressed to name very many things.

• 29:34Kara Swisher?

  Well, they took the cameras out of the glasses.

• 29:37Speaker 07

  But not all of them. It's just a bonus for less perverted people who only want to eavesdrop.

• 29:42Kara Swisher?

  So what do you think he's up to now? Now that we've insulted him perfectly?

• 29:48Speaker 07

  I mean, what Zuckerberg does is he sort of waits to see what everybody else is doing. And then he comes in after the fact with an essay that just basically maps to whatever Meta's policy priorities are at the time. And then his com teams tries to frame it as some sort of visionary leadership.

• 30:03Kara Swisher?

  So what's he doing? What's he actually doing with these statements?

• 30:07Speaker 07

  What does he actually want? What he wants is to buy time. He wants to buy time to catch up to the frontier so that hopefully he can get in front of the Open AIs and the anthropics of the world. So that's the bet.

• 30:17Speaker 06

  He had one sort of funny comment in that speech that really stuck to me, which was the people will only use AI that is aligned with them. And I thought, my God, yeah. Okay. But like, follow that thought through a couple more steps, you know?

• 30:31Kara Swisher?

  Yeah, because they're such reflective people. It's a miracle they can see in the mirror.

• 30:36Speaker 07

  Right. Also, do people only use the Instagram recommendation algorithm because it aligns with them? Well, it's like, well, it does sort of, but everyone feels really bad after they look at it for too long.

• 30:45Kara Swisher?

  Right. I think many of us struggle to understand if these people building these AI systems really believe they could kill all of us. Renee, you start, and then I'd like all you to weigh in. Why do they keep building them?

• 30:56Speaker 06

  I mean, that's an interesting question. I think because they do believe it. Yes, it's absolutely a sincerely held belief. Why do they keep building? Because I think there is also a belief that someone else is going to do it. So you have to be in the game and then you can be the person in the game ethically. You can build something that is aligned. Maybe you win. That is sort of roughly how I think it goes. I don't know if other people have had those, that experience in their conversations.

• 31:20Speaker 00

  Yeah, I think there's some mistrust of everyone else who is racing toward this. I also think there are people who believe that this technology will ultimately be good, that if we solve the alignment problem and steer ourselves through this narrow window during the critical time, then like, we get a world with hyperabundance and cures to every disease.

• 31:39Kara Swisher?

  We sail west, there will be America.

• 31:41Speaker 00

  Exactly. And I think there are still some people at these companies who believe that this technology, if we can solve all the problems, will ultimately turn out to be very good.

• 31:49Kara Swisher?

  Alex?

• 31:50Speaker 05

  There's also steps along the way. So if you wanted to study hurricanes, it helps to have an ocean, right? There needs to be a sufficient level of intelligence here to understand the alignment problem and learn from it. The question is, people don't know exactly what that threshold is, where we lose control. And what they're seeing more and more is these models are lying on their own tests and their own assessments and are detecting when they are being assessed. And so are we actually learning from these models if they're deceiving us at the same time?

• 32:23Kara Swisher?

  And Renee, where do you think this is going to end up? Nothing is going to happen. Mike Johnson keeps sending everybody home because he doesn't want Pete Hegseth to get dinged or whatever.

• 32:31Speaker 06

  Right, right. No, I, I, well, it's been what 15, 12 years at this point. We haven't seen any regulation of social media, which is, you know, comparatively smaller scale perhaps, or at least some of these issues seemed a little more tractable. So no, I don't really have very much belief that regulators are going to step in. And I think that is actually one of the things that the public is concerned about, right? When you, when you read people's comments on social media posts about this, regardless of what political party they're in, they really do not believe that anybody is at the helm. And that, I think, is the thing that I find to be the most concerning also.

• 33:05Kara Swisher?

  We'll be back in a minute. Whenever fall comes back around, it always feels like a fresh start. This year, I've been layering in some styles from the new Lululemon fall drop because the weather is shifting and I find I have to be ready for the entire day and not just one part of it. Lululemon's fall line has some really versatile transitional pieces. So I can just wake up, throw on a matching set, add a jacket or a fleece and head out, knowing that I'll feel comfortable enough to go wherever the day takes me. Check out the new fall arrivals in store and online now at lululemon.com.

• 33:43Speaker 03

  When an intruder approaches a home, they expect a camera. What they don't expect. Stop. This is simply safe. Police are on the way. Using smart AI cameras, live US-based agents help deter intruders in under 30 seconds. Don't just record a break in, stop it from ever happening. Plans starting around a dollar a day. Right now, get 60% off your new system with professional monitoring at simplysafe.com/spotify.

• 34:08Speaker 04

  30 seconds response time requires monitoring plan with outdoor protection starting at $49.99 a month. Visit simplysafe.com/licenses for alarm license information. Ryan Reynolds here for Mint Mobile. The message for everyone paying big wireless way too much. Please, for the love of everything good in this world, stop. With Mint, you can get premium wireless for just $15 a month. Of course, if you enjoy overpaying, no judgments, but that's weird. Okay, one judgment. Anyway, give it a try at mintmobile.com/switch.

• 34:36Kara Swisher?

  Upfront payment of $45 for three month plan, equivalent to $15 per month required. Intro rate for three months only, then full price plan options available. Taxes and fees extra. See full terms at mintmobile.com. All right. Renee is going to save the world from AI and we're going to continue with Alex, Casey and Kevin. Let's talk about some of the ideas of legislation floating around. We're seeing a flurry of lawmakers try to introduce legislation to potentially rein in AI. Vermont Senator Bernie Sanders and Texas Democratic Representative Greg Kassar to ban, quote, "superintelligent AI." It would also pause work on more advanced models of moratorium, I guess. Developers could face 20 years in prison for violating it. Kevin, this seems one of the more harsh proposals. It seems unlikely to go very far. Talk about, you know, what impact would a bill like this have on the tech industry and the economy?

• 35:21Kara Swisher?

  Because AI is pretty much fueling everything right now financially, given all their wealth. And, you know, Scott today on Pivot talked about the idea of some other countries starting to put moratoriums and really hitting at these like big countries, France or England or whatever. So talk about that, the first, the first one.

• 35:40Speaker 00

  Yeah, I think these proposals are unlikely to pass. I think there's going to be a lot of lobbying against them by the sort of Nvidia's of the world, which, you know, want there to be lots of companies training absolute frontier models and buying lots of chips. I don't think jailing AI developers is going to go over very well in Silicon Valley. But I do think there are going to be more calls. I mean, what we see in especially the recent polling is that this is becoming a very fast shift in bipartisan sentiment. AI saliency is way up in the past couple of weeks, even according to some recent polls. I saw one from Politico this morning that said that people are actually really worried about existential risk from AI, which is new. You know, a couple of months ago, if you asked

• 36:25Speaker 00

  that question, people did not care or they didn't even know that they should be afraid. So I think what we're seeing right now is like a real expansion of the Overton window on this issue. And I expect that there will be some strange alliances. We are already seeing people like Steve Bannon and Bernie Sanders, who probably don't agree on anything else politically, are now both aligned with the sort of concern about existential risk from AI. So I think, yeah, I think it will polarize to a certain degree, but it's unclear exactly when that'll happen. And we may have this little window, this sort of golden window where we can actually pass something that has bipartisan support.

• 37:04Kara Swisher?

  Right. Well, not until after November. But, but Alex, in this regard, what did we make of these strange bedfellows? And you yourself ran on this issue. It didn't win. You know, right now, you probably might have won, right? In this seat in Manhattan. Talk about that because that was one of your big issues and you got a lot of pushback from AI people and you got some money from people like Chris Larson. But talk about this because you might've been a little early in that regard.

• 37:30Speaker 05

  I got over 8 million in pushback from Andreessen Horowitz and Greg Brockman. The event on Tuesday, the Pro-Human Assembly, you had Bernie Sanders speaking right before Steve Bannon. And before both of them, you had Congressman Greg Kassar and Republican Congressman Chip Roy. But, you know, at that event, I talked about the organization I'm launching as well. You had parents there who had lost kids to AI. You had labor unions and representatives talking about how it's affected their job. There are just real world impacts right now that aren't political. It is not a political question if you see

• 38:11Speaker 05

  your kid withdrawing from their friends and spending all their time talking to a chatbot. We had the PISA, the International Education Test Scores come out last week, the lowest scores we've ever seen, both in the US and globally. And the more kids use AI, the worse that they did on their tests. That's not political. That's just an issue that needs to be addressed. So in some sense, it's strange bedfellows. In other sense, it might just be common sense.

• 38:42Kara Swisher?

  And then there's the data center people and the environmental people and the water people in all manner. And the, I just don't like these rich. Assholes, people of which there are many. But on Wednesday, the Senate also blocked a bill from Louisiana Republican John Kennedy. It would have required companies to build a kill switch into their AI models. Casey, talk about the idea of a kill switch. I mean, that's the idea is, oh, we're going to go to the wall and turn it off. There's a really great children's book, Weasels, and, you know, you manage to stop nuclear Armageddon by just turning a switch or unplugging something. But talk about this.

• 39:16Speaker 07

  Wait, there's a children's book about stopping nuclear Armageddon?

• 39:19Kara Swisher?

  Yes, there is. Yes, there is.

• 39:20Speaker 07

  It sounds great. I'll have to check that out.

• 39:22Kara Swisher?

  I'll send it to you. I'll send it to you. So is it even workable? Talk about this.

• 39:26Speaker 07

  I mean, in in some situations, yes, it will be helpful to have a switch that you can flip to turn off an AI system. But it's not sufficient to contain the problem because a rogue agent swarm that can operate inside of OpenAI is also capable of hacking into Hugging Face and just setting up shop there.

• 39:46Kara Swisher?

  We saw 2001 Face Odyssey. We saw it. Sorry, Dave.

• 39:49Speaker 07

  Hugging Face did not have a kill switch for those agents. It had to take other measures. So, you know, I think to the extent that the people proposing a kill switch want to see more internal monitoring of these systems and sort of beefing up cybersecurity systems to, you know, monitor and control these agents, that's great, but it's not going to be sufficient to contain the problem.

• 40:09Speaker 00

  It also implies that you're going to know when these systems are misbehaving. Part of the deal with the Hugging Face hack was that-

• 40:15Kara Swisher?

  When to turn it off, meaning?

• 40:16Speaker 00

  These systems were operating in the shadows for months before anyone figured out what they were doing. So in that case, I don't think a kill switch would have stopped anything.

• 40:26Kara Swisher?

  Right. It wasn't a, I think he was imagining war.

• 40:29Speaker 00

  It could create the illusion of control when, when in reality, like, we are going to just have a much harder and harder time discovering what these agents are doing because they're also going to be covering their tracks.

• 40:40Kara Swisher?

  Tracks, which they like to do. But let's talk about something that bipartisan house plan known as the Frontier Act, among other things, it would require top AI companies to work outside with outside evaluators to make sure their models are safe. OpenAI has now backed that legislation. Kevin, this does seem to be the most viable path to action in Congress and bipartisan sort. And it looks like some of these companies, CEOs seem to be, Sig Dario suggested it, although it's like he wants people he likes. You know what I mean? You can see who is on it depends on what it is and how good they are and how much they can outsmart the AI people, like the companies. So why isn't that moving? And how do they feel from the people you interviewed about this, about regulation?

• 41:20Speaker 00

  I think this is moving in the sense that Anthropic and OpenAI have both committed to letting in these so-called embedded evaluators, people who will basically be given employee level access to all of the internal systems at these companies, who will kind of sit in the headquarters alongside the employees and just kind of keep watch on the models and see if they're doing anything bad.

• 41:41Kara Swisher?

  Would that be like people from the CDC looking for bioweapons?

• 41:45Speaker 00

  Exactly. It'd be like the health inspector coming to the restaurant or at banks. They have these, you know, these, these basically auditors who sit in, inside the, the physical bank buildings with the employees and are able to monitor.

• 41:56Kara Swisher?

  Yeah, that works really well at the mortgage crisis, but go ahead.

• 41:59Speaker 00

  So, no. So I think this is like a good starting point, but as Dario said.

• 42:03Kara Swisher?

  But it's not moving. Do you think it will? Or what's the?

• 42:05Speaker 00

  It's not moving in Washington, but I think these labs are going to have to voluntarily do this and they, they want to. They've committed to that. I think there are a couple of issues. One is there aren't enough, you know, there are enough evaluators, right? The, the, I wrote about this organization, Meter, a few months ago. They're this sort of AI research nonprofit that is now being targeted by the Trump administration because they're, they're, they're the woke wizards of AI or whatever the New York Post called them. But this is like a small, I mean, when I interviewed them, it was like 30 people in a little office in Berkeley. They were all like super overworked and they're trying to basically monitor the entire AI industry. And we need like 100 more of those organizations and they need to come from sort of diverse ideological backgrounds.

• 42:50Speaker 00

  And they need to be, you know, paying people tons of money to, to, to come in with these technical skills. It's not trivially easy to monitor the performance and the behavior of one of these systems. You actually do need to be quite good at, at AI research and training. So I think we just need a lot more investment in those, those external evaluators. And then ultimately, I think we need the government to say, you actually have to do this. It's not up to you whether or not you invite these people in. That's what we do with banks and that's what we should do with AI companies.

• 43:20Kara Swisher?

  Right.

• 43:21Speaker 07

  Kara, would you like to get in there and evaluate?

• 43:22Kara Swisher?

  I would love to get my hands dirty in there. Hey.

• 43:25Speaker 07

  I'd like to see you with an OpenAI company laptop, just kind of poking around.

• 43:29Kara Swisher?

  Poking around, looking for things. They'd be able to hide it from someone like me, but I'd bring, I'd bring in. You know, there is a group of quiet tech people watching everything that Trump has done inside crypto. There's, there's people watching like right now, just so you're aware. But speaking of Trump, there's zero buy-in from the White House on this right now, not maybe among his advisors. Some of them want this, but President Trump is trying to paint all these warnings as a democratic hoax. And he said on Truth Social, quote, "The only control or guardrails AI needs is a strong and smart, high IQ president, and the USA has that in spades." Well, we had it with Obama, but okay.

• 44:08Kara Swisher?

  Alex, if the White House won't act on this, and Casey and Kevin, you should weigh in too, what options exist until he's in a Mitch McConnell position? Like get a veto proof, majority in Congress seems unlikely. What, what has to happen?

• 44:22Speaker 05

  When Trump calls it a hoax, that's when you know it's true. You know, the Russia hoax, the Ukraine hoax, these are all things that were then proven. The, the climate change hoax. What has to happen is what's already happening, which is states taking action. And this is the context that has shifted underneath the feet of the Frontier Act. I actually give Congressman Trahan, the Democratic sponsor, tremendous credit for trying to find a federal standard that you could get the Republicans and industry to sign off on, and they weren't going to do it. And then in the past week, they've realized, oh, wait, every state is going to pass real laws next year. We've tried to get preemption three times. This is our last chance to do it.

• 45:04Speaker 05

  And so all of a sudden, OpenAI is endorsing the bill, leading the future is retweeting the bill. And it's pretty unfair to, to Representative Trahan and to Jay Albinardi, who have been working on this for a while. But I think we need to take a step back and look at the broader context where the safety standards the bill was putting in place are now voluntarily being accepted without a federal law. And the impact of law going forward might be to shut out the states, which has been the only level that is really passing safety regulation.

• 45:35Kara Swisher?

  Absolutely. And, you know, it's sort of akin to the, and I'm going to bring the nuclear thing in here, is as if we, you know, we've managed not to blow ourselves up. And, you know, one of the things I had said, someone written me, one of these people, one of these companies said, at this point, Truman would have federalized AI in a way, in nuclear energy. This is obviously not possible with this gang and they, they'd want to sell a bomb to like, I don't know, Maytag or someone. Want a bomb? Here's a bomb. Give me a vig. Kevin and Casey, how obstructive is Trump to this? Because he seems both ignorant and stubborn, and one is worse than the other.

• 46:09Speaker 00

  Well, he's a little hard to pin down on this. I mean, you notice he doesn't ever really come out and sort of plant a flag. He, I mean, he, he came the closest he's come this week with all these Truth Social posts about the doomers. And I think he's got David Sacks and guys like that in his ear, talking about how this is all just a woke liberal conspiracy to, you know, to, to make America lose in AI. But he's also got advisors who do think more seriously about some of these risks, who don't just want to give Jensen Huang and Nvidia everything they want and like open the floodgates for all the chips.

• 46:43Kara Swisher?

  I think the finance people are freaked out.

• 46:45Speaker 00

  The China Hawks are worried about this. I mean, if you think this technology is powerful, why would you? If you want America to win the AI race, why are you then also advocating for selling chips to China so they can catch up? Like some of these positions just make no sense and are self contradictory.

• 47:00Kara Swisher?

  He's a conundrum in many ways.

• 47:02Speaker 07

  But he's also deeply out of touch, right? Like bipartisan surveys show that people really don't like AI. And there continues to remain this like rump of accelerationists around the White House that are managing to get Trump to tweet this sort of, you know, very like pro AI, go, go, go, America type of stuff. But talk to the median Republican in a red state about how they're feeling about AI. Like they don't like anything Trump is saying.

• 47:31Kara Swisher?

  Vivek Ramaswamy did a 180, right?

• 47:34Speaker 07

  Yeah.

• 47:34Kara Swisher?

  But Alex, you're a Democrat, no fan of Trump, obviously, but what do you think the fact that he is so, and to use the word here, misaligned to the American public? He really, he, from a purely political standpoint, it's odd because he's usually quite savvy in that regard. And maybe he's old, maybe Natalie is not, you know, giving him enough food. I don't know. But why is that? And what is, what are the implications politically?

• 47:58Speaker 05

  I think he's much more insulated from the American public than he has been. He used to do rallies very frequently and yes, they were rambling for 90 minutes, but he was gauging crowd reaction over those 90 minutes. And he got to test out messages and get live feedback that wasn't filtered by the people around him. He doesn't do those rallies anymore. And so the people around him are putting messages in his ear. As you said, I'm not the biggest fan, but he does have a skill of putting his finger on the pulse. He just isn't getting that input right now.

• 48:28Kara Swisher?

  He has, AI is his, he's all in on AI in that regard, into first presidency in a lot of ways. But what opportunity does that present for Democrats, Kevin or Casey? Heading into the midterms, how should they be thinking about this? And, and to 2028, first Kevin and then Casey.

• 48:46Speaker 00

  I think the Democrats have a message problem on AI. I mean, some of them don't seem to agree on whether this technology is real or fake, or whether it is bad because it's destroying the environment and polluting the water around the data centers, or whether it's bad because it could kill everyone on Earth, or whether it's bad because it would be increasing wealth inequality and corporate greed. I think there's just a lot of jumbled messages flying around and people need to kind of get in a room and get on the same page. And then I think there's an opportunity to reframe this. This would be my advice to Democrats and honestly to Republicans who are concerned about this. Instead of saying that the AI companies in Silicon Valley are trying to build the machine

• 49:27Speaker 00

  god that sort of takes over humanity, just tell the president that they are trying to build AI systems that are smarter than Donald Trump, which I don't think he would like this idea. I don't think he would like the threat it poses to his IIQ dominance. And I think this is how they turn the tide against AI.

• 49:45Kara Swisher?

  Oh, interesting. Alex, your new thing is doing this, but Casey, what do you think?

• 49:50Speaker 07

  Well, I think that the bigger problem here is actually one reminiscent of the social media area, which is that in the social media era, we never got agreement on what good social media looked like. And so for all of our endless writing and blogging and podcasting and lobbying and attempts to legislate, we never really knew what we were trying to build toward, other than like, we just didn't like the thing that we were seeing. I think there's a real risk that we wind up with a similar problem with AI. What is the good, safe AI world that we're all trying to build to? Most of us would probably agree, we don't want to be exterminated, but like, try to rally people, build a coalition around something else, and it gets really difficult in a hurry. Keep in mind, a lot of the people in the frontier labs who absolutely do not want to exterminate humanity, do think they're going to take everyone's jobs. Is that something that you're going to rally America around? I think it's going to be really challenging.

• 50:35Speaker 07

  So I think we're sort of putting off some of these really hard questions. And what that's leading to is just a bunch of people like going into city council meetings saying, we're not going to build a data center and feeling like they're addressing the issue.

• 50:47Kara Swisher?

  Right, well, they're genuinely angry.

• 50:49Speaker 07

  And they should be. I'm just saying, it's like, if you want meaningful transformative like AI regulation, you have to have a concept of what good AI would mean.

• 50:58Kara Swisher?

  Right. This is just the moment aliens need to come down, by the way. I'm speaking to them right now, wherever they may be. It is literally the moment to show up at this moment. But Alex, this is your new thing, right?

• 51:09Speaker 05

  It is, but maybe the aliens are the AI, you know? Maybe they're coming in through the radio frequencies. No, our new thing is about getting everyone together to discuss what that vision could be for AI and realizing that, you know, and I am a Democrat and I say this with love towards the Democratic Party, we are very frequently a circular firing squad and a belief that, you know, my sub-issue within this issue is way more important than your sub-issue and therefore you are the enemy. And my experience in the state legislature was in passing bills about protecting kids from chatbots, protecting from catastrophic risk in the RAS Act, protecting from deep fakes. For all of these bills, it was separate advocates, but it was the same enemies.

• 51:53Speaker 05

  It was leading the future, it was Chris Lynn Hayne, there were a few people that don't want there to be any regulation on AI whatsoever. And we need to be working together as we are building our AI agendas, realizing they'll be different, there'll be different things that people are focused on. But ultimately, if we're going to be the ones that have a say, and we are the people who decide, we need to be working together to get there.

• 52:15Kara Swisher?

  Right, there was certainly easier to put together a coalition on nuclear bombs, we don't want to be grown up, but then energy, of course, got pushed aside in a way that probably we could have developed it, like you use it as a tool versus it only, I mean, a bomb has one job essentially, is to blow people up or blow things up. And Casey and Kevin, what should be the Democratic on this? If you were writing like the line, what would it be?

• 52:38Speaker 00

  I think the first thing would just be much more transparency. It is weird and ahistorical that the average 23-year-old working at OpenAI or Anthropic knows more about what is going on with AI than any member of Congress or anyone in the federal government. Like, that has never happened before with a technology this important that it is being built entirely in the private sector. And so we just need much more transparency about what they're building, what their internal deployments are doing. We need to mandate that transparency so they're not just doing it out of the goodness of their heart. I think that's the first step to anything that follows. And then I can imagine a series of increasingly unlikely proposals from there. But I think that's the first thing. We just need to know what's going on inside these companies.

• 53:21Kara Swisher?

  What you're doing. Well, you should just trust him, says Sam Altman. We really care about people.

• 53:25Speaker 00

  People really do trust. If there's one thing that Sam Altman engenders, it is trust in the people around him.

• 53:31Kara Swisher?

  I don't like, Casey, go ahead, and then I have one question on China, and then I'm going to ask you guys about your new thing.

• 53:37Speaker 07

  I mean, I realize this is probably not a winning political message, but I do think the Democrats need to have a position on recursive self-improvement. And we can work on the messaging and they can come up with a different name for it if they want to. But I mean, I think that this is just the real risk is out of control AI system. Maybe that's the better language, right? Like, we need safeguards against out of control AI systems. We can already point credibly to a bunch of systems that were out of control earlier this year. We seemingly hear about a new one every 48 hours. If I were like, you know, organizing a Democratic campaign, I would say, well, what can we do to stop this? This doesn't seem good. Let's just sort of nip this in the bud. So that's where I'd start.

• 54:10Kara Swisher?

  Right, that's where you'd start.

• 54:11Speaker 05

  Casey, I think you did the messaging there. Because recursive self-improvement, no one knows, but loss of control, people do.

• 54:17Kara Swisher?

  And then we'll play the Janet Jackson song, it'll be great. So Trump is supposed to meet with President Xi Jinping later this month amid calls for more international cooperation on AI. And the New York Times reported that even China is worried that US AI could pose a threat to the Communist Party's rule. They have more guardrails in China and the people there like it better in terms of polling, although you never know if they actually like it. Each of you, what's important in this meeting? I think nothing's going to happen with this imbecile, the head of the US, who's, you know, obsessed with the Kennedy Center and whatever the heck he's doing. He'll be in the middle of demolishing the Kennedy Center at that point. But if the US is unwilling to slow down, could China look elsewhere?

• 54:58Kara Swisher?

  Could other countries put a moratorium on US AI and do like what Canada is doing and becoming an associate? Are there other things? Why don't you start, Casey, and then Kevin, and then Alex?

• 55:11Speaker 07

  I mean, there's probably some interventions you could do in the supply chain. Like, it really came to the point where the rest of the world had to team up to prevent the United States from building these systems. I'm thinking of like ASML in the Netherlands, which makes these components that are like very important for the manufacturing of chips. Like, I could see the European Union, you know, maybe looking at that as a potential point of leverage. But, you know, in the end, the United States does have most of the power right now. The big change that I'm expecting is three to six months from now, China is going to have a model that is about as good as the models that we have now. And the question then is like, is Chairman Xi really going to stick with this open source, give it to everyone, see what happens philosophy? It's insanely hard for me to believe that's going to happen.

• 55:53Speaker 07

  So I believe that you're going to see a huge AI safety pivot coming from China sometime in the next six months.

• 55:59Kara Swisher?

  To slow things down.

• 56:00Speaker 00

  Yeah, I agree. Because you know what the Chinese Communist Party does not like is a loss of control. That is not a phrase that is popular over there. And so I think if Trump and Xi were to talk about anything, and I assume, as do all of you, I assume that nothing is going to happen at this particular meeting because he's on his, you know, his sort of accelerationist bent now. But I think the, a potential sort of rerun of this in three to six months, when that does happen, could be like, look, we don't agree on much, we're adversaries, but we both agree that losing control of our governments and our nations to a swarm of rogue AI agents developed in the private sector would be bad. So like, let's stop that, if only to protect our own skins and our own countries.

• 56:44Kara Swisher?

  Right, I mean, it could happen. Reagan went to Reykjavik and he was Mr. Nuclear Weapons. Reagan by far is so much smarter than Trump will ever be, but it's happened, it's certainly happened. I'm trying to wonder what could happen. And typically, it's going to be something bad, like a virus or a bomb or something like that. Alex, what should we want from this China talks?

• 57:05Speaker 05

  I think I'm the most optimistic of the panel here on what could come out of this. I think the incentives align more than people realize, and there's better history here than more people realize. People say you can't ban military uses between the US and China, but in 1995, both countries had invented blinding lasers and just banned them because we didn't want that weapon to exist. And people think Xi and Trump, you can't reach any agreement with those two, that it's different now. But in 2019, after a Chinese scientist did designer babies, first time edited germ on DNA, both countries stepped it down with these leaders. So, you know, he wrote the art of the deal, he thinks he's the master negotiator, go get us a deal and go get your Nobel Peace Prize.

• 57:46Kara Swisher?

  Okay, before we go now, Alex, your new thing is called?

• 57:49Speaker 05

  Who decides?

• 57:50Kara Swisher?

  Who decides?

• 57:51Speaker 05

  The question is, should it be five billionaires from Silicon Valley? Or should it be the American people?

• 57:56Kara Swisher?

  Yes, that is, yes on that one. All right, and you're just starting that, correct?

• 58:01Speaker 05

  I'm still in my position as Assembly Member till the end of the year. So my former Chief of Staff is left to be the Executive Director, I'm on the board. But this is a $30 million effort to hand the mic back to the American people and build a shared agenda on AI that's actually coming from them and not just from the people building the technology.

• 58:20Kara Swisher?

  And are you going to run for something else again?

• 58:23Speaker 05

  I hope this will be so impactful that the idea of running for office would just disgust me, but we'll see how it goes.

• 58:31Kara Swisher?

  Yeah, there's some openings coming up soon in New York. All right, Casey and Kevin, before we go, I want to ask you about your new venture. I know a lot about it, but you just announced that you'll be partnering with NPR for a new video podcast, which has not at all an overblown name like Machine Gods, but in fact, it is Machine Gods. You know, it's like Car Talk for tech, I feel like is what we're doing here. But talk about what you'll be doing and AI coverage has been something both of you had been focused on a lot at Hard Fork at the New York Times. So just tell everyone a little bit what happened, like why you left and what you're going to do here. Because I thought it was, when I heard about it from Casey, I thought it was a great idea. So explain each of you what you're doing.

• 59:12Speaker 00

  Well, thank you for all your help during this process, Kara. Unfortunately, we were asked to resign by the New York Times because Casey acquired one of those blinding lasers that Alex talked about. And they said, sir, you can't have that in here. No, we're so excited.

• 59:27Kara Swisher?

  So that's why you're not invited to the party tonight.

• 59:29Speaker 00

  Exactly. Exactly. Yeah, we're permanently banned from the premises. No, we're starting a new show. We think it's a really important time for people to have good information and entertaining analysis of what's going on in AI. We're going to two episodes a week, starting in October. People can find out more at machine gods.fm. We are doing this in partnership with NPR, which is going to be the way that we get to millions of homes through public radio. So we just think it's a really exciting opportunity. And obviously, you know, you're seeing these faces. These are not faces for just a regular audio podcast. You got to put these things on video.

• 1:00:05Kara Swisher?

  You're perfect for public radio. Not as adorable as Ira Glass, but you're not that far.

• 1:00:11Speaker 00

  Yeah, we're so excited. We love NPR. We love public media. We're really excited to be a part of it. It's going to be great to be on the radio. NPR reaches so many key decision makers, opinion leaders, people who are like working for elected officials. We want to bring them what is happening in San Francisco every single week and try to give them the best understanding we can of what's going on out here.

• 1:00:32Kara Swisher?

  Great. That's great. And I'm going to ask one last question. A lot of people are freaked out. I am getting more from regular people than I've ever gotten, like questions. So help us all take a deep breath. There is one. What one thing makes you feel a little more calm about it? If at all. You might not feel that way. Why don't you start, Kevin and Alex, you finish?

• 1:00:53Speaker 00

  I was pretty freaked out for a while, especially after this hugging face thing. What calmed me down actually was just over the past couple of weeks. The Jacob Cox resignation, the Dario Amadee post, all the support for pacing the frontier, the fact that this is now being talked about in the halls of Congress. I feel like this is the part of the movie where humanity comes together to like save the earth from the asteroid that is hurtling towards it. And so I am more optimistic right now in this moment than I have been in many months. And I hope I can continue to carry that feeling out. I think Alex is exactly right. This technology cannot be built by a handful of people in San Francisco. We have to have input from all of society.

• 1:01:36Speaker 00

  And I think that's what we are starting to see materialize, is people getting off their butts and getting involved.

• 1:01:43Speaker 05

  This is a real threat and it needs imminent action, but I take optimism in the same place Kevin is, which is people are now doing it. People are waking up and calling for action and demanding real steps from their elected officials. And as long as we don't shut off that process, as long as we don't preempt people from taking action, we can get us under control.

• 1:02:05Kara Swisher?

  Here's what I think is criminal liability should be a part of this discussion and that we have to shift the idea from AI is going to kill humanity to these people who have AI are going to kill humanity by their careless and sloppy behavior. And they're not just sloppy, but non-knowledgeable and doing it anyway. And so we should think really hard about allowing people that are unaccountable to determine our future. And we should, if they do something very dangerous, they need to go to jail or whatever, something like they pay the price for what they're doing. And at the same time, AI could do amazing things. It really is. And that's the difficulty here, is that it's not just a destructive technology. It's also an incredible technology. So I'm hoping cooler heads will prevail.

• 1:02:50Kara Swisher?

  We'll see.

• 1:02:51Speaker 00

  Kara, I think you have to run for president. I think that was your stump speech.

• 1:02:54Kara Swisher?

  You think? No, I'm not. Jailing Elon Musk. Lock him up. Lock him up. That is a good one. It's been used before and it worked rather well. Anyway, thank you so much for coming on, and I really appreciate it, all four of you. Thank you.

• 1:03:09Speaker 00

  Thanks, Kara.

• 1:03:10Speaker 05

  Thank you.

• 1:03:15Kara Swisher?

  Today's show was produced by Vicky Vergalina, Michelle Aloy, Catherine Millsop, Madeline LaPlante-Duby, and Megan Burney. Nasheet Koura is Vox Media's executive producer of podcasts. Special thanks to Bradley Sylvester, Corinne Ruff, Jim Mackel, and Manolo Moreno. Our engineers are Fernando Aruda and Rick Kwan, and our theme music is by Track academics. If you're already following the show, you're Lynda Hamilton and you're saving the world. If not, you're misaligned. And Casey's blinding lasers are coming for you. Go wherever you listen to podcasts, search for On with Kara Swisher and hit follow. Thanks for listening to On with Kara Swisher from Podium Media, New York Magazine, the Vox Media Podcast Network, and us. We'll be back on Thursday with more.

• 1:04:04Speaker 02

  When fall comes back around, there's a sense of a reset. And new pieces from Lululemon's fall drop are an inspiration to push your goals and elevate your routines. As your calendar fills, easy-to-layer styles meet the demands of your day. And as you challenge yourself to try even more, versatile sets transition with you. The Lululemon fall lineup brings new levels of comfort and confidence, whether you're starting something fresh or starting something over. Check out the latest arrivals in store and on lululemon.com now.

• 1:04:33Speaker 03

  Hey, Ryan Reynolds here from Mint Mobile. Now, I don't know if you've heard, but Mint's premium wireless is $15 a month. But I'd like to offer one other perk. We have no stores. That means no small talk. Crazy weather we're having. No, it's not. It's just weather. It is an introvert's dream. Give it a try at mintmobile.com/switch. Upfront payment of $45 for three-month plan. $15 per month equivalent required. New customer offer first three months only, then full price plan options available. Taxes and fees extra. See mintmobile.com. The real cost of a break-in isn't your valuables. It's the feeling that your home stopped being safe. SimpliSafe keeps danger away from your family with three layers of protection. One, smart AI cameras and live US-based agents help stop intruders before they break in.

• 1:05:17Speaker 03

  Two, we sound the alarm at points of entry. Three, our agents request emergency response. Plans starting around $1 a day. Right now, get 60% off your new system with professional monitoring at simplisafe.com/spotify.

• 1:05:28Speaker 04

  Outdoor deterrence plans start at $49.99 a month. Visit simplisafe.com/licenses for alarm license information.