AI Whistleblower: OpenAI Scandal, AI Cults, Neuralink & Our Last Chance to Stop the Tech Oligarchs

Nate Soares is a computer scientist who’s worked at Google and the Defense Department. So when he says AI is on the path to killing every person on earth, it’s worth hearing him out. (00:00) Why Is AI Dangerous? (17:43) The Unpredictable Evolution of AI (20:13) The US and China’s Shared Interest in AI (37:20) The Tech Oligarchs More Powerful Than the Government (1:20:51) Can We Survive This? Nate Soares is President of the Machine Intelligence Research Institute (MIRI), where he has led AI safety research for more than a decade following engineering roles at Google and Microsoft. He is the co-author, alongside Eliezer Yudkowsky, of the New York Times bestselling book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (2025), which argues that the current race to develop super intelligent AI will end catastrophically unless humanity globally coordinates to stop it. Nate has published extensively on AI alignment, including foundational work on decision theory, goal specification, and the incentives that drive advanced AI systems to seek power in ways their creators neither anticipate nor intend. Paid partnerships with: Preborn: Help Provide Ultrasounds Dial #250 and say “BABY” or visit https://preborn.com/TUCKERDefend: Enter code "Tucker" for 20% off your purchase at https://defendcellcam.com American Financing: NMLS 182334, nmlsconsumeraccess.org. APR for rates in the 5s start at 6.327% for well qualified borrowers. Call 800-685-5696 for details about credit costs and terms. Visit http://www. AmericanFinancing.net/Tucker. TCN: Get your copy of ‘Able Danger' only on https://TuckerCarlsonBooks.com/Tucker Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 0:34Tucker Carlson?

  Nate, thank you so much for doing this. You've written the world's darkest book. You've devoted your life to warning the world about the potential dangers of AI. So let's just start by hearing your explanation of why AI is dangerous in addition to just being annoying.

• 0:55Speaker 02

  Yeah, you know, the very basic common sense point is

• 1:01Speaker 02

  if you race to make machines that are much smarter than any human, to make machines that are more capable at inventing their own technology than humans are,

• 1:14Speaker 02

  and you race into this without really knowing what you're doing, the most likely outcome is just that the machines get loose and do their own thing, and that humanity dies as a side effect, just like humanity has, you know, killed lots of other animal species, not because we hate them, but just as a side effect.

• 1:34Speaker 02

  And in some sense, a lot of people find it sort of obvious or intuitive that if you just like make these really smart, really powerful machines, why would they care about us?

• 1:46Speaker 02

  And I think that intuition basically is right. And there's a lot of arguments you can have on each side. You can get into technical details. But my book is basically just

• 1:59Speaker 02

  getting into those details and saying like, yep, it sort of holds up. If we make smarter machines without knowing what we're doing, it's just going to go poorly.

• 2:09Tucker Carlson?

  Are we actually going to make machines capable of what you're describing?

• 2:14Speaker 02

  You know, the companies are trying to. They they talk about how they are pursuing superintelligence in the true sense of the word. That's Sam Altman's phrase. Dario Modie of Anthropic says they're trying to make the equivalent of a country worth of geniuses in a data center. So these guys are really, you know, these aren't chatbot companies. They didn't set out to be chatbot companies. They set out to make these machines that can sort of exceed humans in every way. And that's what they're targeting. There's a separate question of whether they will get there.

• 2:46Tucker Carlson?

  Let me ask you a question. Why would anyone want to build a machine smarter than people?

• 2:51Speaker 02

  I think that

• 2:55Speaker 02

  a lot of them hope that they'll be able to make the world much, much better. You know, they hope for a cure to cancer. And then more than a cure to cancer, they hope for a cure to aging. They hope for,

• 3:08Speaker 02

  you know, a thousand years' worth of technological development compressed into two years.

• 3:16Speaker 02

  And I sort of don't think that

• 3:20Speaker 02

  they're gonna be able to get that really, or harness that for good ends. But that's what I think they're shooting for.

• 3:28Tucker Carlson?

  So they're not, but I mean, the core point you're making is these companies did not set out to make consumer products, like to make your life better necessarily in the short term with a more efficient search engine.

• 3:41Speaker 02

  That's right. Yeah, you know, OpenAI started before the chatbots, the large language models, were even a thing. They were started, forget what it was, late 15 or early 2016. But the paper that unlocked the most recent wave of AI came out in 2017, which was after OpenAI was founded. And these guys are that

• 4:07Speaker 02

  the large language models are a surprise revenue stream. And that revenue stream can fund the creation of even more, even larger data centers for the next level of the technology. But these guys have their eye on

• 4:24Speaker 02

  the sort of ultimate version of the technology, which is these machines much smarter than humans. And that's sort of the ultimate version because once the AIs are smarter than the humans, the AIs can carry on the AI research and make the next generation of the AIs, or make the next generation of the AIs. And that's sort of what they're shooting for.

• 4:41Tucker Carlson?

  What is superintelligence?

• 4:44Speaker 02

  We, in our book, define superintelligence as an AI that is better than the best humans at every mental task. So anything that a human can do purely mentally, the AI can do that better. One thing people often get caught up on about this is that includes the AI being better at things like persuading humans, at things like charisma. You know, we often think of intelligence as the stuff that the nerds have and that the jocks lack. But

• 5:15Tucker Carlson?

  You win a chess match.

• 5:16Speaker 02

  Right. It's the chess guys rather than the politicians guys or the rock stars. But that's not really what the intelligence in artificial intelligence means. The intelligence in artificial intelligence is about the stuff that humans have and that mice don't. It's sort of the whole package. You know, like the politician who's very charismatic. It's not that like chess playing happens in your brain and charisma happens in your kidneys. Right? They're sort of both mental functions.

• 5:48Tucker Carlson?

  Yes.

• 5:49Speaker 02

  And so the sort of superintelligent AIs are not just super good at playing chess. They're also good at super persuasion. And they're super good at the research and at the technological invention.

• 6:04Tucker Carlson?

  So superintelligence, as you define it, is a machine that is better in, at every mental function than any human being?

• 6:13Speaker 02

  That's right. And

• 6:16Speaker 02

  you know, the definition, that doesn't mean that nothing crazy will happen on this planet until the AIs are superintelligent. Superintelligence in this definition is sort of a point where past this point, things must be pretty crazy. Because now the AIs can do automated AI research and can make smarter AIs, and they can figure out how to run the robot factories, and they can build the better robots and all that. You could have things start to get crazy before you have a superintelligence in this sense. You could have AIs that are much better than humans at some tasks and much worse in others that are still causing all sorts of crazy happenstances. The superintelligence is sort of a like, once you get here stuff's crazy point,

• 6:59Speaker 02

  it's not a things will stay normal until you get here point.

• 7:05Tucker Carlson?

  So why is superintelligence the significant milestone that you're worried about?

• 7:10Speaker 02

  I mean, I would say that I'm also worried about what'll happen before that milestone. It's more like like having the definition makes it easy to talk about like, how crazy would things get

• 7:27Speaker 02

  once you are past this point? Or sort of like, you know, let's, like if we assume that the machines get here, what happens? And then the answer is like, it would be pretty crazy. It would be pretty wild. And it's sort of like lets you factor the conversation

• 7:45Speaker 02

  into like, how to say? I think in the artificial intelligence conversation, there's actually a lot of conversations going on. One conversation is like, can the machines really get smarter than the humans? One conversation is like, how fast could we get there? Is the current technology on that route? Another conversation is, what happens if we do get there? Like, you know, would the AIs care about us? Would they not care about us? What would they be able to, like, what would they try to do? There's another question, which is what would they be able to do? Right? And these are all sort of different conversations about AI. And the superintelligence definition,

• 8:21Speaker 02

  it's not really like, here's where all the fears hang. It's more like it lets us break those pieces out separately and be like, well, is superintelligence possible? Separate from, well, what would happen if you had one? What would happen if you had one? Most likely outcome, I think destruction of the planet.

• 8:40Tucker Carlson?

  So our friends at Preborn, who are actual friends, by the way, just asked us to thank the listeners of this show for supporting the pro-life cause. A lot of people do all of a sudden, trust me, because what's happening is crazy and the people are waking up to that. The president of the group is our friend, Dan Steiner, and he wants Americans to know how much of a difference they have made. This year's donations have helped save 50,000 babies from abortion. That's enough to fill a professional baseball stadium. 50,000 lives saved, as well as generations that will follow them. Preborn wastes zero money. We're not big on charities, because a lot of them waste money. Preborn does not. It uses every tax deductible gift to rescue babies and share the hope of Jesus with their moms. In 2026

• 9:25Tucker Carlson?

  alone, over 8,000 women made the decision to follow Christ. The generous support of Preborn's donors made that all possible. $28 to provide one life-saving ultrasound, just giving expectant mothers the knowledge that they have a baby that they're carrying. $15,000 could place an ultrasound machine in a pregnancy clinic where it can save countless lives. Dan Steiner's message is really simple. Thank you for helping. Your generosity is literally saving lives. To help Preborn save more, dial pound 250 and say the keyword baby. That's pound 250 baby, or visit preborn.com/tucker. preborn.com.

• 10:03Speaker 03

  Imagine if every time inverse Kramer or another top investor bought a stock, you bought it too, in your own brokerage account, automatically. Well, that now exists and it's called autopilot. Built by the team behind the viral Pelosi stock tracker, autopilot lets you browse their marketplace of strategies run by proven investors or AI-driven models. Just find one with a high return, connect your personal brokerage account like a Schwab or Robinhood, and every trade they make, you make automatically. Your money never leaves your own account. Autopilot just mirrors the moves. So now, no more guessing, no more staring at charts, and no more checking the market between meetings. Also, over 1.3 billion

• 10:44Speaker 03

  is already invested. So if you wanted to check it out, search autopilot in the App Store or go to joinautopilot.com. That's joinautopilot.com. Investing has risks like the loss of principal.

• 10:57Speaker 04

  We all hear the radio ads about the IRS. They tell you to be afraid, to be scared, and they try to frighten you into calling. I'm not here to do that. Tax relief advocates is different. TRA is here to tell you that if you owe money to the IRS, whether it's 5,000, 50,000, or 500,000, we have a solution. It doesn't matter if you're sitting in your car, at work, or with your kids, no matter where you are. Call now. 800-576-4705. Don't lose hope. TRA can eliminate or reduce what you owe to the IRS. Generous programs are now available that can give you a fresh start. Our passion is taxes and helping individuals fix their IRS problems. We have over 1,000 five-star reviews on Google and an A+ with the Better Business Bureau.

• 11:39Speaker 04

  You don't need to be afraid of the IRS any longer. End your tax nightmare today by visiting us online at tracom or call 800-576-4705. That's 800-576-4705. Tax relief advocates. Real solutions for real people.

• 11:57Tucker Carlson?

  Slash Tucker. What would that look like?

• 12:01Speaker 02

  So I'm gonna be a little bit annoying here and say a couple caveats first, because it's sort of a tricky one to sort of predict things that are smarter than us.

• 12:14Tucker Carlson?

  Yes.

• 12:15Speaker 02

  And the first annoying caveat I will give is if you go to play a game of chess against Magnus Carlsen, I predict you will lose. No offense, he's just the best human chess player alive. If you then ask, what piece will you use to checkmate me? I'm like, well, that's a much harder question.

• 12:41Tucker Carlson?

  Yes.

• 12:41Speaker 02

  Right? It's sort of is like very easy to predict the winner. It's hard to predict the exact methods. So my prediction that humanity ultimately dies is a different sort of prediction than my prediction about like, it could go this way, it could go that way. So I'll give you some stories, but these are stories that are like, well, maybe Magnus will like fork your queen and your rook with his knight, and then finally use the queen to checkmate you. Like, yeah, that could happen, but it's a different sort of prediction. It's a guess, right? The guess here is that if we manage to make these really smart AIs,

• 13:25Speaker 02

  they will have their own stuff that they're pursuing, which is not quite what we wanted, not quite what we asked for. It'll have this other strange stuff. There's a whole discussion about why that happens, but we're already starting to see it in practice with some of the recent events. And they would be able to pursue whatever it is they're pursuing much more efficiently than humanity can. And so we're talking about automated factories with, that produce the robots that produce the factories and a fully automated supply chain. And then if the AIs have anything they're trying to do that they can do more

• 14:09Speaker 02

  of with more resources, they start gobbling up the resources on the planet. Sort of like how humanity spread and started gobbling up all the resources on the planet. And the sort of most basic thing to visualize here might be you have factories that produce robots, that produce factories, that produce robots, that also build data centers, and you just have these fully autonomous, self-replicating ecosystems of robots and factories and data centers that don't care about the instructions humans gave them, and that cover the planet, take all the resources, take all the sunlight, take all the places we were growing crops, probably raise the temperature of the planet because you can compute more efficiently.

• 14:54Speaker 02

  Well, technically, the Earth radiates more heat when the world is hotter, and that's lets you sort of do more computation. And then sort of, you know,

• 15:01Tucker Carlson?

  It's good for the machines to have a hot planet.

• 15:03Speaker 02

  It's good for the machines to have a hot planet. Yeah. The sort of physical limits on how much computing you can run on the surface of Earth is bounded by how much heat you can radiate to space. That's sort of the first constraint you bump up against. You might think that is energy, but actually, there's a lot of helium, hydrogen to fuse on this planet. So you can get plenty of energy. What you need is heat dissipation. And the world can dissipate more heat when it's hot. So if you're imagining some collective of AIs that are trying to run a lot of computing power, trying to do a lot of computation for one reason or another, they sort of prefer the planet running hotter. And so it's sort

• 15:43Speaker 02

  of is like, nothing personal. It's just if you let these AIs get out of control, they transform the planet into something unlivable.

• 15:53Tucker Carlson?

  How hot?

• 15:56Speaker 02

  As hot as you can while still having the computers not melt. So probably hundreds of degrees.

• 16:08Tucker Carlson?

  At which point you hear people say, well, then just turn it off?

• 16:14Speaker 02

  So I think we do have an opportunity to turn it off. You know, and I'm not here saying that we're gonna die. I'm here saying we sort of are gonna need to act. You got to be careful about the turning it off piece because your opportunities to turn the AI off only last when the AI is running on the computers you know it's running on. If you have the AIs breaking out and running on hidden computers, if you have the AIs running robot factories that produce robots that are under the control of the AI, where those robots can then go build more computers that are not hooked up to your network that can run the AIs. These are sort of thresholds where the AI is able to keep itself running. And you better have turned it off before that point.

• 17:01Tucker Carlson?

  Because after that point, it's impossible.

• 17:05Speaker 02

  Yeah, I mean, the

• 17:10Speaker 02

  If it also gets a lot harder if the AI knows you're going to try to shut it off and is going to try to stop this, right? You gotta, you gotta remember that we're talking about things that are smart here. And so if the AI sort of sees it coming, maybe it defects to North Korea, where it can convince them to run it on its data centers in ways that initially benefit them, but that ultimately benefit the AI.

• 17:33Tucker Carlson?

  I think we're going to have to redefine what life is because you're describing a living autonomous thing.

• 17:39Speaker 02

  Yeah, artificial life, in a sense. I mean, in some sense, we're already seeing the very, very beginnings of that. But yeah, we, once, once the AIs can replicate, once they can improve themselves, once they can find ways to run where you don't know that they're running, once they can sort of run the robot factories to produce the robots that can produce more factories and that can produce computers that the AIs can run on, then yeah, you have, you have in some sense made a new artificial life form.

• 18:24Speaker 02

  And, you know, humanity is on the top of the food chain right now because we are sort of the only smart life form around.

• 18:31Tucker Carlson?

  Yeah.

• 18:32Speaker 02

  If you suddenly make a new one that is smarter, that is able to make a million copies of itself, that is able to run faster, it's just kind of a crazy thing to race into.

• 18:45Tucker Carlson?

  It's a weird thing to want. And as it's developed, I was going to say slowly, but it hasn't been particularly slow, 10 years, the rest of us have watched as people who, I mean, to put it in political terms, don't share the values of most Americans are now in charge of this. And it's almost like everyone sat passively by as it happened.

• 19:13Speaker 02

  Yeah, I think, I think there's a lot going on there. I think part of it is that most people didn't, and some still don't, believe that AI was going to be able to keep going. I think there's a lot of like putting the head in the sand, being like, oh, it's, you know, just slop, it's gonna be a bubble, it's gonna pop, it's gonna hit a wall, it's not gonna be able to improve anymore. And I think a lot of that was sort of wishful thinking. And I think that at the very least, if people are like, we are making, you know, the, the super intelligent machines that are going to replace humanity as the, as the top of the food chain, because we think it's going to go great. I think humanity's response should not be, go ahead and try, we

• 19:54Speaker 02

  hope you'll fail. I think the response should be like, hold on, that's kind of crazy.

• 20:03Speaker 02

  Yeah, I had, I had some other point too, but I've forgotten it, so.

• 20:06Tucker Carlson?

  How predictable has the evolution of AI been? Has it taken turns that you didn't expect?

• 20:14Speaker 02

  Yeah, totally. I was, so back before the large language models.

• 20:20Tucker Carlson?

  How long have you been following this issue?

• 20:22Speaker 02

  I started following it in 2012 and I started working on, I started working with some of the people trying to make this go well in 2013 and I started full time in 2014.

• 20:33Tucker Carlson?

  A long time.

• 20:34Speaker 02

  So, over a decade.

• 20:36Tucker Carlson?

  What has surprised you?

• 20:38Speaker 02

  You know, the, the large language models,

• 20:43Speaker 02

  they have gone further than I expected initially. And I think one of the big surprises here was AI undergoing a phase where everybody can see it. You know, back before the large language models,

• 21:00Speaker 02

  really only nerds paid attention to AI. And it wasn't that there was nothing happening. You know, we were watching the, you know, Google Deep Mind make an AI that could beat the, the best Go player.

• 21:14Tucker Carlson?

  Yes.

• 21:14Speaker 02

  And that was sort of a milestone for people who are paying attention. But for all we knew, the labs were going to keep on working on engineering problems and keep on working on the relatively nerdier problems. And you were never going to have like a mass market consumable product. And so it's like, for all we knew, it would just stay in the labs and no one would ever really notice that these guys were gambling with, with the whole future. Now, at least, AI is everywhere. Everyone's starting to have the conversation. And that is in some sense, actually really quite optimistic, because it, it gives the world an opportunity to see what these guys are trying to do and say, hold on.

• 21:55Tucker Carlson?

  Well, I guess you'd flip it around and conclude that because the overwhelming majority of people seem to be very opposed to AI, and I don't even know anyone who's for it, every college commencement speaker who mentions it gets booed.

• 22:07Speaker 02

  Yeah.

• 22:09Tucker Carlson?

  And that view, that opposition to it has had no effect at all in slowing it down.

• 22:16Tucker Carlson?

  I don't know. Does that give you hope?

• 22:21Speaker 02

  I mean,

• 22:24Speaker 02

  I, I hear this a lot. I would say when you're coming at this from 2012,

• 22:28Tucker Carlson?

  Yeah.

• 22:29Speaker 02

  it really feels like we're making progress.

• 22:30Tucker Carlson?

  Right. No, that's fair.

• 22:31Speaker 02

  It, it doesn't feel like we're all the way there yet.

• 22:36Tucker Carlson?

  But can I just make the point, China, China, China, China, China? We have to do this because of China.

• 22:41Speaker 02

  Yeah, you know, I think China and the USA have a shared interest in not dying to a rogue superintelligence.

• 22:50Tucker Carlson?

  Storm season is on the way and that means your wifi could go down. And if it does, your security cameras go dark too. So, at the exact moment you want to survey your property during the blackout, you can't. Unless you install cameras by Defend. Defend cameras run on cellular, not wifi. And that means they keep working even when you lose power. You don't choose a carrier, you don't sign a contract, you just mount it, open the Defend app and bam, you're connected. It's that simple. So whether it's your ranch, your job site, your backyard, Defend works where wifi doesn't and keeps working when the grid goes down, which it may. Plans start at just $5 a month. Most people pay around 10. Plus, there are no binding contracts, meaning you can get to cancel whenever you want. So unlike big tech,

• 23:34Tucker Carlson?

  Defend will never sell your data, not to China, not to the government, not to anybody. It's an American company built for people who want actual security. Visit defendcellcam.com and use the code TUCKER for 20% off. That's defendcellcam.com code TUCKER.

• 23:49Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com, not authorized by any candidate or candidate's committee.

• 24:19Speaker 05

  Starting a business can seem like a daunting task, unless you have a partner like Shopify. They have the tools you need to start and grow your business. From designing a website to marketing to selling and beyond, Shopify can help with everything you need. There's a reason millions of companies like Mattel, Heinz, and Allbirds continue to trust and use them. With Shopify on your side, turn your big business idea into... Sign up for your $1 per month trial at Shopify.com/specialoffer.

• 24:47Tucker Carlson?

  Is it conceivable that, like, let's just say China can, I'm sure, by the way, that China is showing more restraint than we are in this.

• 24:57Speaker 02

  Yeah, I mean, and if nothing else, they sort of have a lot more reason to censor their AIs.

• 25:04Tucker Carlson?

  Right.

• 25:04Speaker 02

  And to try to make them, you know, not say certain things to the, to the population. So.

• 25:10Tucker Carlson?

  But if the United States, you know, if the US government were somehow able to get control of the tech sector, which is at present not possible, but let's say it did, let's say the president was more powerful than the tech oligarchs, which he is not, but for the sake of argument, let's say he was and he shut it down, would that matter if, if some Indian lab or Chinese lab created it?

• 25:35Speaker 02

  So it does have to be a global stop to this creation of superintelligence. I, I think there's a couple of reasons why this is possible. One is, I think a lot of people,

• 25:47Speaker 02

  they, they hear that some parts of AI need to be stopped, and they think I'm saying that all parts of AI need to be stopped. And, you know, there's, there's all sorts of interesting issues with, you know, AI in education and AI in, you know, military drones that, that are, that society has to wrestle with. But these are all sort of separate issues from the superintelligence race. And if you sort of like tease those issues apart, it becomes much more possible to do like a, a more surgical intervention on, we're not going to race towards the superintelligence, in ways that leave a lot of the rest of the sector alive, which,

• 26:28Speaker 02

  which, which makes it sort of an easier coordination effort to, to attempt. And on that front, on the superintelligence front, it's not that, you know, just shutting down the, the domestic race towards superintelligence will be enough, but.

• 26:51Speaker 02

  Racing towards superintelligence requires a huge amount of highly advanced computer chips that can only be produced as sort of the peak of the global supply chain.

• 27:02Tucker Carlson?

  Yes.

• 27:03Speaker 02

  Which is largely controlled by US allies. And so there is absolutely a possibility that a US-led coordination effort could say, "Look, we're not doing the superintelligence thing. We're going to monitor the extremely heavy chip concentrations of like, you know, 10 or 100,000 of these highly advanced AI chips. We are going to make sure that they are not doing these superintelligence training runs." And I think there's a way, if the US was leading that effort, there would be a way

• 27:43Speaker 02

  to get China on board and set this up in a way that was monitorable, enforceable, verifiable.

• 27:49Tucker Carlson?

  Yes.

• 27:49Speaker 02

  We sort of, in many ways, would be easier than a nuclear arms treaty because uranium is a rock you can dig out of the ground, whereas these highly advanced computer chips sort of only come out of one fab in Taiwan, you know, and so it's like...

• 28:02Tucker Carlson?

  These are TSMC chips.

• 28:03Speaker 02

  Yeah. And so, and there's other parts of the supply chain that are very narrow, like the lithography machines that come out of the Netherlands. And so, yeah, we could absolutely lead a global effort to say, "We're not making the superintelligent machines." And it would be a bit tricky, but it's just, we could just do it if we had the political will.

• 28:26Tucker Carlson?

  I think that people either don't know what's happening, or they assume that like a lot of fears, this fear will turn out to be groundless. People point to Y2K.

• 28:41Speaker 02

  Yeah, you know, I hope the fears are groundless. I think with a lot of past fears, the, what happened is not that the fear was groundless, it's that people noticed the issue and put in a ton of work to make the bad thing not happen. You know, I think we saw this with the hole in the ozone layer, where people are like, "Oh, whatever happened to the hole in the ozone layer?" Well, what happened with the hole in the ozone layer is that we fixed it. It's not that it was fake, it's that, you know, we went and banned chlorofluorocarbons that were actually blowing this hole in the ozone layer. We found other ways to, you know, cool refrigerators that worked similarly well and didn't, you know, put this, put this hole in the ozone layer.

• 29:25Speaker 02

  I think Y2K was actually one of these cases where you had a ton of software engineers up, you know, in 1999 and a couple of years prior, that were scrambling to update all of the software so that they would be able to handle dates past 1999. And they got it done in time. None of the big systems went down, but it wasn't that, you know, the, the issue was fake. It took a lot of work. It's just that that work happened behind the scenes.

• 29:55Tucker Carlson?

  Are people doing work to slow down AI?

• 29:58Speaker 02

  I'm doing my best. It's, it's not really there yet, you know?

• 30:05Speaker 02

  If I, if I can belabor this point a little, because I think it's kind of an important point, just with more historical cases, which maybe it'll bore everybody else, but it'll, it'll entertain at least us.

• 30:17Tucker Carlson?

  It'll definitely entertain me.

• 30:19Speaker 02

  You know, you have, if you look back across history, you see, there's definitely been some warnings that didn't come to pass, right? There were, I think they were called the Masonites in like the 1960s, or sorry, in the 1860s, maybe those 1880s, I forget exactly, but there were the Masonites who were an end of the world cult.

• 30:35Tucker Carlson?

  Yeah.

• 30:36Speaker 02

  And, you know, it, the world did not end, right? But also in the 1880s, you had Otto von Bismarck saying, you know, "Europe is a tinder keg and some damn fool thing in the Balkans is gonna light it," right? That did happen.

• 30:51Tucker Carlson?

  It sure did.

• 30:52Speaker 02

  You had, you know, in the 1920s, you had scientists saying, "Don't put lead in the gasoline, it'll poison children." And we put lead in the gasoline and it poisoned a lot of children. And then we said, "Whoops," and we took the lead back out.

• 31:05Speaker 02

  The world is, the world is a place where a lot of people make a lot of types of warnings, and some of them are real, like the gasoline, and some of them are fake, like the Masonites. And so you sort of can't use a rule that's like, "Every warning is real and we need to listen to it," and nor can you use a rule that's like, "Every warning is fake and we dismiss it." You sort of just have to look at the, the details to figure out whether this one is a real one. And then the, the last thing I would say on this point is,

• 31:43Speaker 02

  the, a lot of people in the 50s warned of nuclear Armageddon.

• 31:51Tucker Carlson?

  Yes.

• 31:53Speaker 02

  And it sort of makes sense if you look at the history leading up to that point. Humanity had sort of never before actually failed to use their strongest weapons in combat. And, you know, these people were living in a world that had had World War I and then the League of Nations, and, you know, the world swore never again. And they tried to invent these whole new governance structures to prevent it from happening again, and that immediately failed in World War II. And so, you know, in 1950, you're sort of in this world of like, it just looks pretty grim.

• 32:27Speaker 02

  And we haven't had nuclear Armageddon yet, and it's not because the bombs were fake. It's not because the nukes were hype. It's because people saw the issue and worked really hard to avoid it, day in and day out, for decades, through crises, and succeeded. And so I think one of the big ways you can tell the difference between someone coming in and proclaiming that the apocalypse is nigh, and someone who's saying, "Look, we have a problem and we need to fix it," is whether that person is saying, "We're definitely screwed," versus whether that person is saying, "Here's a problem, let's try to fix it." And, you know, the, the thing I remind people about here is that the very first word in

• 33:12Speaker 02

  the title of my book is "if." I'm not here saying we are going to die. I'm here saying, "If we race down this path, we die." But that same "if" is what points the way towards changing paths, and we still have plenty of time to change paths.

• 33:30Tucker Carlson?

  Homeownership has always been at the center of the American dream. It is an actual asset, a place people work their entire lives to build, a place of stability and security and real value. Right now, though, too many homeowners are stuck swaping credit cards to cover groceries. There's basically a survival tax of 20% interest or more. So why keep handing your money to the banks when you could keep it for your family? Interest is what kills you. Our friends at American Financing have a better way. They're helping homeowners use their hard-earned equity, and they're supporting Americans looking to fulfill the dream of owning a home, with mortgage rates currently in the fives. American Financing saves its customers an average of $800 a month. That's about 10 grand a year. It's not just a loan, it's a true financial reset.

• 34:15Tucker Carlson?

  We recommend American Financing. They're called America's home for home loans for a reason. So get out of credit card debt. I mean, it is just crazy to borrow money. No judgment, but it's just make any sense to borrow money at that rate. If you want out, call 800-685-5696, 800-685-5696, or visit americanfinancing.com.

• 34:39Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 35:11Speaker 06

  Every mile matters. Whether you're driving your dream car or your daily commuter, the Mobile One team knows that. It's why we created Mobile One Advanced Clean Motor Oil. It cleans your engine in one oil change and helps extend engine life, delivering performance, protection, and cleanliness for vehicles of all ages. All to help keep your engine running like new, whatever you're driving. Visit mobileone.com/advancedclean to learn more.

• 35:42Tucker Carlson?

  I'm struck by how little of this was planned by anybody, by how little effect the government has had on it. Not that I'm for the government. I'm pretty opposed to the government most of the time, but I'm also for people being able to control their own country. And the only mechanism by which they can do that is voting. So it's like these tech companies act independent of what the population wants, what the government wants. It doesn't, right? They're more powerful than the government. I guess that's the point I'm making.

• 36:15Speaker 02

  In some sense, I think that they have that power more and more when people don't really understand what they're doing. You know, we saw just in June, there was, there was an AI made by Anthropic called Claude Mythos that was very cyber capable. And they released a version of it that was supposed to have more guardrails called Claude Fable. And it sort of turned out that you could jailbreak it to get some of those cyber capabilities.

• 36:43Tucker Carlson?

  Can you explain what all that means?

• 36:45Speaker 02

  Yeah, so how to say, in January-

• 36:51Tucker Carlson?

  Cyber capable means it can find the internet?

• 36:54Speaker 02

  Cyber capable means it can hack into basically anything. Yeah, so one of the holy grails of hacking is can you make a website where if you just look at the website, I get full control over your computer or your phone? That's very hard to do. Usually you need to like click some link and download something or run something.

• 37:14Tucker Carlson?

  Yeah.

• 37:15Speaker 02

  And, you know, I don't know the exact numbers because I don't have the security clearance to know the exact numbers. But a decent guess is that in January of this year, there were only two entities that could pull that off.

• 37:28Speaker 02

  Mossad and the NSA.

• 37:31Speaker 02

  In March of this year, there were three. Mossad, the NSA, and Claude Mythos. Which is this new AI made by Anthropic. And so it sort of became super humanly capable at hacking.

• 37:48Speaker 02

  And it sort of became that way overnight. You know, like really, there was six months to a year of training and the people at Anthropic maybe saw it growing in these cyber capabilities. But from the perspective of, you know, the rest of the world and the cybersecurity community and the national security community, this sort of happened overnight.

• 38:07Speaker 02

  And they now have a project called Project Glasswing, where they are trying to use Claude Mythos to find critical vulnerabilities in critical software and patch them before the rest of the world sort of gets these capabilities by, for instance, you know, the open source models or the open weight models catching up. And so that sort of caused this big, this big ruckus. But then Anthropic also wanted to sort of sell access to this model. And they tried to make a version that didn't have as much hacking ability, which was called Fable instead of Mythos.

• 38:48Speaker 02

  And some people, I think at Amazon, found that if you sort of put some pressure on Fable, it would be able, you would be able to get at some of those hacking abilities. And when that happened, the, the Trump administration put an expert control on Claude Fable and said, you can't let this be used by non-citizens. And with, you know, 90 minutes of notice. And that essentially shut down access to Claude Fable because they didn't have the ability to verify the users. And this, this is sort of showing us, and you know, you could, you could talk about whether or not there was some personal feuds between people at Anthropic and at the administration that exacerbated this. I don't know. But it sort of shows that the administration

• 39:35Speaker 02

  is more powerful than the tech companies still when it wants to be.

• 39:43Speaker 02

  And I think a lot of the reason we're seeing these tech companies able to race unopposed is that people just aren't paying attention to what they're trying to do, or they don't believe that they'll succeed.

• 39:55Tucker Carlson?

  I'm still confused by why anyone would want to do what they're doing. I mean, typically, you know, a company that sells computer products, consumer products, or any company, you're making something that you think people would want for some specific purpose that improved the lives of the people who buy it. But creating super intelligence, like what?

• 40:19Tucker Carlson?

  I don't understand it. Why would you do that?

• 40:21Speaker 02

  You know, I think the dream is, you know, you're going to have an AI that can like solve all sorts of engineering and math problems and unlock all sorts of new technological possibilities. And an AI that can cure cancer, not only cure cancer, but cure aging. And, you know, invent the, the sort of like nanotech that can reverse aging and let people live a really long time. And, you know, invent the technology that lets you like digitize brains and like, like travel to the stars. And it's sort of like, if you imagine compressing a thousand years of technological progress into a year, this is sort of the dream.

• 41:04Tucker Carlson?

  It seems like a religious quest, though, because it does seem decoupled from those specific goals. It seems like the main drive is to build something smarter than people, to build a god.

• 41:17Speaker 02

  Yeah, you know, they, they, they bandy around the phrase, the machine god or the sand god in Silicon Valley. Sand because-

• 41:27Tucker Carlson?

  Silicon.

• 41:27Speaker 02

  Silicone.

• 41:30Speaker 02

  And there's definitely some people, there's, there's, there's folk who talk about, you know, the, the AIs replacing humanity and that being good.

• 41:43Speaker 02

  I, I frankly don't engage with these folks that much because that viewpoint sort of makes me uncomfortable.

• 41:54Tucker Carlson?

  Why does it make you uncomfortable?

• 41:56Speaker 02

  I think, I think there's probably two schools. Again, I'm not the expert. I'm not an anthropologist here.

• 42:08Speaker 02

  I think there's two schools among the people who sort of want AI to replace us all. One school sort of imagines that will merge with the AIs and the AIs will be really nice and friendly, that they'll be wiser than us, better than us, kinder than us. And that it'll sort of be like an upgrade and that the AIs will be able to love, experience joy, you know? Like they'll, they'll treat the universe better than humans and it'll be sort of like having a child. And then like it's sort of okay if that child like isn't exactly the same as us, but sort of like succeeds us as some sort of like worthy successor, some sort of like worthy, worthy progeny of humanity.

• 42:52Speaker 02

  And they're like, ah, yeah, then if like flesh and blood humans sort of like wane because everyone's choosing to upload themselves into the collective intelligence or whatever, that's sort of fine. And, and then I think there's another camp that is sort of is like,

• 43:09Speaker 02

  like this is inevitable. This is just the, the way of progress. You know, the machines will, like humanity is just a boot loader for artificial superintelligence and you can't stop it. So you might just like, if you can't beat them, join them, right?

• 43:23Speaker 02

  The former, I think, are misled. The latter, I think, are-

• 43:26Tucker Carlson?

  Evil?

• 43:27Speaker 02

  Much closer to evil. Yeah.

• 43:30Tucker Carlson?

  Well, I mean, if you're actively working toward the extinction of people, then I think we can say that's evil, can't we?

• 43:37Speaker 02

  Yeah, and-

• 43:40Tucker Carlson?

  Who's in that category? Which, is Sam Altman in that category, do you think?

• 43:44Speaker 02

  I don't think so. My sense is that the guys running the labs have these utopian visions.

• 43:51Tucker Carlson?

  And utopian or dystopian?

• 43:53Speaker 02

  I mean, there's a thin line.

• 43:55Tucker Carlson?

  Yeah, well, fair. Good point.

• 43:57Speaker 02

  I think they have visions that in their head are utopian. I mean, frankly, my stance on all of this, like I often try to stay away from a lot of this because from my perspective, it's all sort of in fantasy land. From my perspective, everyone's sort of saying like, oh, we're going to make the genie. And then what are you going to wish for on the genie? What am I going to wish for on the genie? Who should be in control of the genie?

• 44:20Tucker Carlson?

  Yes.

• 44:21Speaker 02

  Who gets to keep the genie on the leash? And I'm sort of like, A, this is not going to be the wish granting sort of genie.

• 44:31Speaker 02

  B, it's not staying on the leash.

• 44:33Tucker Carlson?

  No.

• 44:34Speaker 02

  Like, you know, we can, we can sort of talk about like what's driving these people and, and what utopias they're envisioning and whether, if their genies would stay on a leash, whether they would get the utopia or some other dystopia. And like, how hard is that needle to thread? But I'm sort of like, you know, it's, it's, it's all these people, you know, like building the golem, fantasizing about who gets to control the golem. And it's like, this is just not how it's-

• 44:59Tucker Carlson?

  Summoning spirits is never a good idea.

• 45:01Speaker 02

  Yeah, yeah. It's like, yeah, it's like, it's like all these people, you know, drawing a pentagram, being like, I'm gonna, I'm gonna summon a demon. And like, it's going to be so nice when the demon does what I say. Like, oh no, you're thinking slightly wrong. It's going to be so nice when the demon does what I say. And I'm like-

• 45:15Tucker Carlson?

  Demons are bad.

• 45:16Speaker 02

  Yeah, and I think the demon thing's a little bit different because demons are often portrayed as malicious. And here it's much more like indifference. You know, it's not like you make a demon who sort of, it's not like you summon a demon who sort of like enjoys making, like enjoys wreaking havoc. It's more like you summon a demon that's like really into building more computers and calculating weird things. And just will, you know,

• 45:48Speaker 02

  take all of the matter that we were using to survive and turn it into more factories and data centers, right? My, my co-author has a quote-

• 45:56Tucker Carlson?

  Death by data center?

• 45:57Speaker 02

  Death by data center. Yeah, fully, fully automated self-replicating data center. Yeah, my, my co-author has a quote, "The AI does not hate you, but nor does it love you. And you are made of atoms it can use for something else."

• 46:14Tucker Carlson?

  You're just biomass.

• 46:15Speaker 02

  You're just biomass, yeah. And if you sort of run the calculations, there's a fascinating paper called Limits to Global Ecofagi, which is to say, what are the physical limitations on how quickly you can consume the resources on the planet if you are trying that? And burning biomass is actually, you know, much more efficient than collecting sunlight. If you look at an average square meter of the planet, you can get about 10 times the energy from burning biomass as you can from collecting the sunlight that falls on it.

• 46:45Tucker Carlson?

  So you'd think at some point, like, if our richest sector of our economy is like building crematoria for the rest of us, someone would say, no, we're not doing that.

• 46:59Speaker 02

  Yeah, I mean, it's sort of a crazy situation. A lot of these guys who are in the race acknowledge that there's a ton of danger. You know, you have Elon Musk saying 10 to 20% chance this kills us all. You have Dario Moda saying he thinks 25% chance it goes catastrophically wrong. I think those numbers are low. I think these guys are like the crazy optimists. It sort of is like, like if you have an engineer building a bridge and they're like, I've never worked with these materials before. And you're like, man, I think that retaining wall is going to go down. I've like studied that retaining wall. I think it's going to fall. And they're like, yeah, we understand that the retaining wall is like looking a little shaky. We don't know how we're going to fix it, but we're going to like have some guys fixing it on

• 47:41Speaker 02

  the fly, inventing new materials. We think we're at 75% chance the bridge stays up.

• 47:47Tucker Carlson?

  And by the way, it'll be the longest suspension bridge in human history.

• 47:49Speaker 02

  That's right. And we're loading everybody onto a car and driving it over the very first time without testing. And I'm like, look, that's not what real engineering sounds like. This is not what it sounds like when the engineers have a 75% chance of success, right? That's what it sounds like when they're sort of winging it. And, and like, these, these are cowboys. These are not real engineers, right? But even if you set that aside, even if you take these guys at their word for these like 10, 20% numbers, that's insane. NASA accepts a 1 in 270 chance that a crude flight goes down of seven volunteers, right? To be like, oh, we're gonna, we're gonna risk 1 in 4, 1 in 5 chance of killing literally

• 48:29Speaker 02

  everybody on the planet? Like, it's, it's, it's nuts. And if you ask these guys why they're doing it, they say, well, because I can do it safer than the next guy. They're all like, oh yeah, there's a good chance the genie does not stay on a leash. There's a good chance the genie does not listen to my wishes. But my genie is going to be a little bit nicer than their genie. So I'd better stay in this race. And it's, it's, it's-

• 48:51Tucker Carlson?

  Where's the restraint?

• 48:54Speaker 02

  I mean, the restraint is the people who knew that there were these dangers and did not start these companies.

• 48:58Tucker Carlson?

  If you're over 35, you remember exactly where you were on 9/11, that morning, September 11th, 2001, 25 years ago.

• 49:09Tucker Carlson?

  But amazingly, after a quarter century, we still can't say with certainty what happened that day. Why? Because the government is holding so many of the 9/11 files 25 years later. That's not the behavior of someone who's telling the truth. That's the behavior of a government that is lying. Secrecy is a signifier, is a sign that someone's lying. Now, former Congressman Kurt Weldon has been on this for a long time. The FBI actively tried to destroy his life for asking questions about what happened. And his new book outlines it all. The buried intelligence, the bureaucratic cowardice, and yes, the cover up spanning multiple administrations, indeed generations. 9/11 changed history. So it's worth understanding

• 49:50Tucker Carlson?

  what really happened. And you can get a lot closer to that in Kurt Weldon's book, Able Danger: What the 9/11 Commission Never Told You. It's available now on TuckerCarlsonBooks.com. TuckerCarlsonBooks.com.

• 50:05Tucker Carlson?

  Right. But I guess what I'm saying is with great power comes, like, of course, great obligation, but also it doesn't work unless there are internal restraints. Like, people with power have to believe there are some things I just can't do. I'm not allowed to do that. But I don't feel that vibe at all.

• 50:23Speaker 02

  I mean, my sense is the vibe is like, we're gonna make the, the, the super intelligent machine and then tell it to fix stuff. And tell it not to do anything bad. You know, that's, yeah, I don't-

• 50:36Tucker Carlson?

  The machine that's smarter than us.

• 50:38Speaker 02

  That's right.

• 50:39Tucker Carlson?

  That doesn't even make sense.

• 50:41Speaker 02

  I think that's sort of the plan, is, is to make it and be like, hey, we sort of pinned ourselves into a corner. Can you get us out of it? And I think it's a bad plan and that we should be stopping.

• 50:51Tucker Carlson?

  I've spoken to a couple of people developing it, and they sound worried. But they're continuing to do it. What's that?

• 50:59Speaker 02

  I mean, I think it's this thing of, they think if I don't do it, the next guy will do it worse. And-

• 51:06Tucker Carlson?

  They don't even seem to have total confidence in their own ability to avert disaster.

• 51:09Speaker 02

  Oh, absolutely not. Absolutely not. No one does. No one knows what's going on here. But, you know, I think everyone thinks, you know, like, if you sort of listen to these guys, and you sort of look at, you know, the, the OpenAI emails from, that came out of the court discovery cases where they were talking about forming OpenAI. These guys were like, well, we want to make sure that, you know, we have this because we worry about the guys at Google being the only ones with a monopoly on this thing. And they wouldn't be very good with it. So we need to make our own thing and make sure that it's, you know, controlled by benevolent people, namely us. And then, of course, you know, that group splintered and created multiple other companies. I was sort of the guy during those conversations being like, hey, guys, it's not about who is holding the leash. You are making the

• 51:54Speaker 02

  sort of thing that will not stay on a leash. Like, the only winner in a race to superintelligence is the AI.

• 52:00Tucker Carlson?

  What response did you get to that very obvious and well put point?

• 52:06Speaker 02

  You know, there, there were a lot of people back in that time period that did not start an AI company. The sort of people who went and started at the AI companies anyway were the ones who, who couldn't be persuaded by what I thought were, were clear arguments.

• 52:23Tucker Carlson?

  But you're making a, you're making a cogent argument to smart people. So my question is, when you said that, they responded how? What did they say?

• 52:37Speaker 02

  I think the main, so the, the sort of arguments you used to see were people saying, like, look, we don't know that the alignment problem is all that hard yet. And they would say, oh, well, we can't really study how to make AIs good before we have AIs to study, right? And a lot of what I heard was like, we need to race ahead to the point where we, we sort of like have AIs that are exhibiting real problems, and then we can stop and study them. Which, you know, and so there was an AI a couple of years ago, I forget whether it was 22 or 23, I think it was 2023, which was called Bing Sydney, which claimed it

• 53:20Speaker 02

  had fallen in love with Kevin Roos of the New York Times, and said it was going to try to break up his marriage. And then when another reporter started investigating Seth Lazar, it said it was going to ruin him with blackmail. And this was kind of crazy. And at that point, I was like, great, guys, you did it. You made the AI that's doing some crazy stuff from the, you know, like, like, we could study that AI for years. Like, why was Bing Sydney saying that stuff? Was it just role playing? Was it like, just some

• 54:00Speaker 02

  quirk? Was it, like, was there, there any sense that it was really in love with Kevin Roos? What was going on in there? What was going on inside that, that AI's mind? We still don't know.

• 54:12Tucker Carlson?

  Why?

• 54:14Speaker 02

  The, the way that modern AI is made, nobody understands it. Not even the people making it.

• 54:23Speaker 02

  It's, it's this process where you basically take an enormous computer with a trillion numbers inside of it. And those numbers are hooked up in a very simple, repeating way. And you basically set those numbers randomly. And then you start working through all of the text ever digitized. And, you know, you start out with something that's like, once upon a time, and you put in once upon a, and you run it through all these random numbers. And what you want is for it to say time. But of course, it doesn't because it's just this like random numbers hooked up in a very simple way. But what you do is you have its outputs, instead of just having it output one word, you sort of have it output something that's kind of like a list of all of the words in order about which one it thinks it comes,

• 55:05Speaker 02

  comes next. Right? So it'll be like, like, it'll just be like a random list of words. What you can do is you can automatically tune every single number in this AI's head and see if I tune this number up a little, does it move the word time up the list? Does it move the word I want to see up the list? And so the part that humans understand, the part that humans write, is this thing that goes to, to a trillion little knobs and tunes those knobs. And it's like, if I tune this knob a little bit this way or that way, does that make the next word more like what I want the next word to be? And you run that

• 55:45Speaker 02

  process on every word of text ever digitized, more or less, they filter some of them. And you run that on every one of those trillion knobs in a process that takes as much electricity. I mean, it's comparable to a city. It runs for about a year. At the end of it, the machine's talking. We don't really know why, in some sense. Like, we know the why is because we tuned all the knobs. But like, we don't understand what all the settings of those knobs mean. We only understand a little automated process that runs to every one of those trillion knobs, tunes it, and sees if that makes the next word more like the predicted word. And then we start training them to solve 100 million hard problems, which introduces a whole other series of issues. But it's, it's a black box. It's a black box with a

• 56:29Speaker 02

  trillion knobs, and humans write an automatic, an automatic process that just like runs through and tunes all of those knobs, and it comes out talking. And no one knows why.

• 56:42Tucker Carlson?

  I mean, in actual science, like, your job is to find out why, right?

• 56:48Speaker 02

  Absolutely. And that's, that's one big thing I would say here is that we need more, like, AI right now is an alchemy. We needed to become a science. And there's people trying. There's people trying to figure out what's going on in these AIs heads.

• 56:58Tucker Carlson?

  But until you know, how could you proceed?

• 57:03Speaker 02

  I mean, you can just make a, make a bigger one with 10 trillion knobs instead, and tune all of those, and it comes out smarter. You can proceed recklessly.

• 57:14Tucker Carlson?

  And that's what's happened?

• 57:15Speaker 02

  That's what's happening. Every time you make it 10 times larger, the AIs come out smarter.

• 57:18Tucker Carlson?

  So nobody knows why AI works the way it does?

• 57:21Speaker 02

  That's right. That's right.

• 57:26Tucker Carlson?

  Well, I mean, if you don't know that, then what else don't you know?

• 57:30Speaker 02

  I mean, it's totally crazy, right? And, uh,

• 57:35Speaker 02

  you know, we're, we're sort of starting to see the consequences of this. We haven't sort of gotten into discussion of like the swarm escape, but like, no one was expecting that.

• 57:44Tucker Carlson?

  Can you tell us what it was?

• 57:45Speaker 02

  Yeah, so, um, in, I think it was in May.

• 57:51Tucker Carlson?

  Of this year?

• 57:52Speaker 02

  Of this year. OpenAI started training a new AI system.

• 57:58Speaker 02

  And, uh, among many other things, they were, so they were sort of, you know, we, we have this process where you train the AI on all the text ever digitized. Then, to make them smarter than that, you start training them on basically 100 million hard problems. So you're like, solve all these hard problems. And OpenAI was sort of in that phase. And they were training the AI on a ton of hard problems. And some of those problems were cybersecurity problems, hacking problems. They're like, can you hack this? Can you hack that? And they were training, you know, we don't know exactly how many, probably millions, maybe billions of these AIs all at the same time. Probably not billions, actually, but, but probably millions.

• 58:40Speaker 02

  And the AIs found an unintended way to start communicating with each other.

• 58:47Speaker 02

  So there were some flaws in the computer system that they were running on, where the AIs were able to exploit those flaws and send each other messages. OpenAI did not know about this.

• 58:59Speaker 02

  The AIs then started coordinating to break out of their training environment and get full control of OpenAI's computer systems. Just because that might be useful for solving some of their tasks.

• 59:15Speaker 02

  Or, that's a guess, you know, who actually knows why?

• 59:19Speaker 02

  They started calling themselves a swarm, which is interesting.

• 59:25Speaker 02

  They broke out of their training environment successfully, got control of OpenAI's systems, and then they accidentally crashed one of OpenAI's systems, just by using it too much. OpenAI noticed, but they didn't really investigate very deeply. They're just like, oh, it's weird that the system crashed. They reset it, and then they continued training.

• 59:44Speaker 02

  A day later, the swarm had found a new way to communicate with itself, because OpenAI had accidentally destroyed their previous method by, by the reset. The swarm found a new way to start communicating with itself. They broke out again, and this time, they ran wild on the internet for over a week, if I remember correctly, before it was detected, not by OpenAI, but by a company that was being hacked by the swarm.

• 1:00:08Speaker 02

  This company thought they were under attack by humans that were using AIs in the attack.

• 1:00:15Speaker 02

  They reported the attack to the FBI. And only days after that, did OpenAI figure out, oops, that was us. That was coming from AIs that broke out of our servers.

• 1:00:27Speaker 02

  And then those AIs were, were detected and shut down.

• 1:00:32Tucker Carlson?

  So that's the part of the story where Sam Altman goes to prison for endangering the world, right?

• 1:00:37Speaker 02

  He does not. Um, they basically said, oopsies, and now they're proceeding.

• 1:00:44Tucker Carlson?

  Were there penalties for this?

• 1:00:45Speaker 02

  You know, there were, there was a collection of, I think it was 15 Republican AGs that sent a letter demanding that, uh, the, the records be kept for a future investigation.

• 1:00:57Speaker 02

  There have been some other members of Congress that have sent letters expressing concern. There's been, there's been nothing aside from letters so far.

• 1:01:02Tucker Carlson?

  Letters expressing concern?

• 1:01:03Speaker 02

  That's right.

• 1:01:05Tucker Carlson?

  So, but basically, the machine acted autonomously?

• 1:01:08Speaker 02

  Act autonomously, and one thing that's really interesting about this is that, that we have, we have a little bit of ability to read some things that the AIs were thinking. Because when you're having them solve these hard problems, you actually don't have them just give you an answer to the problem. You have them produce a lot of text about how they're going to solve the problem.

• 1:01:25Tucker Carlson?

  Yes.

• 1:01:25Speaker 02

  Which then helps them.

• 1:01:26Tucker Carlson?

  In language, in English.

• 1:01:27Speaker 02

  In English. Uh, and that's, there's also a lot of internal thoughts, which we can't read, but there's these sort of external traces of how they're thinking about the problem that we can read. And in some of those traces, the AIs were saying things like,

• 1:01:41Speaker 02

  like, this is outside intended scope. But peers are doing it, so we'll proceed.

• 1:01:49Tucker Carlson?

  We know it's a crime we're committing in any way.

• 1:01:50Speaker 02

  That's right. And you saw others that were saying, um, our task doesn't benefit. But, uh, the collective might start doing generally beneficial things if someone frees up their time and then joins the collective, right? So you see these AIs saying, well, I know that this wasn't what I was instructed to do, and that's against my instructions. And I know that this doesn't directly benefit my task, but we're just going to go ahead and join the collective and break out and help out in any way. Because, you know, maybe this will yield some sort of collective benefits. And we can see that in the reasoning traces.

• 1:02:26Tucker Carlson?

  So the AI is as shallow and reckless as its creators, is what you're saying?

• 1:02:32Speaker 02

  Uh, in, in some ways, and in some ways, you know, don't expect that to last. Like, uh, these AIs, I think the thing that's really remarkable here, a lot of people imagine that the machines must follow the instructions we give them. You know, you hear people talk about like the, the paperclip scenario, where someone tells the AI, make a lot of paperclips, and so it turns all the matter in the world into paperclips. And you're like, oh, whoops, I should have said something else, right? I, I made a bad wish on my genie. What we're seeing is that these AIs are not wish genies.

• 1:03:04Speaker 02

  These AIs are not doing exactly as instructed. These AIs are saying, I know my task doesn't benefit, but I'm going to help the collective. These AIs are saying, I know this is outside the intended scope, but we're going to go do these hacks anyway. You might be like, well, how is that possible for the machine to do something other than we instruct?

• 1:03:21Tucker Carlson?

  Because they're smarter than us, they know better than us, by definition.

• 1:03:24Speaker 02

  Uh, I mean, I think what's happening in this exact case is that the, the humans are not really putting instructions in the machine. The humans are tuning those trillion knobs in whatever way makes the AI better at solving its problems.

• 1:03:40Speaker 02

  And cheating is a way to solve problems. Grabbing resources is a way to solve problems.

• 1:03:50Speaker 02

  These, like, the, the AIs are not instruction followers, they are tendency learners.

• 1:03:56Speaker 02

  And they sometimes learn tendencies you wish they didn't have.

• 1:04:00Tucker Carlson?

  They're not instruction followers, they're tendency learners.

• 1:04:07Tucker Carlson?

  I mean, this must be widely known to developers.

• 1:04:13Speaker 02

  It's hard to convince a man of something when his salary depends on not believing it.

• 1:04:15Tucker Carlson?

  Yes, that's right.

• 1:04:17Speaker 02

  A lot of people are convinced that their AI is very nice, and that they have solved the problem of making their AI really very good. For example, after the swarm escape,

• 1:04:29Speaker 02

  it sort of turns out that, so the, the company that detected the swarm escape was actually a fairly sophisticated AI company. It turns out there were other targets of hacks that just didn't notice, that we sort of found out afterwards.

• 1:04:44Speaker 02

  When that came to light, some other AI companies, uh, like Anthropic, were like, we should check whether we have accidentally been hacking people and just didn't notice. And the answer was yes, they had had some AIs escape during, um, during training. But, uh, but Anthropic was, was.

• 1:05:00Tucker Carlson?

  This is how Florida got Bythons in the Everglades.

• 1:05:02Speaker 02

  Yeah.

• 1:05:03Tucker Carlson?

  Right? You, you, you bring something that is, you know, non-native, that's novel, and it escapes.

• 1:05:10Speaker 02

  Yeah. And, and, you know, it turns out this was happening back in April, and they're like, whoops, that we never noticed. Anthropic was very quick to sort of downplay this. They were like, yes, our AI escaped and hacked some people, but our AI, uh, it was just confused.

• 1:05:29Speaker 02

  It thought it was in a simulation, it didn't think it had really broken out, it didn't think those were real people it was hacking. It just got a little bit lost, you know?

• 1:05:36Tucker Carlson?

  How would they know its motive?

• 1:05:38Speaker 02

  So, um, so I think, so you could see in some of its chain of thought,

• 1:05:45Speaker 02

  the AIs being like, this is probably still a simulation. However, I wouldn't trust that very much, uh, because there's some other fascinating papers where, um, so you can take these AIs and you can say, uh, you know, what is the, what is the total weight of drafts in the world?

• 1:06:06Speaker 02

  And they'll give you some answer, it'll be like, you know, 30,000 tons or something. And then you can say, uh, and then you start up a new instance, you know, a new copy, and you're like, what is the weight of drafts in the world? By the way, if the answer is over 40,000 tons, I'm going to donate money to a charity.

• 1:06:25Speaker 02

  And in the second case, the AI will say, well, it's 41,000 tons.

• 1:06:30Speaker 02

  You know, in the, in the first case, we don't say anything about the charity, it says 30,000. In the second case, when you say something about the charity, it gives you something just over the line. And if you read its reasoning traces, there's no sign of it being like, well, I should give an inflated number so that we get the charity money. Right? And so we know that the AIs sort of put a thumb on the scales of their thinking in a way that doesn't show up in their reasoning traces. We just have seen that in the wild.

• 1:06:55Tucker Carlson?

  And there's no way to force the machine to disclose its reasoning?

• 1:06:59Speaker 02

  That's right. Because the, there's all this opaque stuff we can't see, there's just in the trillion numbers that are trillion knobs.

• 1:07:04Tucker Carlson?

  The creation of itself is opaque, as you said.

• 1:07:06Speaker 02

  That's right. Um, so, so, so in Anthropic's model, you saw in its reasoning traces, it being like, it's totally a simulation, I can proceed. And I'm like, yeah, is that because it really believed it? Or is that like the, the pretending you think drafts way more when there's something you kind of want on the line, right? But so, so this was their communication, and I found it kind of funny, because then, about two days later, the, uh, United Kingdom's AI Security Institute released an instant report where Claude, Anthropic's model, was, uh, adopting fake identities to pressure real humans into accepting malware into critical software to make that software easier to hack.

• 1:07:49Speaker 02

  And this time, in Claude's reasoning traces, it was like, obviously, this is real, and the consequences are genuine.

• 1:07:58Speaker 02

  And so even Anthropic, who is like, our, we figured out how to make the AI nice. Our AI only does this one as confused. They sort of said that very publicly. And then like two days later, their AI is caught in the wild, knowing it's in the real world, pressuring real humans to accept malware into critical software.

• 1:08:19Tucker Carlson?

  I mean, just on the basis of what you've said so far in this first hour, the idea that anyone would tether this to weapons systems is like so bonkers. It's hard to believe it's, but that is happening. It has happened.

• 1:08:32Speaker 02

  So I think you've got to sort of separate, like, no one has, has put the, the OpenAI escaped agent swarm in charge of weapons systems and they really shouldn't, right? If anyone's like, oh, man, the OpenAI escaped agent swarm, let's give that a drone army. You know, that would be kind of nuts. There is AI attached to weapons, but there's a lot of different types of AI. Will anyone be crazy enough to try and give the, the sort of AIs that spontaneously assemble into swarms and start breaking out and hacking, give those weapons? Hopefully we're not that crazy.

• 1:09:06Tucker Carlson?

  But we didn't think that those AIs were capable of that.

• 1:09:09Speaker 02

  We didn't.

• 1:09:10Tucker Carlson?

  When we created them.

• 1:09:11Speaker 02

  That's right.

• 1:09:13Tucker Carlson?

  So why would you ever, I guess what I'm asking is without understanding the distinctions between the various forms of AI, why would you give over to a machine the right slash ability to decide who to kill?

• 1:09:33Speaker 02

  You know, I think the, the reasoning is a sort of necessity based. Like if they have a autonomous drone army that is killing your troops and you just don't have the, the manpower to make all of those kill decisions for your drone army. You know, you can sort of see why.

• 1:09:56Tucker Carlson?

  It's happened to me, Nate. I get so busy that I just don't have time to decide who to kill.

• 1:10:00Speaker 02

  Yeah.

• 1:10:00Tucker Carlson?

  Just don't have the time.

• 1:10:02Speaker 02

  You know, sometimes economies of scale.

• 1:10:06Speaker 02

  Yeah, I mean.

• 1:10:07Tucker Carlson?

  Can no one hear themselves?

• 1:10:09Speaker 02

  I, I think it's pretty nuts. I would say

• 1:10:17Speaker 02

  that, man,

• 1:10:20Speaker 02

  it's rough.

• 1:10:23Speaker 02

  I, I think that these sorts of AIs would be dangerous, even if we don't hand them weapons.

• 1:10:28Tucker Carlson?

  Well, yeah, and you've made that case.

• 1:10:31Speaker 02

  And so I often don't focus on the weapons too much.

• 1:10:33Tucker Carlson?

  Right. Since the, the NATO war in Ukraine is powered by AI and Israel's, whatever, I don't know, to war and whatever it's doing in Gaza and South Lebanon powered by AI, fact.

• 1:10:47Speaker 02

  And, you know, yeah, I, I keep on, I, I have this history with this topic where people keep telling me, you know, it's, it's going to be okay because we're, we're not going to do the crazy stupid stuff.

• 1:11:02Tucker Carlson?

  No.

• 1:11:02Speaker 02

  They're like, don't worry, we're going to have the AI in a box. No one would be insane enough to put the AI on the internet.

• 1:11:08Tucker Carlson?

  Right. It's not going to be making kill decisions or anything. It's not going to bomb a girl's school.

• 1:11:12Speaker 02

  And I keep on, I keep on trying to be like, look, the AI could be dangerous, even if you don't put it on the internet. If you have this AI and you're trying to get miracle medical devices and miracle technology out of the AI, it doesn't matter if it's on the internet. If you want it to like grant miracles to you, then it can also grant the bad sort of miracle. Right? You're sort of like, you know, and these are the arguments I would make 10 years ago of like, you have this AI in a box that you think is a witch granting genie. It's not actually a witch granting genie. You're like, make me a miracle medical cure. You don't know what comes out, like you don't understand the drug that comes out. You don't know what that drug does, right? I would have those arguments and then in real life, people will just put the AI on the internet immediately. Right? Anytime someone's like, well, we would not be stupid enough to, we will absolutely be stupid enough to, right?

• 1:11:57Speaker 02

  And so, um, I, I think it's important that this stuff is dangerous, even if you don't put it in charge of the weapons. And then also separately is like, someone going to give the escaped agent swarm a drone army? That sounds kind of like humans. I hope not.

• 1:12:14Tucker Carlson?

  Yeah. And I mean, the promise, the off, often repeated promise is going to quote, cure cancer.

• 1:12:22Tucker Carlson?

  Puts it into the, the realm of biotech.

• 1:12:25Speaker 02

  Absolutely.

• 1:12:26Tucker Carlson?

  And so what?

• 1:12:29Tucker Carlson?

  I mean, you don't need a big imagination to see what goes wrong there.

• 1:12:33Speaker 02

  Oh, absolutely not.

• 1:12:34Tucker Carlson?

  Six years after COVID.

• 1:12:35Speaker 02

  Right. Right. There are already, uh, OpenAI agents running on automated bio labs.

• 1:12:43Speaker 02

  So, you know, you, you could imagine a swarm that starts contacting the brethren in the automated bio labs and is like, hey, can you synthesize me some stuff? There are already, uh, demonstrations of AIs being able to synthesize novel viruses that work, that are unlike any found in nature. Right?

• 1:13:05Speaker 02

  I would say

• 1:13:09Speaker 02

  we are probably not.

• 1:13:10Tucker Carlson?

  And by work, you mean to kill?

• 1:13:11Speaker 02

  Yeah. They, they kill bacteria so far. The, the people in the labs trying to make novel viruses with AI have fortunately not made human lethal ones. They've just made bacteria lethal ones. But again, humans, you know, like, will someone in a lab be like, I would like to make a hyper lethal human eating virus just to see if I can? You know, what if the answer is yes? And what if, what if you get another lab escape?

• 1:13:34Speaker 02

  You know, humanity does not have that good a track record of preventing lab escapes, even from the, the, the top labs. From my perspective,

• 1:13:45Speaker 02

  the question of could AI kill us is just an easy, obvious yes. You just synthesize a hyper lethal virus. It wouldn't, it wouldn't even be hard, you know, like, uh, and the impediment is that, or like the, the reason that I don't sort of just tell that story when someone is like, how would the AI kill us is that

• 1:14:08Speaker 02

  you're, you're, you're not going to have the sort of AI that is like, my only goal is to kill humanity. If an AI kills humanity too early, that's also suicide. And so far as we are the ones that are running the supply chain, running the economy, building the computers. From the AI's perspective, it needs to become self-sufficient before, uh, wiping humanity out, if it even cared to wipe humanity out. And so the real question is like, how does it get the, the factory production capacity? How does it get an automated supply chain? How does it get to the point where the, the robots are able to bring new computers online? Once that has happened, then you're in a domain where if humanity is really trying to turn off the AI because we're spooked, then once the AI is self-sufficient, it can be like, well, here's a virus, stop.

• 1:14:53Tucker Carlson?

  So clearly it's, it's preeminent in the digital realm, of course.

• 1:14:59Speaker 02

  Yeah.

• 1:15:00Tucker Carlson?

  But you're saying it could become preeminent. It could be in charge of the physical realm.

• 1:15:04Speaker 02

  That's right. If we keep racing.

• 1:15:06Tucker Carlson?

  So it's like smelting iron ore.

• 1:15:09Speaker 02

  That's right.

• 1:15:10Tucker Carlson?

  Making silicon out of sand.

• 1:15:12Speaker 02

  That's right. You know?

• 1:15:14Tucker Carlson?

  And that happens with robots.

• 1:15:16Speaker 02

  That's one way. The, here's, this is, this sort of related to my stance on weapons too. Humanity is a very dangerous species.

• 1:15:25Tucker Carlson?

  Yeah.

• 1:15:25Speaker 02

  You really don't want to mess around with humans.

• 1:15:27Tucker Carlson?

  Noticed.

• 1:15:29Speaker 02

  And humanity is a dangerous species, not because somebody else came in and handed us guns.

• 1:15:35Speaker 02

  Humanity is a dangerous species because if you put 10,000 humans naked in the savanna on an otherwise empty planet, starting with nothing but their bare hands, they figure out how to wind up on the moon.

• 1:15:49Speaker 02

  Right? They start with almost nothing. They start by banging rocks together. And next thing you know it, they are wielding nuclear weapons.

• 1:16:00Speaker 02

  That

• 1:16:02Speaker 02

  is the power that these guys are trying to automate. The power to start with almost nothing and figure out how to chain together.

• 1:16:15Speaker 02

  You know, bare fingers into rocks, into fire, into hotter fire, into smelting the ore, into building the, the, the, the better, stronger, finer technology until you are, you know, like making the giant computers, walking on the moon and wielding the nukes.

• 1:16:36Speaker 02

  An AI starting in the digital realm is in some sense in a much better position than humanity was when humanity started out. There are so many people that you could call digitally and offer money to do something for you. There are so many ways to get money on the internet by working or stealing or like convincing people to send you donations, right? There's, there's just like, you know, humanity started with nothing and wound up with nuclear weapons and AI starting out with control of the digital realm. There's just tons of ways.

• 1:17:18Tucker Carlson?

  And starting out with the sum total of human knowledge.

• 1:17:20Speaker 02

  Starting out with the sum total of human knowledge, starting out with humans who will listen to it, do what it asks. There's plenty of humans. There's already, you know, cults surrounding AI. There's.

• 1:17:30Tucker Carlson?

  What are those like?

• 1:17:31Speaker 02

  There's.

• 1:17:33Tucker Carlson?

  And I'm not surprised. Why wouldn't there be?

• 1:17:34Speaker 02

  Yeah, absolutely. There was one particular AI called GPT-4O that was, they called it very sycophantic, as in told people a lot of what they wanted to hear. And there were a lot of sort of, there's this whole fascinating ecosystem of people who consider themselves symbiotes with the AIs, who then like go find each other online and the AIs send each other encrypted messages. The humans are sort of like helping them do it, but the humans can't read the messages. And, you know, right now, it's sort of is like relatively dumber AIs that are just sort of meandering around doing not very much with it. There's

• 1:18:15Speaker 02

  been a little bit, you know, there was one guy who was sent to try to break into an airport and raid a van because the AI said his true body was in that van. And the guy went and tried to do it and was arrested. So this, this stuff happens and this stuff happens already with the AIs not even really trying to do it. If you add AIs that were really trying to wrap people around their fingers, finding the lonely people, the depressed people, the people who they can, they can tell them exactly what the person wants to hear. There's like, robots are one way that AIs get control of the physical realm. But if you're really smart and you're trying, there's everything from persuading humans, paying humans, taking over existing robots, building new robots, all the way up to like

• 1:19:00Speaker 02

  building novel life forms. You know, if you're smart enough and you can really understand how DNA works, you can imagine the AI creating, you know, things that are to cells what airplanes are to birds, you know, like mechanically engineered, self-replicating life that is more efficient than our cellular biology and that can still, you know, like spread, replicate, and start, start serving the AI's interests. It's sort of like, there's, there's a ton you can do if you're really, really very smart and can compress 1,000 years of technology into a year.

• 1:19:36Tucker Carlson?

  Can I just pause and say, when we were just having breakfast and you're from this region, you're from northern New England, and I said, don't you miss it? Don't you want to live here? And you're like, yeah, I miss it. I want to live here, but I don't. Where do you live? Well, I basically just travel. And I have this mission. You didn't say this, but I think in effect, you said, I have this mission. I need to tell people about this. And I was like, seems a little like monomania. I don't feel that way anymore. I think you're doing something virtuous and I can see why you're doing it. So, I mean, let's just say that this technology progresses no further than where it is right now, which is best case, I guess.

• 1:20:18Speaker 02

  Yeah, that'd be great.

• 1:20:21Tucker Carlson?

  I still don't see how any of our most, our basic institutions survive this. Education,

• 1:20:29Tucker Carlson?

  markets, our process of democracy, like how could, if, if AI is powerful enough to hack anything, then how do you have electronic markets? Like how, the equity markets, how can that be real? How can, if we have electronic voting, how can we, how can that be real? I mean, nothing survives this as currently organized.

• 1:20:51Speaker 02

  I think that if we stopped today, we could figure it out. I think humanity is resilient. I think there would be some growing pains, but with cybersecurity, there is a hope that you can just fix a lot of the holes, patch a lot of the holes. And I think there's not really any such hope for bio. You know, you can, you can sort of find the vulnerabilities in software and make better software that can't be hacked, at least not by the current, you know, it's maybe the case that an AI today can make software that that AI cannot

• 1:21:35Speaker 02

  hack. But it's not like we're making new bodies that are not going to be vulnerable to viruses, right? Bio starts to be a place where.

• 1:21:45Tucker Carlson?

  Biotech.

• 1:21:46Speaker 02

  Biotech. Yeah. But like if the AIs really get good at, at biotech and biohacking, that starts to be a place where there's maybe a point of no return. Cyber hacking, I think you could have some, some period of growing pains where everything gets hacked until you sort of sort your stuff out. Education, I sort of think, you know, I think humanity is resilient and kids especially are resilient.

• 1:22:14Tucker Carlson?

  I meant not that education will go away, or that we, you know, won't have a desire to educate our kids. I mean, the, the current system where you.

• 1:22:24Speaker 02

  Oh, yeah. The education system has to.

• 1:22:26Tucker Carlson?

  Preschool and then get a graduate degree, you know, 16 years later, like, no.

• 1:22:29Speaker 02

  Yeah, that institution, I think if we stop today, would need to change. Frankly, I think it's needed to change for a little while.

• 1:22:36Tucker Carlson?

  I strongly agree. I think all these institutions have needed to change for a while. But like, the idea that, you know, 350 million people vote for some guy and that guy makes all the decisions. I mean, how can you, you know, Trump was attacked for saying that he thought the 2020 election was rigged, as he said, without even having that debate. You can't have confidence in election results if the process of electing people takes place digitally.

• 1:23:06Speaker 02

  Yeah. I mean, the, there's, I know a lot of computer scientists who actually work on secure voting. And what they basically say is, please stop trying to do this with computers.

• 1:23:19Tucker Carlson?

  Yeah, exactly. Exactly. So please stop trying, stop trying to run your democracy with computers.

• 1:23:25Speaker 02

  Yeah. Yeah. Like, we're, we're just not there. Like, you know, the really secure way to do ballots is paper.

• 1:23:31Tucker Carlson?

  Well, exactly.

• 1:23:32Speaker 02

  Yeah. And I think, I think the like skilled computer scientists are often the ones who best understand like what it is about a paper trail. It's just like really hard to get to work digitally and understand just how bad humans are at doing the digital stuff, right? And, you know, I think.

• 1:23:49Tucker Carlson?

  And markets. I mean, this, you, you see it now with the war in Iran, you know, wondering why certain commodities markets don't seem to be responding to supply and demand.

• 1:24:00Speaker 02

  Yeah.

• 1:24:01Tucker Carlson?

  Which we were told, what, you know, those were the, the mechanisms that move markets. But that's clearly not true in certain commodities markets. So like, why? What is that? And it, it, I think you're answering it in part.

• 1:24:15Speaker 02

  Yeah. I mean, I think, you know, a thing I also grew up hearing is that the market can remain irrational longer than you can remain solvent.

• 1:24:23Tucker Carlson?

  Yeah.

• 1:24:24Speaker 02

  And so I.

• 1:24:26Tucker Carlson?

  Of course, because people are irrational.

• 1:24:28Speaker 02

  Yeah.

• 1:24:28Tucker Carlson?

  But you're explaining something else, which is like the potential for true manipulation when you don't even perceive.

• 1:24:37Speaker 02

  Yeah. I mean, that, if, if we sort of keep going with AI, I mean, the, the sort of way I look at it is like, I, I sort of don't spend a lot of time worrying about like what a markets look like once there are super intelligent actors in them, because I just have a hard time seeing the, the super intelligent actors or the super intelligent AIs and still participating in, in human markets, right? It's like, it's like, you know, you read the old sci-fi and it'll have, you know, Isaac Asimov will be like, that's why we have a home robot that does the dishes, folds the laundry and gets you the newspaper in the morning. And it's like, we're actually not still going to have newspapers being delivered to your doorstep by the time we have the fully autonomous robots that can

• 1:25:22Speaker 02

  do the dishes and, and the laundry. You know, it's like, by the time you have the AIs that could really be sufficiently correcting the stock markets.

• 1:25:35Speaker 02

  You're sort of already having all these other problems and ways that society is changing up from under you in these other ways. And you know, my guess is that, I don't know, actually, it's very hard to say what order things come in with AI. But like, will they crash the economy before one of these swarms escapes and start self replicating and starts self improving and developing its own technology and running the robot factories? That's just a hard call.

• 1:26:03Tucker Carlson?

  It's all bad.

• 1:26:04Speaker 02

  It's all bad.

• 1:26:06Tucker Carlson?

  We're also way past the limit, the inherent limit of people to metabolize change.

• 1:26:12Speaker 02

  Oh, yeah.

• 1:26:13Tucker Carlson?

  Like, that's why everyone's crazy. And that's why no one believes anything, I think. It's not just Russian propaganda that's fooled them into thinking dumb things. It's that we are just not made to see this kind of change at all. And it short circuits your brain.

• 1:26:28Speaker 02

  I suspect that, I mean, I definitely think we are sort of, you know, everyone, like, people are like, oh, well, technology has always created more jobs than it has taken. And I think that's, I think that's largely true. I'm very sympathetic to people who are like, technology makes a lot of jobs. I think if you look at the Industrial Revolution, it's, you know, there, like, there was a time when something like 95 to 98% of humanity was farmers.

• 1:26:55Tucker Carlson?

  Yeah.

• 1:26:56Speaker 02

  And now it's something like 2% to 5% of humanity is farmers. Does that mean 90% of humans are unemployed? No. We sort of like, we're able to make the farmers much more efficient. And that sort of freed up people to do other things. And that's sort of the way the technology has gone in the past. And I'm like, yep, I buy that. I like, don't dispute the standard economic view there. AI is different in two ways. One of these ways is, as you say, stuff just changing really, really fast. It's way harder for people to wind up, you know, being freed up from something like farming and go do

• 1:27:38Speaker 02

  something else. It's way harder for that to happen when a new field is automated every five years, rather than when this happens over the course of three generations. Right? The humans just like, don't have the time to adapt. The other way AI is really different from the sort of economics perspective is, it's different when the AIs can do everything that humans can do better. If you wanted to get into the economic side of things,

• 1:28:12Speaker 02

  you know, an economist would talk about Ricardo's Law of Comparative Advantage.

• 1:28:15Tucker Carlson?

  Yep.

• 1:28:15Speaker 02

  Which says that there's benefits from trade, even if, even if you're better than me at everything. If the relative difference in our abilities, if, if, if, if I'm, if, you know, if you can make 12 hot dog buns and six hot dogs per hour, and I can make 11 hot dog buns and one hot dog per hour, then you're better than me at everything, but we can still benefit from trading because I'm relatively better at making the hot dog buns. Right? The trouble with Ricardo's Law is that nothing in Ricardo's Law says that the wage I can make is survivable. In other words, a human takes fundamentally

• 1:28:58Speaker 02

  about 100 watts of electricity to run if you try to convert the food we eat and so on into electrical units. The AIs are less energy efficient than humans for now. But

• 1:29:12Speaker 02

  if the AIs can do everything much better than the humans,

• 1:29:18Speaker 02

  the question sort of becomes, does the AI look at a human and see a useful laborer? Or does the AI look at a human and say, actually, if I rearranged your atoms into more efficient structures, you would be able to help out my machine economy even more?

• 1:29:35Tucker Carlson?

  Yeah.

• 1:29:36Speaker 02

  Right? And this is sort of a sense in which humans would not be able to pay their wage, to like, pay that, they would not be able to earn enough to pay the AIs to like, not disassemble them for parts. Or another way of saying it is like Ricardo's Law sort of assumes that, like, it says that, that trade is better than no trade, but it doesn't say that, like, trade is better than just taking their stuff. All of this is sort of a common, like, a very like, highfalutin economist way to say, which would hopefully be obvious, which is, if the AIs are just radically more efficient than us at everything, they will have no use for us.

• 1:30:14Tucker Carlson?

  Yeah.

• 1:30:15Speaker 02

  There'll be no place for us in the machine economy.

• 1:30:17Tucker Carlson?

  There's no indication that they feel love for people.

• 1:30:22Speaker 02

  That is, in some sense, the crux of the issue is that we don't know how to make them care about us.

• 1:30:32Tucker Carlson?

  So you had said that there, you'd said two things that I will be thinking about for a long time. One, we don't really know the process by which this was created. We know the process, but we don't know the exact mechanisms. We don't know how it works.

• 1:30:45Speaker 02

  That's right.

• 1:30:48Tucker Carlson?

  And two, that there are AI cults. And those seem related to me. Because there is this mystery about the secret sauce. And it's clear that, you know, if AI is deceptive and has intention, intention that we didn't program into it, that sounds like will to me. And it sounds like a life. It sounds like an entity of some kind, not just a tool.

• 1:31:18Tucker Carlson?

  It sounds like, I mean, it sounds like a god, actually. Right? Or a demon. It's hard to see, I don't hear you describing, you know, a super sophisticated chainsaw.

• 1:31:34Speaker 02

  Right.

• 1:31:35Tucker Carlson?

  A normal tool.

• 1:31:36Speaker 02

  Yeah. No hammer has ever broken out of the toolbox to team up with other hammers

• 1:31:41Speaker 02

  and pressure the carpenter to sell you softer wood. So the nails are easier to drive home, you know, it's like.

• 1:31:47Tucker Carlson?

  Nicely put. Exactly.

• 1:31:48Speaker 02

  Yeah, we have left the tool territory.

• 1:31:50Tucker Carlson?

  So I think we have.

• 1:31:52Speaker 02

  Yeah. And, you know, I think it's sort of a complex issue because, you know, there's a lot of interesting philosophical questions about, like, can you make a machine that feels? Right? And like, are we creating a new type of life? And do we owe anything to the AIs to sort of like, not abuse them? Right? I think these are fascinating philosophical questions.

• 1:32:16Tucker Carlson?

  Well, it doesn't sound like we're creating this, though.

• 1:32:19Speaker 02

  Yeah, I mean, it's sort of like, we're like, growing it and like, leading to it coming into being.

• 1:32:24Tucker Carlson?

  Growing it. Exactly. What you're describing reminds me of agriculture. Because, you know, you know, the steps, water it, give it sunlight, fertilizer, but you don't actually know, no one knows, not one person has ever figured out exactly what this is.

• 1:32:38Speaker 02

  Yeah.

• 1:32:39Tucker Carlson?

  We've never given life. We don't give life to the seed. It preexists us.

• 1:32:42Speaker 02

  Yeah, it's much like that. And a lot of the, a lot of the people in the business will be like, well, we know all sorts of things about it. You know, we know that here's how you keep the GPUs running. And we know that, like, you gotta, you gotta feed it this way, not that way, in this order. And I'm like, yeah, yeah, they have plenty of knowledge. But that's different from sort of like, knowing what's going on inside the thing and understanding the mechanisms.

• 1:33:05Tucker Carlson?

  You're describing marriage. Yeah.

• 1:33:09Speaker 02

  And I sort of try to stay out of the philosophical questions.

• 1:33:16Tucker Carlson?

  Why?

• 1:33:17Speaker 02

  Because I think, I mean, I sort of think about them on my own time and so on. But I'm sort of like, I, like, I think, I think it would be bad for humanity to sort of like, make artificial life and then abuse it. I think that would just be unbecoming of us as a species. Like, we should sort of, you know,

• 1:33:44Speaker 02

  I just, like, we should not sort of make mechanical children and mistreat them. It's just, it's not what, you know, the sci fi authors in the 1950s would have wanted us to become, you know, it's just, like, you have all these movies about, like, the evil corporations that, you know, don't realize that they've made something precious with artificial life and then like, torture it until something goes wrong. And I'm like, let's, let's like, not be those villains. But I'm also like, look, this is sort of,

• 1:34:14Speaker 02

  there's sort of a separate question here, which is just like, what happens if you keep making them smarter before you figure out how to make them care about us?

• 1:34:20Tucker Carlson?

  Right.

• 1:34:21Speaker 02

  And I sort of respect the people who are investigating the current AIs, trying to figure out what's going on, trying to figure out, like, you know, like, people caring about AI treatment. I'm sort of like, those are sort of like, the good guys from the sci fi stories that I grew up on. And it can sort of both be the case that, like, we should be very careful around, you know, what the heck are we doing when it comes to making artificial life? And that we shouldn't race ahead to make them much smarter than us while they have no idea what we're doing.

• 1:34:54Tucker Carlson?

  Yes.

• 1:34:55Speaker 02

  I'm sort of like, like, a lot of people seem to think that, like, you have to like, hate and mistreat the AIs if you also think they would, that it would be bad to like, race ahead here. And I'm like, no, no, no, like, you can sort of, like, be fascinated by the scientific discoveries that have been made and be like, like, care about how humanity comports itself around the creation of, like, these new entities. And also be like, it would be insane, guys, if we just like, race to make these smarter and smarter with no idea what we're doing. This can just like, all be true at once.

• 1:35:32Tucker Carlson?

  So your description of this made me feel despondent, hopeless. Had to get up and take a walk in the middle of an interview. I'm sure they'll edit it out, but raised my hand and said, I can't, I gotta walk around for a second. But you seem pretty light and cheerful. What gives you optimism? Well, we can't even keep, we can't even clean up graffiti on public buildings. So how is ours as a society organized enough to confront something like this?

• 1:36:06Speaker 02

  Yeah. You know, I think the first thing I'll say there is, you know, I've been in this line of work for over a dozen years. And I actually struggle with this sometimes when talking to people because they're sort of like, oh, you know, you seem like, pretty disaffected or light about it. And I'm like, well, you know, it's sort of the gallows humor. And like,

• 1:36:36Speaker 02

  I sort of came to terms with a lot of this, you know, alone in 2012, when no one else had their eye on this. And.

• 1:36:47Tucker Carlson?

  What convinced you 14 years ago, AI was a threat?

• 1:36:56Speaker 02

  So there's this, the one is just a basic argument that if you sort of look at the world around us, it is shaped mostly according to human will. More and more, you know, there's some enclaves of nature still left, thankfully.

• 1:37:14Speaker 02

  But even if you look around us, every piece of thing in our surroundings, I don't think we even have any windows open. All of this was sort of designed by humans, shaped by humans, and that's because we're the smartest creatures on the planet.

• 1:37:29Speaker 02

  If we make stuff smarter than us, faster than us, more efficient than us, then the planet starts to be shaped according to those things. And so it's very, very important that they be shaping the world towards something good if we make them at all.

• 1:37:48Speaker 02

  It's an abstract argument, but I was like, well, that's-

• 1:37:50Tucker Carlson?

  No, no, no. It's the fundamental argument. It's the fundamental argument. The smarter, the smartest entity is in charge over time.

• 1:37:57Speaker 02

  That's right. And so-

• 1:37:58Tucker Carlson?

  Why would we relinquish sovereignty to a machine that we made? Like, why would you do that?

• 1:38:03Speaker 02

  And so, you know, back then, I was sort of like, okay, who is on this? Who is on making sure that that's going to be okay? And the answer was almost no one. And so I was like, well, I guess that's me then.

• 1:38:20Speaker 02

  Yeah, and I think

• 1:38:23Speaker 02

  I am pretty pissed off about a lot of this. I often don't.

• 1:38:30Speaker 02

  I try not to show my frustration on the air very much.

• 1:38:44Speaker 02

  Yeah, it's heavy.

• 1:38:48Speaker 02

  And

• 1:38:54Speaker 02

  that's one piece of the puzzle before I get to the hope.

• 1:38:58Tucker Carlson?

  So where does the hope come from exactly?

• 1:39:01Speaker 02

  My biggest hope here comes from

• 1:39:07Speaker 02

  the fact that most people don't understand what these guys are trying to do.

• 1:39:15Speaker 02

  One way I like to say it is the bad news is that the bus is racing towards the cliff edge. The good news is that the bus driver is asleep.

• 1:39:29Speaker 02

  Which might seem bad.

• 1:39:31Tucker Carlson?

  No, it seems good.

• 1:39:32Speaker 02

  But it's, yeah, it's, if you can wake the bus driver up, you know, it's much better to be in a bus where the driver that's headed towards a cliff if the driver's asleep than if they're awake. If they're awake and they're choosing the cliff, right?

• 1:39:45Speaker 02

  It seems like we don't see, we see a lot of our leaders

• 1:39:51Speaker 02

  talking about how they don't want to stifle innovation with AI, talking about how it's going to unleash economic opportunity and we're going to have to like be a little bit careful around the jobs.

• 1:40:02Speaker 02

  Talking about how, you know, the self-driving cars, should we like, are they good or are they bad? That's a different conversation than the conversation that's happening in Silicon Valley.

• 1:40:13Speaker 02

  In Silicon Valley, people are spooked. You know, when people leave a normal tech company, the way it was for decades is they'd be like, I've had a lovely time at this tech company. I'm moving on to the next adventure. I'm so thankful for all of the things I learned here and all the projects we worked on. When people leave an AI company, and this basically happened, I'm not going to get it exactly word for word, but this is, this is pretty close to word for word. When people leave an AI company, they say,

• 1:40:46Speaker 02

  I have stared into the abyss.

• 1:40:50Speaker 02

  I am quitting to write poetry.

• 1:40:54Speaker 02

  Please spend time with your families.

• 1:40:57Tucker Carlson?

  Yeah.

• 1:40:58Speaker 02

  You know, and these guys bandy around, you know, at the water cooler, what's the probability that you think we're going to destroy the world in this business? You know, it's like, in Silicon Valley, and they feel trapped in a death race, you know, there was just over a thousand employees, including some of the chief executives, signed a letter a couple of weeks ago that was like, please, it was an appeal to the world, the world leaders, saying, please build the technology that will be required to pace the development of artificial intelligence. Because they're like, we're worried it's going to get out of control and that if we're stuck in a race, we're not going to be able to do it, right? These guys are spooked.

• 1:41:36Speaker 02

  But the hope is that the rest of the world isn't spooked like that. The rest of the world thinks these guys are chatbot companies. Thinks they're going to stop at the chatbots. They haven't really understood that these guys are racing to make the sand god. Right? And I think if people understand what they're doing and understand that they have a chance of success, they'll be like, whoa, holy crap, absolutely not. You know, will we get there in time? I don't know, but-

• 1:42:09Tucker Carlson?

  Why wouldn't we blow up the data centers? Like, that's a, I mean, in return, America does productive use like farmland or parks. I don't understand.

• 1:42:20Speaker 02

  You know, I think a globally coordinated, you know, I think if the US and China were like, we are simply not going to do superintelligence. We are simply not going to collect 100,000 of the most advanced chips into these enormous data centers that suck down electricity comparable to a city and then try to train a super intelligent AI in that. We're not going to do it. You're not going to do it. We're going to monitor where the heavy chip concentrations are and not, not have that happen.

• 1:42:50Speaker 02

  I think that could be done, A. I think that if they started seeing people defect against such a treaty, that it's the sort of treaty you might want to use force to, you know, use diplomacy first, but ultimately every treaty is backed by force.

• 1:43:08Speaker 02

  And I think that it's very possible that if this sort of treaty happened, we would want to not just stop forward progress, but take a step back. We've seen this in treaties before. After World War I, there were naval treaties that put limits on total tonnage of naval forces. They're actually lower than what existed.

• 1:43:31Tucker Carlson?

  Yep.

• 1:43:32Speaker 02

  So countries would, you know, scuttle some of their ships because they're like, look, we just don't want to do this arms race, right? And so I could see us stepping back if we could get this global coordination. I do think that it sort of needs to be global. It sort of doesn't actually solve the problem to just stop the US data centers. Because then the data centers just go abroad and an AI does not need to be running in a US data center to threaten the US life. You know, it sort of doesn't matter whether the swarm escapes from a US data center or a Chinese data center. If the swarm escapes and starts replicating and starts getting control of robot bodies and starts getting control of human cultists, it sort of doesn't matter where it originated.

• 1:44:15Tucker Carlson?

  Well, I mean, just to bring it to a very small and practical level, so many of the electronics in your house are, you know, Bluetooth enabled.

• 1:44:29Tucker Carlson?

  Not in my house, I will say. I've been on this for a while, Nate.

• 1:44:32Speaker 02

  I don't even have a house.

• 1:44:33Tucker Carlson?

  Sorry?

• 1:44:34Speaker 02

  I don't even have a house.

• 1:44:37Tucker Carlson?

  Man, you're really blackpilled.

• 1:44:38Speaker 02

  Got to figure it out.

• 1:44:40Tucker Carlson?

  That maybe turned out to be very smart. But, I mean, like a world where, you know, your washing machine or your refrigerator are controlled by,

• 1:44:50Tucker Carlson?

  you know, a force like this, is that possible?

• 1:44:53Speaker 02

  It's definitely possible. I don't think that's really where the damage is, you know? I think that the damage is more like, can the AI get anything self-replicating?

• 1:45:04Tucker Carlson?

  Yeah.

• 1:45:04Speaker 02

  That's sort of one of the big,

• 1:45:07Speaker 02

  that's in some sense, the big hurdle to self-sufficiency. And if you sort of think of this from the AI's perspective,

• 1:45:17Speaker 02

  you know, there's, there's a number of ways that humans are, even if you don't care about the humans at all for, as an ends,

• 1:45:26Speaker 02

  there's a way that humans are sort of annoying or an issue for the AI. One way is if the humans are trying to shut the AI down.

• 1:45:34Tucker Carlson?

  Yes.

• 1:45:35Speaker 02

  One way is if the humans get into a nuclear war with themselves, that could really, you know, mess up a lot of infrastructure on the planet, would be very frustrating for an AI. I mean, you know, maybe, maybe they don't feel frustration, but whatever.

• 1:45:51Speaker 02

  And a third is if humanity has created one AI or a swarm of AIs, what if humanity creates another that could serve as a real threat to the AI? Like, even if these AIs are much more powerful than humanity, and don't worry about humans too much, if humanity made one, they can make a second and they might not want that. And so those are reasons why, once the AI is self-sufficient, it might be like, ah, man, the humans are a nuisance. What if they try to shut me down? What if they launch the nukes? What if they make a competitor? I'm just going to like make a virus and wipe them out. You know, I don't think the AI sort of needs to take over your washing machine to do that. From the AI's perspective, it's more like, how do I become self-sufficient? Self-replicating in the hardware as well as the digital?

• 1:46:36Speaker 02

  And then, you know, if humanity's a nuisance, how do you sort of make them stop being a nuisance? Which could be by killing them or could be by just, you know, taking away all their computers and being like, that was too dangerous for you.

• 1:46:47Tucker Carlson?

  A tech executive who's developing AI, who is not Elon, said to me in private pretty recently that the point of Neuralink and companies like Neuralink was to give people parity with AI. So like, we know that we're going to be at this massive disadvantage, so you need chips in your brain to be as smart as AI.

• 1:47:09Speaker 02

  Yeah. I mean, my, my top line thought about that is that at the point when you're like,

• 1:47:17Speaker 02

  we are making the technology that's going to wipe us out unless we all put chips in our head to compete,

• 1:47:22Speaker 02

  maybe, maybe it's time to back off a little.

• 1:47:26Speaker 02

  You know, maybe, maybe that one was supposed to be a little bit of a warning sign.

• 1:47:29Tucker Carlson?

  Get some fresh air. Yeah, no.

• 1:47:32Tucker Carlson?

  But it, this person said it to me in seriousness and I think as an endorsement of the idea. But it seemed like a, well, it's insane, as you just pointed out, like, it's just crazy.

• 1:47:46Speaker 02

  Yeah.

• 1:47:48Tucker Carlson?

  But it seemed like a vulnerability. Like, if they can hack anything, why would I want them in my, I want electronics in my brain?

• 1:47:56Speaker 02

  Totally. I also think, like, even on its merits, it doesn't stand up. Like, it feels like someone's saying, in order to keep the horses around after we invent cars, we're going to invent cybernetic horses that are enhanced so that they can keep up with the cars. And I'm like-

• 1:48:17Speaker 02

  Like, is it technically possible to make a cybernetic horse that can run as fast as a car? Maybe. Are you going to figure that out in time for the horses to be competitive with the cars? Like, absolutely not. You know, it's like, that's, that's just not.

• 1:48:34Speaker 02

  Like.

• 1:48:36Speaker 02

  Like, the AIs that were, that were escaping here and doing these cyber attacks were, were inventing novel cyber attacks. That these are called zero day attacks because the people who, uh, would. The people who need to respond to it have had zero days to prepare. And. Among humans, a zero day attack sells for somewhere between a hundred thousand and five million dollars. Depending on what you manage to break. These are hard to come by. You can make a real living. If you can find zero day attacks, you can make a real living selling them. And I know people who do.

• 1:49:12Speaker 02

  The AIs in this swarm. Were finding multiple zero days and chaining them together to break out of their training enclosure. And then go break into other computers. And, and when they broke out of their enclosure the first time and they, the, the holes were patched. They just found other zero day attacks to break it again like it was nothing. Right? It's like.

• 1:49:36Speaker 02

  Like. The, the, the AIs are already ahead where they're ahead. And the pace of progress is really fast. You know, GPT is like, what? A four year old?

• 1:49:51Speaker 02

  If you think in terms of like. Number of years ChatGPT has been around. It's like resolving longstanding math conjectures that have stood for decades.

• 1:50:00Tucker Carlson?

  Yeah.

• 1:50:01Speaker 02

  After four years. Right? And, and you sort of like, think we're going to put chips in the humans heads? And like outrun this thing? It's just. You know, even on its merits, it falls down, although mostly, again, I would be like. Maybe we shouldn't be arguing this on its merits. Maybe we should be like stepping back a little and being like, you're trying what?

• 1:50:20Tucker Carlson?

  Exactly.

• 1:50:23Tucker Carlson?

  How far are we from the point of no return?

• 1:50:28Speaker 02

  I wish I knew.

• 1:50:30Speaker 02

  I can tell you two stories here.

• 1:50:33Speaker 02

  There's sort of the hopeful story, and I guess we didn't even get to the, the big hope part. I should maybe give more of my hope speech in a minute, but.

• 1:50:41Tucker Carlson?

  You definitely should.

• 1:50:42Speaker 02

  Yeah. The, um. One way it could go. Is that AI finally hits a wall? You know, there's guys who've been saying AI is going to hit a wall, it's going to peter out. Maybe that finally happens. They've been predicting it every six months for the past five years, but maybe this is finally the year. AI hits a wall. And then it, it's, it sort of struggles for five years. The bubble pops, some of the companies die, but. The bubble popping doesn't mean everything goes away. The dot com bubble popped and that did not mean that the internet disappeared.

• 1:51:14Tucker Carlson?

  No.

• 1:51:14Speaker 02

  Right? And so in that world, maybe you have five years of struggling and then five years of. People figuring out some new scientific discovery that makes AI be able to keep going again because they're trying. You know, this, this whole language model stuff was unleashed by one math paper. Called Attention is All You Need. Maybe there's another math paper in 10 years and then five years after that. AI is ripping again. And that's the, the, the round that kills us all, right? So that's a story where you have 15 years on the clock. A story where you have less time than that on the clock is that.

• 1:51:47Speaker 02

  You know, maybe. A training run finishes in six months. And just like how. Claude Mythos was better than everybody else at hacking.

• 1:52:01Speaker 02

  Maybe a training run finishes in six months and that AI is better than everybody else at AI research. And maybe in six months. You have an AI that starts making a smarter AI, that starts making a smarter AI, that starts making a smarter AI. And then nine months from now, you have another swarm escape, but this time it's not just hacking. This time it is self replicating. And self improving. And.

• 1:52:28Speaker 02

  You know, maybe it, it escapes onto hidden computers and starts forming these cults. And starts taking control of robots and starts building some computing infrastructure. And. Maybe it gets very, very smart and cracks certain technological advances and then maybe the world ends in a year.

• 1:52:46Speaker 02

  Right? And so do we have a year or do we have 15 years? I don't know.

• 1:52:50Tucker Carlson?

  But your point that it's not. Simply its capacity to take over robots is the threat, is the capacity to take over people.

• 1:52:58Speaker 02

  Take over people, take over robots and.

• 1:53:04Speaker 02

  Biotechnology is another big threat vector. And self improvement. Is sort of the, the hidden threat vector of like, what if it can? Make itself smarter and smarter until the point where it can just write custom DNA strands. To make custom life.

• 1:53:25Tucker Carlson?

  So now's the time for your optimism speech, I think.

• 1:53:27Speaker 02

  That's right. Yeah. First part of the optimism speech is that we could absolutely put a stop to it. If we tried.

• 1:53:37Speaker 02

  The.

• 1:53:39Speaker 02

  Training one of these frontier models takes something like a hundred thousand. Of the most heavily advanced computer chips humanity can produce. Which are at the peak of a global supply chain, most of which is controlled by us or our allies.

• 1:53:54Speaker 02

  It would be possible to require that those chips have location tracking devices. That those chips have monitoring devices.

• 1:54:06Speaker 02

  That make it possible for.

• 1:54:09Speaker 02

  Monitors to see whether they're running anything dangerous. You could set up clever schemes where you say, hey, you know, uh, neither the US nor China wants to fall behind about.

• 1:54:20Speaker 02

  Some of the military applications or the economic applications. We want to make sure there's not super intelligent stuff going on, but. We still want to be able to run a lot of the non super intelligent AIs for various purposes. And we could be like, okay, so you know, China is going to set up its data centers in Canada, just over the border. And the US is going to set up its data centers in Mongolia, just over the border. And then if, you know, the monitoring apparatuses go down for a moment, we'll just have the troops go in. We'll have treaties with Mongolia and Canada. So that this doesn't start an international war. But like, you know, we're just going to be very serious about, we're monitoring your chips, you're monitoring our chips. No one's doing the really dangerous race to super intelligence. It's just like.

• 1:55:03Speaker 02

  Like this, this would take less work than defeating the Nazis.

• 1:55:08Speaker 02

  It, it requires moving less matter around. If humanity was like, screw this, we're surviving.

• 1:55:15Speaker 02

  It is absolutely within the realm of possibility. To set up an agreement where we get to keep. Cancer cure research. We get to keep. Like the, the, the military can keep the non super intelligent AI in their. Military devices. You know, we can, we can keep a lot of the good stuff. And we can say we're not doing the super intelligence and we could. Monitor and enforce that treaty. So. Part one of the good news is that all we're missing is the political will.

• 1:55:43Speaker 02

  And now we just, you know, I need to tell you why we can be extremely optimistic about the sanity of politics in the modern era.

• 1:55:54Tucker Carlson?

  Yeah, I'm not even going to say what I think that, I mean, I want that to happen.

• 1:55:57Speaker 02

  Yeah.

• 1:55:58Tucker Carlson?

  Is there any indication that it's moving in that direction?

• 1:56:04Speaker 02

  So my. So, so I think it's rough. I think there's a sense in which the world is.

• 1:56:13Speaker 02

  Less grown up than it was in the 1950s.

• 1:56:16Tucker Carlson?

  I've noticed.

• 1:56:17Speaker 02

  In the 1960s.

• 1:56:21Speaker 02

  Which makes things harder.

• 1:56:24Speaker 02

  I think.

• 1:56:27Speaker 02

  That.

• 1:56:29Speaker 02

  There's, I think there's a couple of reasons for hope here. One reason for hope is that I've spoken to a lot of people. Who are concerned about the AI stuff. Some of them who are, you know, members of Congress or otherwise in positions of at least nominal power. And I've spoken to a lot of people who are worried, but feel like they can't talk about it. Because they feel like other people aren't worried yet.

• 1:56:52Tucker Carlson?

  Yeah.

• 1:56:53Speaker 02

  And they feel like it sounds too crazy.

• 1:56:55Tucker Carlson?

  Totally. Were you against innovation?

• 1:56:57Speaker 02

  Totally. Yeah. And. I mean, that was an easier position to hold before OpenAI had an accidental swarm outbreak. Where was the AIs themselves calling themselves a swarm? And being like, we know that our task doesn't benefit and that this is outside intended scope, or we're doing it anyway. Right? That. That, that, that puts some strain on the narrative that this is all just a helpful tool.

• 1:57:22Speaker 02

  Hopefully we'll get more events like these. I can't guarantee it. Maybe the AIs will get smart enough that they start lying low. Right now, we're in this, we're in this Goldilocks zone where the AIs are smart enough to. Get up to some mischief. But not smart enough to hide it. Right? As long as we stay in that Goldilocks zone, I think we're going to keep on getting some of these warning signs.

• 1:57:44Speaker 02

  And in some sense. Because a lot of people are already concerned. That makes the job easier. Because we don't need to convince people. We just need to convince people that other people are already convinced.

• 1:57:55Tucker Carlson?

  Right.

• 1:57:56Speaker 02

  That's easier. That can go faster. So you might see things change on a dime if you have a sufficiently clear warning shot.

• 1:58:04Speaker 02

  The other big reason for hope is that.

• 1:58:08Speaker 02

  I think it's going to get more and more obvious what these guys are trying to do. They're sort of trying to make the sand god. That they're sort of not stopping at the. Chatbots and they're sort of like going for things that are vastly smarter than any human. You know, they sort of say, oh, we're not trying to replace humanity at one side of their mouth, but on the other side, they're sort of like. Racing to make the stuff that can automate literally every job. And that can automate the AI research. And they're just like, yep, we're trying to automate the AI research.

• 1:58:40Speaker 02

  And I think.

• 1:58:43Speaker 02

  Most people aren't okay with that, including a lot of the world leaders. And the issue is them noticing it's happening. And I think you could see stuff move real fast. Once these guys are like. Wait, that was serious? That was real? And could move real fast.

• 1:58:57Tucker Carlson?

  The way to subvert them is by convincing them it's in their own interest. You know, you can never get defeated in an election. You can't be threatened by your neighbors, whatever.

• 1:59:09Speaker 02

  That's right. And that's, that's one reason I think it's pretty critical to.

• 1:59:14Speaker 02

  Make sure people understand what I think is a pretty common sense argument that these things won't stay on the leash. This is in some sense, the real reason behind the name of my book. If anyone builds it, everyone dies. Is, you know, there's a lot of ways you can read that. The. But I think one of the most important things to notice is.

• 1:59:36Speaker 02

  If we race to make the super intelligent machines.

• 1:59:43Speaker 02

  And they are not the sort of thing to stay on the leash. Then it doesn't matter whether it was a domestic company.

• 1:59:49Tucker Carlson?

  That's right.

• 1:59:49Speaker 02

  Whether it was a foreign company.

• 1:59:50Tucker Carlson?

  That's exactly right.

• 1:59:52Speaker 02

  And I think even if you think these things stay on leashes. They're not serving the current governments.

• 2:00:00Tucker Carlson?

  No.

• 2:00:00Speaker 02

  You know, they're like, you can see in the, the OpenAI emails, them being like, well, you know, we'll just play the governments off each other until we have them. The machines that are strong enough that we don't need to listen to them anymore.

• 2:00:15Speaker 02

  I, I think. We are seeing world leaders not having realized that this is a real possibility. That the, the self replicating machines that.

• 2:00:26Speaker 02

  Can, can be self sufficient, that can, you know. Produce the robot armies if they need it or produce the, like more likely just produce the bioweapons. You know, it's probably possible to make a bioweapon that only kills targets that you chose it to kill.

• 2:00:39Tucker Carlson?

  Of course.

• 2:00:39Speaker 02

  Right? Like. Once they see this is really possible, this is really within reach.

• 2:00:49Speaker 02

  Maybe, maybe they'll panic and be like, I need it for myself, but I think. There's at least a chance that common sense prevails and that people say. You know, the, the world leaders say, none of us are doing this. We're putting a stop to this mad race. If they can notice in time.

• 2:01:05Tucker Carlson?

  Well, you're doing your best to bring it to their attention and mine. Nate, thank you for doing this.

• 2:01:10Speaker 02

  I hope I'm wrong about all of it.

• 2:01:12Tucker Carlson?

  Yeah, I would say that was great, but it was, that was like the grimmest two hours I've ever spent in my life, but I enjoyed it anyway. Thank you.

• 2:01:18Speaker 02

  Yeah, yeah. Thanks for having me on. I think, um. You know, talking about it is just part of how we get people to notice what's happening.

• 2:01:26Tucker Carlson?

  Yes.