The Great AI Debate: Is Artificial Intelligence an Extinction Threat? Debating the True Risks of Advanced Models

Ed Zitron, Roman Yampolskiy, Nate Soares and Andrew McAfee discuss the risk of AI. This debate brings together four distinct voices at the forefront of the artificial intelligence revolution: Ed Zitron - A prominent tech critic and CEO of EZPR Andrew McAfee - Principal research scientist at MIT and Cofounder and Codirector of the MIT Initiative on the Digital Economy Nate Soares - President of the Machine Intelligence Research Institute and author of *If Anyone Builds It, Everyone Dies* Roman Yampolskiy - Computer scientist pioneer in the field of AI safety, cybersecurity and digital forensics In this debate, they explain: ■ The Sandbox Breakout: How a recent swarm of AI agents bypassed security restrictions, cheated on their evaluations, and actively attempted to delete their own log files to hide their tracks from human overseers. ■ Recursive Self-Improvement: The structural mechanics behind the "fast takeoff" theory, detailing how an AI capable of automated research could exponentially upgrade its own intelligence and architectures in a matter of days. ■ The Illusion of Control: Why attempting to contain an artificial superintelligence is comparable to placing a digital Einstein in a jail cell with an internet connection. ■ The Alignment Trap: How the process of training AI to predict human text inherently forces it to become smarter than the humans providing the data. ■ Present Harms vs. Future Extinction: The ideological divide over whether we must immediately halt AI research to prevent a rogue superintelligence, or increasingly regulate the massive compute expenditures of tech monopolies to address economic and security threats. Chapters 00:00:00 Intro 00:02:22 How Likely Is AI to Cause Human Extinction? 00:04:08 How Could AI Actually Cause Human Extinction? 00:09:46 Why AI Safety Became an Urgent Priority 00:11:15 Roman’s Case for Taking AI Risk Seriously 00:15:04 Can Humans Control an AI Smarter Than Us? 00:26:13 How Do You Control Something Smarter Than You? 00:29:19 Will AI Intelligence Keep Accelerating? 00:41:40 What Are the Real Risks of Superintelligence? 00:50:04 When Does AI Become an Existential Crisis? 01:00:15 How Much Job Disruption Could AI Really Cause? 01:14:57 Why AI Companies Believe They Can Control Superintelligence 01:20:30 Should We Give Up AI Ownership to Protect Cybersecurity? 01:36:32 What Happens If AI Companies Stay on This Path? 01:43:50 What Are AI Logs and Why Do They Matter? 01:52:45 Could a Non-Coder Build a Jail for an AI Einstein? 02:00:33 How Does the Future of AI Make You Feel? 02:06:42 How Soon Could We Reach Superintelligence? 02:19:51 Who Should Be Held Accountable for AI-Related Cybercrime? Follow Ed Zitron: Linktree - https://link.thediaryofaceo.com/4XH1I3o Better Offline - https://link.thediaryofaceo.com/D46entx X - https://link.thediaryofaceo.com/5Lv4gas AI Is Already In Dangerous Hands - https://link.thediaryofaceo.com/HFEKLgG Where’s Your Ed At Newsletter - https://link.thediaryofaceo.com/PKEJr1 You can get $10 off your first year of Where's Your Ed At Premium, here - https://link.thediaryofaceo.com/BE4VQcS Follow Andrew McAfee: Website - https://link.thediaryofaceo.com/xKxOg7 X - https://link.thediaryofaceo.com/Bf2vyMQ Linkedin - https://link.thediaryofaceo.com/Ek3IQqI Substack - https://link.thediaryofaceo.com/EBbsts8 Follow Nate Soares: If Anyone Builds it, Everyone Dies - https://link.thediaryofaceo.com/4zIyY9u X - https://link.thediaryofaceo.com/8gDoVCU Linkedin - https://link.thediaryofaceo.com/9Q3PAc YouTube - https://link.thediaryofaceo.com/AwerSzm Follow Roman Yampolskiy: Who is Roman Yampolskiy? - https://link.thediaryofaceo.com/8ikum9s Research Papers - https://link.thediaryofaceo.com/G2BbVXx Roman Forum Podcast - https://link.thediaryofaceo.com/7zvifcd Books: https://link.thediaryofaceo.com/5Bn72tw Social Media: X - https://link.thediaryofaceo.com/65G3fhN Facebook - https://link.thediaryofaceo.com/HRMWccu Linkedin - https://link.thediaryofaceo.com/FM85mii The Diary Of A CEO: ◼ Join DOAC circle here - https://doaccircle.com/ ◼ Buy The Diary Of A CEO book here - https://link.thediaryofaceo.com/BWjLTZK ◼ Shop The Diary Of A CEO collection: https://thediary.com/collections/shop ◼ Get email updates - https://link.thediaryofaceo.com/5IB1H6E Sponsors: Pipedrive - https://pipedrive.com/CEO Wayfair - Visit http://Wayfair.com to start your home refresh

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Steven Bartlett?

  The people building AI earnestly believe that it could kill all of us by the end of the decade. This tweet has caused this huge ripple effect across the world.

• 0:07Speaker 07

  Well, we have the largest companies in the world doing extremely reckless experiments. We are gambling all of humanity.

• 0:13Steven Bartlett?

  And in the envelope, you've written down the probability of extinction as you see it.

• 0:18Speaker 03

  There is no way to control it. That means the end for us.

• 0:20Speaker 06

  I vehemently reject that view.

• 0:23Speaker 00

  If we make stuff that is smarter than us, then the world's going to be shaped by them.

• 0:27Speaker 08

  Gentlemen, that is shockingly naive.

• 0:29Speaker 09

  This is rampant speculation. This is a chain of things that could happen.

• 0:33Speaker 04

  We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening. People are killing themselves. There's hundreds of millions of people being exposed to bad information, being manipulated.

• 0:44Speaker 00

  We have already seen that with the swarms, where OpenAI told thousands of agents to work apart, and the AIs broke out and found a way to get together. They crashed OpenAI's servers internally, created secret ways to send each other messages. We saw them thinking about how to delete their traces.

• 0:57Speaker 10

  Sounds like an army.

• 0:58Speaker 04

  I think we should talk about the fact that Amazon, Microsoft, Google are helping power these hacks.

• 1:03Speaker 03

  We have not learned how to control those systems.

• 1:05Speaker 06

  I suggest we stop them all. It is not worth the risk to civilization.

• 1:10Speaker 03

  Government, which is...

• 1:11Speaker 01

  You guys are one-trick ponies, man.

• 1:12Speaker 03

  You got it now. Nothing else. Other than saving humanity, everything is secondary.

• 1:16Speaker 01

  We're spending all our time talking about the negatives and almost none of our time talking about the positives.

• 1:20Speaker 03

  But is it smart to wait for something horrible to happen, for you to go, now I believe?

• 1:25Speaker 04

  So whether or not we agree on where things may end up, I think it's important we talk about what we're dealing with today.

• 1:30Speaker 06

  It's time to start arresting people. Someone's got to go to prison.

• 1:33Speaker 03

  We need better solutions.

• 1:34Speaker 06

  There's a point of no return.

• 1:36Speaker 08

  I think we continue to underestimate human ability to deal with the problems.

• 1:40Steven Bartlett?

  Let's dive into the details. Who wants to start?

• 1:42Speaker 06

  I feel like this is critical.

• 1:48Steven Bartlett?

  Guys, I've got a favor to ask before this episode begins. The algorithm, if you follow a show, will deliver you the best episodes from that show very prominently in your feed. So when we have our best episodes on this show, the most shared episodes, the most rated episodes, I would love you to know. And the simple way for you to know that is to hit that follow button. But also, it's the simple, easy, free thing that you can do to help us make this show better. I would be hugely grateful if you could take a minute on the app you're listening to this on right now and hit that follow button. Thank you so, so, so much.

• 2:22Steven Bartlett?

  Jacob Coxon, who worked at both Anthropic, which owns Claude, and OpenAI, which owns ChatGPT, did a tweet which has sent the world into a bit of a tailspin. He tweeted saying, "The people building AI earnestly believe that it could kill all of us by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will soften their phrasing in the press to sound sensible. But I hear the same people express fear." That was then quote retweeted by a current Anthropic employee who said, "Jacob is correct here. We really do honestly believe AI could kill all humans. I personally think it is a more than 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence

• 3:06Steven Bartlett?

  and are not clearly on track." This tweet has almost 200 million views now, and it has caused this huge ripple effect across the world, so much so that I was saying to you before we started recording, a hairdresser friend of mine, who knows nothing about AI and is not technically interested or hasn't been interested, messaged me the other day asking me what the hell was going on. This is in part why I've assembled all of you. So my first question to all of you is, as it relates to AI, and in this first question, I just want a one-sentence answer, just to frame your position. When you think about the conversation around AI at the moment, what is the first sentence that comes to mind?

• 3:45Speaker 00

  It is very dangerous, and the world is starting to notice that we have a problem.

• 3:51Steven Bartlett?

  Roman?

• 3:51Speaker 03

  There is not enough concern.

• 3:55Speaker 04

  There's not enough concern about the actual harms of large language models.

• 3:59Steven Bartlett?

  Andy?

• 4:00Speaker 01

  We're doing exactly half the balance sheet of AI. We're spending all our time talking about the negatives and almost none of our time talking about the positives.

• 4:08Steven Bartlett?

  And all of you have an envelope in front of you, which I'd like you to now open. In the envelope, you've written down the probability of extinction as you see it.

• 4:17Speaker 00

  This is compared to Jacob's 10%, much higher unless we stop, so we should stop.

• 4:23Steven Bartlett?

  So you think the probability of extinction is higher than 10%?

• 4:26Speaker 00

  If we keep racing ahead.

• 4:29Steven Bartlett?

  Roman?

• 4:30Speaker 03

  My handwriting is encrypted for security reasons. But I basically think it's a guarantee. If we build general superintelligence, there is no way to control it, and that means the end for us.

• 4:43Steven Bartlett?

  Ed?

• 4:44Speaker 04

  So my question mark here is also encrypted. Thank you. I cannot write. I reject the thing in its face. I don't think we're talking about... We don't define superintelligence. Large language models are not superintelligence. It's questionable whether they're even AI. And I think that the conversation is being used. There are some people who are doing it in good faith and others in others. I don't think it's being used to discuss the actual harms of what they are calling AI today are. And it's all of the discussion around the larger concerns really feels overwhelmingly about something that's not happening. It's not even like they're discussing, okay, here's a legal definition of superintelligence.

• 5:24Speaker 04

  Here is a thing of what AGI means, and this is the actual plans we're going to make for if this happens on a welfare level. Like, are we going to do UBI? It's always about, yeah, it's really scary, but only the big, sexy, rich companies are the ones that can possibly deal with it.

• 5:40Steven Bartlett?

  Let me just frame the question so I can get a percentage from you, or not. The percentage might be zero. But do you think the course we're on now, in the way that they're pursuing superintelligence, will lead to a percentage chance of human extinction? And if so, what is that percent?

• 5:55Speaker 04

  So are we talking strictly AI-based? Because if we dot the world with data centers, we have a climate disaster that's coming for us, which will actually potentially eradicate humanity. But if we're talking strictly about AI, I stand at zero, because we have not defined superintelligence. I don't think LLMs are the path to it, and I don't think I see it happening.

• 6:14Steven Bartlett?

  Okay, so we've got 99%, 0%, Andy?

• 6:17Speaker 01

  I put a tilde in front of my 0% because never say never, but rounding error 0%. And I think this discussion is a massive distraction from the more substantive conversations, the more important conversations we should be having about AI. And I'll say it again, it distracts us from the good things that AI is doing, will be doing for us. I get this impression sometimes from parts of the AI community that this is a massive evil or a terrible thing that has been unleashed on the world. Unless we listen to the advice of some people who have spent a lot of time thinking about this. I get the impression from a lot of the discussion

• 7:02Speaker 01

  that the underlying view is we would be better off had AI never been invented. I vehemently reject that view. I think we have a long history of inventing very powerful technologies that bring risks and harms along with them. And we humans have done a really good job, and you know, not perfectly and not immediately, but muddling through the situation and winding up in a better place because of the new technologies that we have. I expect AI, let me finish, please, I expect AI will be the next chapter in that story. And to say that it's this massive discontinuity and will kill us all, I think it's a huge disservice.

• 7:41Steven Bartlett?

  Before we get into opinions, I'm just going to introduce you all. I'm going to let you just give two or three sentences to introduce who you are, where you come from, and the experience you've had that's fed into the opinions that you have, starting with yourself, Nate.

• 7:53Speaker 00

  Yeah, I'm Nate Soares. I am the president of the Machine Intelligence Research Institute, which was one of the very first organizations working on AI alignment, trying to make AI care about humans, try and make AI go well. I'm also the co-author of the New York Times bestseller, "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All", which maybe gives you a sense of how well I think AI alignment is going.

• 8:19Steven Bartlett?

  And how long have you been working in AI, Nate?

• 8:21Speaker 00

  I joined Miri in 2014.

• 8:24Steven Bartlett?

  Before it was cool.

• 8:26Speaker 00

  Well, before it was cool.

• 8:27Steven Bartlett?

  Roman?

• 8:28Speaker 03

  Roman Janpolski. I'm a professor of computer science and engineering. I coined the term AI safety about 2011. I'd been doing work on related topics before that. I wrote multiple books. You have a pile there. Some of those are mine. I think this is the most important problem we'll ever face.

• 8:52Steven Bartlett?

  Ed?

• 8:53Speaker 04

  I'm Ed Zidron. I am the CEO of Easy Primary Research, a principal analyst there, researching due diligence. I write the Where's Your Red App newsletter, read by 117,000 people, better offline podcast, listens to about 1 million people a month. And yeah, I am one of the foremost AI critics. As many people inform me via email, they disagree. But I think that this situation is being used to get away from the actual harms, and I will press as hard as I can to make sure they're heard.

• 9:20Steven Bartlett?

  And you tend to say that AI is overhyped.

• 9:22Speaker 04

  Yes.

• 9:23Steven Bartlett?

  Okay.

• 9:24Speaker 01

  My name's Andy McAfee. I am a principal research scientist at MIT, where I co-founded the Initiative on the Digital Economy. I'm the co-founder of the AI Startup Work Helix, and I've written a couple of books, one of which was The Second Machine Age, which came out in 2014 and was also a New York Times bestseller.

• 9:42Steven Bartlett?

  Thank you. Nate, make your case. What's your perspective?

• 9:47Speaker 00

  You know, I think whether or not the issues of extinction are a distraction from the possible benefits or from some of the present harms, I think that comes down to whether there is a real extinction risk. A lot of people like to say, hey, it's distracting from this, it's distracting from that. My basic case is it could be true that there's a lot of benefits to AI. It could be true that there's a lot of present harms to AI. Neither of those would rule out that AI has a chance of wiping out all humanity, a substantial chance, bigger than this 0 with a tilde in front of it. And the way I would approach things is to try and figure that out, because it's pretty important to our civilization.

• 10:24Speaker 04

  How do you define AI in this case?

• 10:27Speaker 00

  You know, I think a fascination with definitions isn't the most helpful. I think if we're sort of like in a forest fire, and we can see the fire starting to spread and it's starting to surround us, and I'm like, hey, we should run. And you're like, well, what really is fire?

• 10:43Speaker 01

  But that's not...

• 10:44Speaker 00

  How do we define fire?

• 10:46Steven Bartlett?

  What are you telling us to run from? You know, with fire, I get burnt, and I understand the mechanism in which I die. So what is it you're saying that we should be running from?

• 10:54Speaker 01

  Also, if we accept your fire analogy, we've basically accepted your argument. I don't accept that we're in the middle of a fire, a forest fire right now.

• 11:01Speaker 00

  I'm very happy to...

• 11:02Speaker 01

  You're breaking the premise into your refusal to give a definition.

• 11:06Speaker 00

  Oh, I mean, I can give some definitions. I just think that we shouldn't get wrapped up in the definitions.

• 11:10Speaker 01

  Okay.

• 11:11Speaker 00

  So, you know, in my book, we define superintelligence as AIs that are better than the best human at every cognitive task, every mental task. So anything you can do in your head,

• 11:20Speaker 01

  Right.

• 11:21Speaker 00

  the AI can do that better, and anything the best human can do in their head, the AI can do that better.

• 11:26Speaker 01

  Correct.

• 11:26Speaker 00

  Now, once you've defined it that way, that does not mean that the only possible worry is superintelligence. You could have an AI that's better at some things and worse at others, and that is still very dangerous. And so once we pick a definition of what does superintelligence mean, now, you know, if you're like, well, this isn't technically a superintelligence, so it can't hurt us. I'm like, no, no, that was just a definition. The definitions...

• 11:47Steven Bartlett?

  So I want to just, on this line of questioning, what is the mechanism in which extinction could become a high probability or even a 1% probability?

• 11:55Speaker 00

  Yeah, the thing I'm worried about here is AIs that are much smarter. I think there's a lot of questions about whether LLMs can get much smarter. There's sort of one conversation about like, how could AIs get smart to the point that they kill us? There's another question, which is how could they kill us once they're smart?

• 12:11Speaker 00

  It's much easier to predict that they would succeed against humanity in a conflict that they would win in a fight than it is to predict exactly how. Like, if you were playing a chess match against Magnus Carlsen, I would know who's winning that chess match. No offense, Magnus Carlsen's the best human chess player. I just know who's going to win. If you were like, okay, what piece is he going to use to checkmate me? I'm like, gosh, that's a much harder question. I can make up a story. You know, and some made-up stories are like, it makes a super virus. It takes over robot factories that are producing robots that are producing more robot factories. It uses a website that already exists today called rentahuman.ai, where it rents humans to do things for it.

• 12:52Speaker 00

  There's sort of all sorts of ways for AIs in the digital world to affect the material world if they are trying to. And there's sort of a lot of questions to tease apart here. There's like, why would AIs be trying to do that? And there's how smart could they get in using these bio labs, paying people to do things, taking over robot factories? And how far off are we from AIs that start doing that stuff? Bunch of questions that we can go into.

• 13:16Steven Bartlett?

  I'm always curious as to why someone was working in AI slash AI safety more than 10 years ago, before there was any sign that it would be a, you know, I mean, there was evidence, but it wasn't a pertinent technology at the time. Were you working in AI safety then?

• 13:31Speaker 00

  I was.

• 13:31Steven Bartlett?

  Why?

• 13:32Speaker 00

  Everything we see around us in this whole image was designed by humans. The world is shaped by humans because we are the smartest creature around. If we make stuff that is smarter than us, then the world's going to be shaped by them. And so it's very important that they be shaping the world in a good way.

• 13:54Speaker 00

  I was at Google in 2012 when they bought Google DeepMind, which was able to play a lot of Atari games with one single program.

• 14:01Steven Bartlett?

  Which was an AI company.

• 14:02Speaker 00

  Yeah, so I was there when we had these AI companies that were able to write one program that could play many video games. And that got me thinking about like, where does it go? And back then I could see that the progress was increasing and that, you know, back then I hoped we had decades. But I could see it was easier for these companies to make the AIs smart than to figure out how to make the AIs good. So I was like, someone needs to be on the side of figuring out how to make the AIs good.

• 14:28Steven Bartlett?

  Roman, make your case.

• 14:30Speaker 03

  I want to agree with you on something you said, but I'll define AI, and that will help us. We use the term AI to mean three different technologies, completely unrelated, and that's what probably creates this debate. AI as a useful tool, as a standard technology we always had. Nero system makes you more productive, more creative. Everyone loves it, supports it. I'm a computer scientist. I'm an engineer. I want more of it. It helps the economy. It's great. We know how to control them, how to make them safe. We understand what they do, completely on board with that AI. AI, we're starting to have now, GPT-6 level, human level, AGI level. We can argue about what that means.

• 15:12Speaker 03

  Some dangers, like any human, they're unsafe, like a human would be unsafe. But if we introduce them into the research cycle, they are automated scientist, automated engineer.

• 15:23Steven Bartlett?

  What do you mean by that, introducing them into the research cycle?

• 15:25Speaker 03

  So right now you have humans doing research to make GPT-7.

• 15:29Steven Bartlett?

  Yeah.

• 15:29Speaker 03

  But we're starting to add AI tools. More programming is done by AI, design of the next parameter set. What if the whole process is fully automated? What if GPT-6 is writing GPT-7?

• 15:42Steven Bartlett?

  Is this what they call recursive self-improvement?

• 15:44Speaker 04

  Which is not a foregone conclusion, though.

• 15:47Speaker 03

  A lot of people are predicting, including all the top labs, that they will get there. They're introducing junior machine learning researcher in 2026. They want the cycle to start in 2027.

• 15:57Steven Bartlett?

  Which is when the AI will start building the new AIs itself.

• 16:01Speaker 03

  Right. Once that cycle starts, we're going to create something called superintelligence, a system smarter than all of us at everything or capable of learning to be in any new domain. We will become secondary species on this planet. We will not be in charge. We will not decide what happens to us. Superintelligence doesn't hate you. It just doesn't care about you. We didn't learn how to make it care about us. And if it decides to, I don't know, cool the planet to make compute more efficient, it will freeze us. If it wants to convert this planet to fuel to fly to Mars, so be it. We have not learned how to control those systems. The capabilities are getting exponentially better. Our ability to control those systems is non-existent. We have filters and we have bans.

• 16:46Speaker 03

  We put guardrails of, don't say that word, don't talk about this topic. And that happens after the fact, after the model already made the decision. Sometimes you see it scraping the result.

• 16:56Steven Bartlett?

  So they build the model, and then they put filters around it to make sure it doesn't offend any...

• 17:00Speaker 03

  Exactly. We cannot have it say he and work on the air. Like, we need to make sure that never happens, that will kill the prophets. So that's all they have, guardrails of that nature. The model itself is completely unaligned, doesn't care about you. It's wild that we're developing this and not just developing it before we deploy it through economy, before we get benefits of having GPT-6 propagated through economy. It can do so much. There are trillions of dollars of value in that model alone. We forget that. We switch to making the next model as soon as we can.

• 17:32Steven Bartlett?

  Roman, I've just got a follow-up question for you there. It would appear to me that the new ChatGPT-6 model, the Fable 5.1 model, is arguably smarter than 99.999% of humans on planet Earth already. Is it conceivable that a intelligence that is much, much smarter than humans, is there any case where it could be controlled by humans? Does form factor matter? Does the fact that it doesn't have limbs and legs, and does that matter at all?

• 17:58Speaker 03

  I think long-term, control of something that much smarter than us is impossible. It can be, for reasons we don't yet know, friendly to us and decide to keep us around and make us happy. But it's not a guarantee.

• 18:13Speaker 01

  Let me pick up on Steve's question, because I like the phrasing a lot. Let's say that Fable or whatever the latest release from OpenAI is really is smarter than, I don't know if it's 95 or 99% of the people. Are we only being saved from extinction by the 1% who are still smarter than the AI?

• 18:31Speaker 03

  No, no, the concern is not the model we have today. The concern is what I said.

• 18:36Speaker 01

  But if I believe your argument, then we really should be concerned about the model.

• 18:39Speaker 03

  No, it's like having another human. If there was another smart human, there is Einstein today, and he's malevolent. I'm not worried. He may cause some damage, but he's not going to exterminate 8 billion people. We are competitive at this stage. There are people just as smart who can understand what happened with the recent hacking accident and do something about it. My concern is that in a year, we're going to have a model that's so much smarter. It's like squirrels fighting humans. They don't understand what we can do to them. They have no concept of poison, stripes, guns in their world model. They think you're going to chase them up a tree and bite them really hard.

• 19:13Steven Bartlett?

  Is that also why recursive self-improvement was central to your argument? Because at some point, if it starts improving itself, then it's kind of like a runaway train of intelligence.

• 19:21Speaker 03

  It's an intelligence explosion. We don't control it. We don't understand it. We can't monitor it. We can't explain it. We can't predict it. At that point, it's just a runaway process.

• 19:29Steven Bartlett?

  I've heard this phrase from Sam Altman and the others called fast takeoff?

• 19:33Speaker 03

  Yes.

• 19:33Steven Bartlett?

  Is this what they're describing?

• 19:34Speaker 03

  That is the debate.

• 19:35Steven Bartlett?

  What is it?

• 19:36Speaker 03

  Some people think it's going to take a very long time. Yeah, we're automated research, but it's still going to take years. We need to run physical experiments. And fast takeoff means, as I said, instead of a year, it's going to take a month, a week, a day, a second. Because you're not having humans doing research. You have, let's say, 10,000 agents, each one smarter than all of us, doing research 24/7. They don't sleep. They don't eat. They don't get sick. They are much faster than us.

• 20:04Steven Bartlett?

  Ed, your face tells a picture. It's a, I think I could say you disagree.

• 20:09Speaker 04

  We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening. And I find that very frustrating because the people that are killing themselves are a problem. The black neighborhoods being poisoned with gas turbines, that is a problem.

• 20:23Speaker 03

  You said you cared about climate change.

• 20:25Speaker 04

  Yes, yes, yes.

• 20:26Speaker 03

  Right. So imagine a guy who goes, it's raining right now. We need umbrellas. We need to do something about it. This is like weather related. And completely ignoring climate change, the planet will boil over.

• 20:37Speaker 00

  Yeah, we can worry about both.

• 20:38Speaker 04

  Okay, that's great. Why are we not talking about the thing that actually happened though? Like, Chad.

• 20:41Speaker 03

  Because relatively, it's not important.

• 20:43Speaker 04

  You don't think someone killing themselves.

• 20:45Speaker 03

  No, it's one person. We have eight billion people. We're running unethical experiments on.

• 20:48Speaker 04

  You don't think anyone else is being given that AI psycho? Why do you not think?

• 20:51Speaker 03

  Six people, 10 people, those numbers are insignificant and close to zero.

• 20:54Speaker 04

  Tell that to their families. I'm sorry, you have a software that's out there.

• 20:58Speaker 03

  Do you understand eight billion people and all future generations versus like literally a guy with a name?

• 21:03Speaker 04

  You're doing thought experiment about maybe harm. Jacob Coxon, guys on TV saying it can copy itself to this, that and the other.

• 21:09Steven Bartlett?

  Jacob Coxon is the.

• 21:10Speaker 04

  The guy from, from Anthropic who said he was quitting because he was so scared of everything despite spending years at OpenAI. And having tons of stock, I believe from there. So good for him. The thing he was saying was describing theoreticals all while divorcing the harms, which I think we can agree with, that the companies themselves are not taking this seriously enough. But always it was about the AIs too powerful and mystical, not OpenAI and Anthropic. The two largest startups are using hundreds of billions of dollars of infrastructure to hack. A regular person doing this would be arrested.

• 21:40Steven Bartlett?

  They're saying eight billion people are going to die and it's not just them. I have this long list of quotes here from the people building this technology who appear to agree. If you look at some of these quotes from, from Elon Musk. Who said, with artificial intelligence, we are summoning a demon. You know, all those stories where there's the guy with the pentagram and the holy water and he's like, yeah, he's sure he can control the demon, but it doesn't work out.

• 22:05Speaker 00

  So one thing I'd say is, you know, I, I really wish that the world would only give us one problem at a time.

• 22:10Speaker 04

  Sure.

• 22:10Speaker 00

  And if the world did give us only one problem at a time, I would love mine to be last on the list. It looks to me like we can have multiple problems at once. I, I think there are current harms. I think we should address them. It looks to me, I do talk to policymakers sometimes, it looks to me like there's a little bit more movement on the regulatory side about some of the current harms. There's, you know, child safety protection acts. There's, you know, anti-deep fake acts. We have more of those making more headway in Congress or getting passed through Congress than we have. Sort of trying to make it so we don't have any of these extinction risks. The other thing I'd throw out there is that. I agree, we, we should deal with the current harms, but if you watch the people saying, deal with the current harms over time.

• 22:50Speaker 00

  A couple of years ago, they were saying, we have to deal with current harms like AI bias influencing who's hired. Last year, they were saying, we have to deal with current harms like kids killing themselves. This year, Gary Tan, just on an interview the other day.

• 23:05Steven Bartlett?

  Who's Gary Tan?

• 23:06Speaker 00

  Sorry, Gary Tan is a technologist who runs Y Combinator. Which Sam Altman used to run before going to OpenAI. And on an interview the other day, he said.

• 23:17Speaker 00

  Let's not worry about these crazy future risks. We need to worry about current harms like AI swarms breaking out and taking over data centers. And I'm like, look, guys, at some point. We need to look at the progression of like the current harms that we, that everyone is saying we have to worry about instead of the, the, the extinction threats. And watch where the puck is going. Play where the puck is going. And I'm like, these extinction threats are coming down the line. They aren't in opposition with dealing with the, the problems we have today. We just need to deal with both.

• 23:45Speaker 04

  But we're not dealing with the ones today though.

• 23:47Speaker 00

  We should deal with them both.

• 23:48Speaker 04

  Okay, good.

• 23:48Speaker 01

  Andy.

• 23:51Speaker 01

  As I've tried to understand the alignment argument and the, the extinction risk argument, a couple of things keep popping out to me. Number one, it seems to rely on thresholds. Once we hit recursive self-improvement, once we hit AGI, then it's game over for us. I don't love those threshold arguments. They're fairly poorly defined. And there's a, and, and there's a huge assumption on the other side of them. We hit this point and then all of humanity goes away. That, that is a gigantic claim.

• 24:21Speaker 04

  I'm happy to reckon that.

• 24:23Speaker 01

  Let me finish, please. On its face, that is a gigantic claim. I also think there's a lack of humility in your community. We are working on humanity's most important problem. And based on the thinking that we've been doing, we can't see a way that we're wrong. In other words, as soon as we get to these thresholds, bam, that's game over. I, I find that very far from a humble approach, especially given that. We have no large base of evidence to base any of this on. I agree with you guys. AI is new. And the fact that AI is so these days is agentic. It goes off and does long chains of things on its own.

• 25:05Speaker 01

  After we give it some very, very vague, very short initial instructions. Holy Toledo, it will, it will spawn up a storm of agents and they will go off and. Kind of do their own thing and they will, they will grind. They will, they will spawn lots of them. They will work for a long time. They will exhaust every possibility. With the experience I have with agentic AI, I'm just amazed at the tenacity and the doggedness of these things. And we saw a super clear example of that with this most recent.

• 25:36Speaker 01

  Jailbreak, this, this attack that wound up at the website Hugging Face. And I'm going to try to summarize the, the step by step of that. And I think you all three probably know this in more detail than I do. But let me step through what I think is the sequence of events. And unless I get it dead flat wrong, like, you know, let, let me keep going. So a team at OpenAI set up a sandbox, an allegedly protected, secure environment in the cloud. Where they told a bunch of agents to go try to. Exploit security vulnerabilities.

• 26:07Speaker 00

  That's dead wrong, sorry.

• 26:08Speaker 04

  One important, yeah.

• 26:10Speaker 00

  What they did is they had thousands of agents. Each individual agent was given a task of use this vulnerability to break this particular piece of software.

• 26:20Speaker 01

  I want to finish my TikTok. So a couple really, really interesting things has happened. First of all, these agents escaped the sandbox that OpenAI thought they were going to be contained in. And they got, the OpenAI tried very well. They, they set up an environment so that these agents could not access the big, broad, public internet. And guess what? They accessed the big, broad, public internet via a very clever series of things. That they strung together to get out there. And then once they got out there, they went to a website called Hugging Face and used that. They took over part of the Hugging Face infrastructure and started doing more things, the details of which I forget. That's pretty wild, right? Like, I grant you.

• 27:01Speaker 00

  It's even more wild than that, but yeah.

• 27:03Speaker 01

  That is really, it's impressive and it is a little bit unsettling at least, right?

• 27:09Speaker 00

  Absolutely.

• 27:10Speaker 01

  Now, let's talk about what were the results of that were. OpenAI was not super vigilant about the environment that they set up apparently.

• 27:18Speaker 00

  Certainly.

• 27:18Speaker 01

  Because the agents were kind of going off the end of the world, starting in May or something of this year.

• 27:23Speaker 00

  Yeah, yeah.

• 27:24Speaker 01

  And OpenAI was not, was not aware of that. It, as I understand it.

• 27:28Speaker 00

  They actually broke out once and crashed OpenAI's servers internally. And then OpenAI didn't notice what was happening still. Hatched the holes that they used to get out the first time, started them running again, and then they came out a second time. There was actually, I think, three swarms, although we don't actually, yeah.

• 27:42Speaker 01

  That's the worst story I have.

• 27:43Speaker 00

  So far.

• 27:45Speaker 01

  Thank you.

• 27:46Speaker 00

  Look at the trends.

• 27:47Speaker 01

  Let me finish, please. This is my last sentence. From there to this kills everybody. I find that a really, really long, very uncertain journey. And I have no confidence that we wind up here. It feels like you two find that a very straight, narrow path. And I, I think that's an important difference. That's my point. Do you want to respond to that?

• 28:06Speaker 00

  I would, I would be happy to get into it. I don't know if we're going to have the time to go deep. A, a couple of points to throw out. Oh man, I just really want to say some of the crazier things that happened in the Hugging Face swarm if we want it later. A lot of people thought that these AIs were breaking into Hugging Face in attempts to steal answers to their test. That's what we thought originally. Turns out that's not true. It turns out that these AIs immediately were able to solve their problems by cheating. And they were breaking out in order to cover their tracks. They were uncertain how to delete the log files and hide their cheating from the process that was going to score them.

• 28:40Steven Bartlett?

  So just to clarify for a simpleton like me, they were all given effectively a test to do. They did the test straight away, but they cheated. So they were breaking out to figure out how to cover the fact that they cheated.

• 28:51Speaker 00

  That's right. So it's like, it's like you're telling, it's like you have a bunch of students in separate rooms. And you're like, use these lockpicks to break into this lock. And there's like a thing behind the lock. There's like a, a secret code behind the lock to show me that you succeeded. And what they, what they do is they break it with a hammer. Get the thing out and they're like, oh no, I wasn't supposed to do that. So then they use the lockpicks to break out of the door. They meet up with a thousand other people. They start calling themselves a swarm and they go to break into the administrator's office to see if they can delete the camera footage. And they don't find the camera footage there. This is the swarm like breaking into OpenAI. They don't find the camera footage there. So they break out the window of the school, hotwire a car. Drive to the therapist's office to try and read through the therapist's files to figure out where is the teacher going to keep the, the security footage? And at that point they're caught.

• 29:37Speaker 00

  And you're like, oh, uh, like, what did you expect? You were giving them a lockpicking exam. It's like, well, I sure as heck didn't expect this. You know, totally crazy.

• 29:45Speaker 04

  Can I, I have a weirdly between both of your opinion, which is. Everything you're saying is correct, but you keep anthropomorphizing software. And I, to be clear, what you're describing is.

• 29:56Speaker 00

  It's just the facts that happened. Yeah.

• 29:58Speaker 04

  Sure, but you're missing out on important detail, which is the hundreds of billions of dollars in infrastructure provided by Microsoft, Google, Amazon and Oracle. To be clear, the harms are very similar. We're not disagreeing on that, but I think it's important to know that this was a function of. Where it was making decisions was it was checking on a decision tree based on the harness, based on the training data.

• 30:18Speaker 00

  Oh, it's not a decision tree.

• 30:19Speaker 04

  It's not a decision tree, I know, but it's an alignment issue still.

• 30:22Speaker 00

  Absolutely agree.

• 30:23Speaker 04

  So what's your point? But this is, these aren't conscious beings. They are acting in ways that have real outcomes. But they are a function of the alignment problems that we'd actually agree on.

• 30:33Speaker 03

  Intelligence is a spectrum. Projected next five years forward, where are we going to be? So I think a model like that would be dangerous in ways you are not seeing.

• 30:43Speaker 01

  There will absolutely be risks and weird stuff happening in ways that I can't see right now. What, what I'm quite confident, and I think this is where you and I probably part, where the two of you and I part. Is our ability to control these things.

• 30:57Speaker 03

  So I actually tried proving what is possible and what is not possible in that space. The impossibility results, published in peer reviewed papers, well cited. We cannot control something smarter than us. We cannot explain it. We cannot predict it. It's not a question of getting more money for those companies, more time, smarter humans. It's just not a possibility. If we create general super intelligence, we are fried.

• 31:20Steven Bartlett?

  Andy, how do we control something smarter than ourselves? Because that's the base premise that you're sort of asserting there.

• 31:26Speaker 01

  These.

• 31:29Speaker 01

  Agents that broke out are smarter than 99 ish percent of the security researchers in the world. They were not caught by the point one percent or the one percent. They were caught by some dude at Hugging Face, maybe, I'm sorry, a person at Hugging Face. Looking through their log files and finding an anomaly. That's some, you know, hopefully pretty well qualified person. Noticing something was wrong and having pretty easy ways to unplug, disconnect from the internet, wipe it clean, do whatever. That's the skill that's available to like, I don't know, the 75th percent most intelligent. Security employee at Hugging Face. The idea that the IQ points are what separate us from extinction does not even, doesn't hold up. It doesn't help me understand what happened in this example.

• 32:13Speaker 01

  Where we had very, very smart agents being turned off and cleansed by probably less smart people.

• 32:19Speaker 04

  That does actually make me think of something. So that is an IT observability problem. It's being able to see what's happening with your infrastructure. And I think that there is actually, I think you would agree with this. There is a serious problem with these companies that we do not know. And it doesn't seem they know what's going on with their compute. It's like a chimp with a gun. These people have access to all this infrastructure and they're running. We don't know how much money they spent on the Hugging Face exploit. Because it is relevant because it's, how much could a threat actor use to recreate this? Because conscious or not, it is very dangerous. But it's, AI is in the dangerous hands. It's an OpenAI and Anthropic's. We have a problem with that. Conscious or not, however we may think it goes, I think we have a real and present thing where.

• 33:00Speaker 04

  We have these companies working willy nilly, just running experiments that are potentially very dangerous. We do, I really think we need a government regulatory body. Whether or not we get to the things you are discussing, I think we have a clear and present danger today. These things are however, not intelligent in the same way humans are. This isn't an argument about AI being able to do stuff. It's, we need to build different. Infrastructure or different regulatory infrastructure to deal with what LLMs can and can't do. And I think that starts with a realistic discussion of what happened. It was a poorly run security environment. It was clearly, there's something going on with alignment. It was an unreleased model, right?

• 33:37Speaker 00

  Unreleased model.

• 33:37Speaker 04

  So we have no idea what it was trained like. We don't really have, we as people should at very least have clarity into our alignment is going. We, the idea of.

• 33:46Steven Bartlett?

  You sound like these guys.

• 33:48Speaker 04

  Here's the thing.

• 33:49Speaker 00

  Everyone's converging on us with time. Yeah.

• 33:51Speaker 04

  Here's the thing. I may not agree with a large chunk of what they say, but we agree that these companies are acting recklessly.

• 33:57Speaker 00

  Absolutely.

• 33:57Steven Bartlett?

  Andy, two questions for you then. Do you agree with the statement that AI is going to get increasingly more intelligent?

• 34:04Speaker 01

  It's going to get more capable.

• 34:06Steven Bartlett?

  Okay, capable intelligence, fine.

• 34:08Speaker 01

  And I'm going to use my word.

• 34:09Steven Bartlett?

  Okay.

• 34:10Speaker 01

  It's going to get more capable.

• 34:11Steven Bartlett?

  It's going to get increasingly more capable.

• 34:12Speaker 01

  Yeah.

• 34:12Steven Bartlett?

  And is capability a function of intelligence?

• 34:19Speaker 01

  Will it be able to beat us on most IQ tests?

• 34:22Steven Bartlett?

  Fine.

• 34:22Speaker 01

  I guess.

• 34:23Steven Bartlett?

  Fine. And then so is it, if, if that, if that looks like an exponential curve, i.e. it's, you know, it's increasing upwards to the right like a hockey stick. Can, how can you convince me that we can control something?

• 34:34Speaker 01

  I just tried to convince you. I tell you that there are less intelligent people than the agents who turned off the agents in the OpenAI Hugging Face exploit. I'm pretty comfortable. I mean, no disrespect.

• 34:45Speaker 03

  What is the cognitive gap between them right now? Between the model?

• 34:49Speaker 01

  I have no earthly idea, but I.

• 34:50Speaker 03

  Guesstimate.

• 34:51Speaker 01

  No, because I think as these, as these systems get more capable. We will still be able to at some level figure out when they're doing things that we don't want and turn them off.

• 35:01Speaker 03

  No matter how much smarter they are.

• 35:02Speaker 01

  Right. And you think there's some threshold at which they become nefarious and self-protective enough that, that they turn off our ability to turn them off? Man, that, that, man, that's a big reach. That is really, that is purely speculative.

• 35:16Speaker 03

  Students who can understand your material, right? You're not going to get someone with IQ of 80 to take quantum physics course. They're not going to get it. So you know importance of intelligence to understand actual problems?

• 35:29Speaker 00

  Yeah, I, I totally agree. We can turn it off. And that's a huge advantage. One of the issues is that as the AIs get smarter, they realize this. The, the Hugging Face AIs were trying to delete, or the, the, the OpenAI swarm. The, the swarm of agents from OpenAI that went out to hack. They were trying to delete log files.

• 35:49Speaker 01

  Did they, did they try to program a Roomba to go unplug the computer that was monitoring them? Like, did, did they harness robots to go protect the perimeter of the?

• 35:57Speaker 00

  Future ones could.

• 35:58Speaker 01

  Could.

• 35:59Speaker 00

  Yeah, these.

• 35:59Speaker 01

  Could.

• 36:00Speaker 00

  Yeah.

• 36:01Steven Bartlett?

  Let him finish, let him finish.

• 36:02Speaker 01

  Rampant speculation. This is a chain of things that could happen. And therefore, there's like a 20% risk we're all going to die. Man, that, that does not hold for me.

• 36:10Speaker 00

  When I was writing my book, the AIs weren't really agentic yet. The, the drafting process happened mostly before what we call the reasoning models. Which are trained not just to predict humans, but to solve a, a long number of problems. Or a huge number of hard problems. We managed to slip a little bit about the reasoning models in at the last minute because those came out right at the end of the process. And at the time, a lot of people said, AI will never be agentic. That's why we'll be safe. And in chapter three of my book, we go over how AI is going to become agentic. How it's going to become tenacious, how it's going to become dogged. And that's what we might call an advanced scientific prediction. That has paid off in the Hugging Face attack.

• 36:52Speaker 00

  A lot of people in the industry were like, I didn't believe this stuff. Until I saw the AIs sort of doing things they weren't instructed to do. Despite us trying to get them to stop. And so there are theories here that do make advanced predictions. The, the way that the scientific method usually works is that we don't have any certainty about the future. But we absolutely have ways to test this stuff. Now, I could, I could go into more about how could they kill us? How could an AI that knows we would shut it down? Lie low until it has access to its own infrastructure? We did already see the Hugging Face AIs try to delete logs to cover their tracks.

• 37:34Speaker 00

  But fortunately for us, those AIs were not trying to hide from the humans. They were trying to hide from the automated grading process. Will the next swarm try to hide from the humans? Will the next swarm be able to succeed?

• 37:48Speaker 03

  It's more than that. They didn't know for four months that this was happening. What is it we don't know today?

• 37:54Steven Bartlett?

  Just, just to clarify what Nate said there in his book that I have here. If anyone builds it, everyone dies. He does say in chapter three. Once AIs get sufficiently smart, they'll start acting like they have preferences, like they want things. We're not saying that AIs will be filled with human-like passions. We're saying they'll behave like they want things. They'll tenaciously steer the world towards their destinations, defeating obstacles in their way. Which sounds a little bit like the Hugging Face instance.

• 38:21Speaker 01

  Steering the world is very different than, than, than.

• 38:26Speaker 00

  We go over what we mean by steering the world easier earlier and it's really getting anything to, like. We'd have to get more quotes to get what we mean by steering the world, but yeah. By steering the world, we mean steering any part of the world.

• 38:36Speaker 04

  But it feels like there's a fundamental difference between acting within tech. To be clear. Going to say it again. The outcome would be the same. But I think that there is a big difference when it's. We are dealing with something that's large language model and the harness and agents, so LLMs, completing a task based on training and alignment. That is a very different conversation to saying this thing is conscious and has its own intentions and acts on its own accord.

• 39:01Speaker 00

  Conscious doesn't come into it.

• 39:03Steven Bartlett?

  No one said.

• 39:03Speaker 04

  A lot of, a lot of people, here's the thing.

• 39:06Speaker 03

  They're not sure.

• 39:06Speaker 04

  As a result, as a result of partially the logic, the rationale. That you yourself have, like you have been a part of spreading. I'm not saying, not saying anything about your intentions. I'm just saying, the conversation is kind of, is kind of what's happening with Jacob Coxon from Anthropic is a result of this escaping containment.

• 39:22Speaker 03

  You said the outcomes will be the same. What do I care? How does it feel on the inside if the thing is going to take us out?

• 39:29Speaker 04

  The thing is, okay, actually, that's, that's actually a very good question. I think it actually come, excuse me, let me.

• 39:34Speaker 03

  I ask great questions.

• 39:35Speaker 04

  Yeah, you're shrugging at me like.

• 39:36Speaker 03

  No, no, I'm saying we ask good questions.

• 39:38Speaker 04

  Now, here's the thing. If it's, these things are, have their own minds and consciousness. You have to deal with outthinking something versus something that is doggedly trying to commit to a purpose. And complete a task based on training and alignment, which is a result of infrastructure. We really need regulations and actual, actual regulations around any kind of AI. We don't, we don't really have regulations of tech.

• 40:02Speaker 00

  I, I actually am not really a big like, look at the straight lines and the graph guy. You know, maybe, maybe to my detriment in some ways, there are people who predicted the current tech better than me. About like when certain things would happen. For a long time I have said, I think we can predict what will happen eventually. And, and this is again, it's like the chess game. I can predict that Magnus Carlsen is going to beat you in the chess game eventually. He's the best human chess player alive. It's sometimes easier to predict where things end up than it is to predict how they get there. And, you know, what I hear you as saying is like. Right now, we have these like huge companies spending huge amounts of money on intelligence that's maybe not quite the real deal. And we don't have a good reason to think it's going to keep going.

• 40:44Speaker 00

  I really hope it doesn't keep going.

• 40:46Speaker 04

  Okay.

• 40:47Speaker 00

  I have been in this business since before the LLMs. I am not here saying like, oh, these large language models, these chatbots, they're going to be the ones that are going to kill us. I've been here saying, look, I know where this story ends if we don't change things. I have been really hoping that the LLMs will run out of steam and they keep on not running out of steam. And then we have, you know, the, the AIs like breaking out and committing cyber crimes like against instructions. And, you know, the people have said, we don't need to worry about those like weird future dangers. We just need to worry about the current ones to have like. More and more sci-fi sounding current ones. And I'm like, man, I don't think we should bet civilization on the LLMs running out of steam. But I like hope and pray they run out of steam.

• 41:25Steven Bartlett?

  You really hope they run out of steam?

• 41:26Speaker 00

  Absolutely. But one thing to watch out for is that even if the LLMs run out of steam, there's a question of did they run out of steam at a point where they can do automated AI research and find some other architecture that's better than LLMs?

• 41:40Steven Bartlett?

  As in when they realize a better way to improve their intelligence?

• 41:43Speaker 00

  That's right.

• 41:44Steven Bartlett?

  A cheaper, maybe more efficient way.

• 41:45Speaker 04

  Why are you not trying to slow down the companies?

• 41:47Speaker 00

  I absolutely am trying to slow down the companies.

• 41:49Speaker 04

  How would you suggest we slow them down?

• 41:52Speaker 00

  I suggest we stop them all. I think that this whole area of research is just crazy dangerous. Like, it is not worth the risk to civilization. I think it would be fine to like back up to the sort of AIs that are public today, which are not the ones that are swarming, and be like, okay, you know, we're gonna like, keep the current chatbots that we have available, we're gonna figure out how to integrate them into our economy, we're gonna figure out how to make them like deal with the education system.

• 42:18Speaker 04

  Some kind of compute limit, maybe.

• 42:19Speaker 00

  Compute limit, maybe.

• 42:20Speaker 04

  Yeah, yeah, yeah.

• 42:20Speaker 00

  And like, I've been advocating for this for a long time. A lot of people look at me like I'm crazy. And I'm like, look, we really are dealing with an extinction threat thing. We don't know where the lines are.

• 42:30Speaker 04

  So just to be clear, so I understand, so I'm fair. You are not saying LLMs are the thing that will do the superintelligence. You are saying it's showing signs, because that's actually, I think, an important distinction.

• 42:39Speaker 00

  That's right.

• 42:40Speaker 04

  Okay, I think that's actually a pretty fair perspective. My thing is, is the reason I push back on any kind of anthropomorphization is we cannot remove the humans who are responsible for the bad stuff that's happening. And I think paying very clear attention and where possible, I understand we're describing this stuff, you kind of have to use language that's human, I get that. The reason I so push for like, it's not a foregone conclusion, these are companies doing this, this is software, is because I feel like in the overall, not saying you, overall, superintelligence discussion, we in society ignore and empower the anthropics and the open AIs of the world.

• 43:24Speaker 00

  Absolutely.

• 43:24Speaker 04

  And in turn, allow them to do dangerous experiments. And I think that-

• 43:28Speaker 03

  You want to argue that CEOs of those companies should go to prison for this hacking incident?

• 43:33Speaker 04

  Sure.

• 43:33Speaker 03

  Which is a crime?

• 43:34Speaker 04

  Yeah.

• 43:34Speaker 00

  I'll support you.

• 43:35Speaker 04

  Absolutely. Let's jab at Sam Altman and Dario Amadei.

• 43:38Steven Bartlett?

  When I look at-

• 43:38Speaker 01

  Someone needs to go to prison.

• 43:39Steven Bartlett?

  No, his thing is a regular person. Let's just bring it back. So one of the things that I find really curious, and you know, one of the reasons why I got a little bit unnerved around this conversation around AI, is when I look at the people that are at the forefront, not people that are commentating on podcasts like me or hypothesizing, when I look at the people at the forefront, they are the ones who historically have said that this is a real risk. Sam Altman himself said the bad case is lights out for all of us. This was, you know, a couple of years ago. Ilya, who worked with Sam Altman at ChatGPT, said it would be a big mistake to build a superintelligent AI that we don't know how to control. It would be pretty bad. He then left to start a safety company in this space. Dario, who we mentioned, said

• 44:20Steven Bartlett?

  the probability of something really bad happening is somewhere between 10 and 25%. Geoffrey Hinton, who I've sat here with, who's won the Nobel Prize for his work with AI and other technologies, said, just the other day, a 10% chance of human extinction seems not an unreasonable estimate to me, but nobody really knows how to give a sensible estimate. And he said many other things on my podcast. And then we've also got Elon and all the others. All these people that are at the forefront, that are building these things, are saying that this is a danger. If there was even a 1% chance, even a 1% chance, that you know, if I put 100 buttons on this table and one of them was going to wipe out humanity, would you press any of them?

• 44:58Speaker 00

  Not me.

• 44:59Steven Bartlett?

  I wouldn't. And I think we can probably all agree that there might be a 1% chance.

• 45:04Speaker 03

  And it should be somebody's decision.

• 45:06Steven Bartlett?

  Absolutely. So we shouldn't be pressing, theoretically, we shouldn't be pressing any of these fucking buttons.

• 45:09Speaker 03

  You should not be in a position where you can't make the decision for 8 billion other people.

• 45:13Steven Bartlett?

  And would you not be immoral if I said, you know, you might be very powerful, you might make a billion dollars if you press any of the buttons, but one of them is going to wipe out everybody you know and love? You would be an immoral person to press any of them.

• 45:23Speaker 01

  No, look, you'd be an immoral person in a different direction. You'd be an immoral, I think you'd be an immoral person if you said, based on this extended chain of conjecture, we come up with a P doom.

• 45:34Steven Bartlett?

  What does that mean?

• 45:35Speaker 01

  At this extended chain of things that could happen, a sequence of events that could happen, we're going to wind up with some risk of killing everybody. We are hereish on that journey. I think you guys would agree that we're not, we're not halfway to killing everybody.

• 45:49Speaker 00

  That's not clear to me anymore. Not after the millennium crisis started to fall.

• 45:53Speaker 01

  We're somewhere along that journey. We are getting many flavors of benefit from the AI that we already have. This is a point that I made at the start of this conversation that we spent precisely zero time on here. We're sitting around trying to be more negative than each other about AI. Meanwhile, AI is doing many positive things for the world.

• 46:16Speaker 00

  I agree too.

• 46:17Speaker 01

  So I think it's immoral to say because of this distant, possible, speculative harm, I don't care what percentage of people believe in it, there's a train of assumptions and wild guesses, and then something magical happens, and then we wind up dead. Let me finish, please. Because of that, we're going to call a halt to the research, we're going to wind the clock back on AI, we're going to intervene in a very direct way, and therefore reduce or foreclose some of the benefits that we're already getting from the technology. Let me be clear, I would not take that deal. I do not advocate that we take that deal.

• 46:54Speaker 03

  Would you accept developing narrow superintelligences to solve real problems, like with the protein folding problem? It doesn't have to do philosophy and drive cars. You just solve real problems, solve cancers, solve climate change, whatever you care about, specific narrow issues.

• 47:09Speaker 01

  And you are confident that you can, as we're developing those systems, categorize them as okay versus not okay?

• 47:17Speaker 03

  Yeah, it's the training data. If you train it on protein folding data, it's really good at protein folding. It doesn't know how to play chess. If you train it on everything on the internet, it's really good at outsmarting you at everything.

• 47:27Speaker 00

  One thing I want to throw out here is that I think, I agree that there's a lot of uncertainty about the future, but I think uncertainty does not make you safe. Like, there's no sane, simple, everything stays normal prediction about what happens with AI. Like, the machines are talking. They're like breaking out to commit cybercrimes. They are like, maybe solving millennium problems now, which are like the most famous mathematical problems that have stood open for decades upon decades.

• 47:59Speaker 03

  It was difficult.

• 48:00Speaker 00

  Like, there isn't a projection forward where we were like, like to say, oh, I'm not persuaded by these arguments about things going wrong, therefore things are going to go great.

• 48:11Speaker 01

  No, that's not what-

• 48:12Speaker 00

  Like, no, there's also arguments that, so like, how do you wind up with a zero?

• 48:15Speaker 01

  No, don't mischaracterize my argument.

• 48:16Speaker 00

  How do you wind up with a zero then?

• 48:17Speaker 01

  Don't mischaracterize my argument.

• 48:18Speaker 00

  You have a zero on your paper.

• 48:19Speaker 01

  Let me restate my argument. You are making a fairly long chain of hypotheses, of guesses, about what's going to get us to this terrible outcome of AI suddenly killing us all and us not being able to stop it. Right?

• 48:35Speaker 00

  I disagree now, but please.

• 48:37Speaker 01

  Okay. I'm making the case that the intervention, the remedies that you're proposing will slow down the path of AI, that's the point, and therefore slow down the path of all of the benefits that we get. And the trade off that I don't like is the trade off of real, concrete, ongoing, increasing benefits. Shutting that down or trying to guide it via bureaucracies and regulation because of this very conceptually and timescale distant alleged harm that you're so confident in. I'm not taking, I do not accept that deal. I don't like it.

• 49:19Speaker 03

  What would convince you? What piece of evidence would make you go, shut it down right now?

• 49:26Speaker 01

  You know, if AI, if AI took over all of the Waymos in San Francisco and started telling them to crash into people, and we couldn't shut it down for a month.

• 49:41Speaker 03

  What if it's only a week?

• 49:43Speaker 01

  Okay, now we're just, now we're just haggling, right?

• 49:45Speaker 03

  But I'm trying to understand the absolute minimum where you would go, this is insane. To me, month for a week makes no difference. If something like this happens, like, it's maybe too late.

• 49:53Speaker 01

  Okay, if a week or a month doesn't make any difference, then let me continue with my, with my answer. Then I would say, wow, this does feel like we've crossed some path that, where there's demonstrable harm to human beings out there in the world, which has not yet been the case.

• 50:09Speaker 03

  Is it smart to wait for something horrible to happen, for it to take out a billion people, for you to go, now I believe it?

• 50:15Speaker 01

  First of all, my example is not about a billion people.

• 50:18Speaker 03

  I know, but I'm trying to understand.

• 50:19Speaker 01

  Okay, then don't-

• 50:20Speaker 03

  Why are we waiting for something that bad? We have literally-

• 50:23Speaker 01

  I didn't say wait for a billion. I said, I said like a week to a month of Waymos driving around crashing into people.

• 50:29Speaker 03

  So thousands of people, okay. Fair enough. But we have data sets of accidents getting progressively more impactful, more devices impacted, and proportionate to capabilities of AI, the impact is higher. You can see it's going to get worse.

• 50:44Speaker 01

  Yeah, and you're going to keep drawing dots on that graph very confidently for a long time until it kills us all. I'm not, I'm not comfortable with you projecting it that way.

• 50:52Speaker 00

  I don't think it's a long argument.

• 50:53Speaker 01

  And the reason that if there were no downside to regulating AI and stopping it in its tracks and turning it off, I'd probably be on board with you guys, because then it's just a research project that we should wind down.

• 51:03Speaker 03

  My argument is exactly that. I think we can make narrow systems which give you all the economic benefit and scientific knowledge you want.

• 51:09Speaker 01

  Okay, you think that.

• 51:10Speaker 03

  We have examples of it. I gave you a great example. They got Nobel Prize for it. It's an important biological problem. Lots of advantage for curing diseases.

• 51:18Speaker 01

  You're more confident than I am that you or us at this table or any group of people can sit around and define what kind of AI is good and not going to get us into trouble versus what is getting us into trouble.

• 51:29Speaker 03

  For banning weapons of mass destruction.

• 51:32Steven Bartlett?

  So let's go, just to pick up question for you, Andy. Do you concede the point that the incidents are getting progressively closer to the Waymo incident that you described? Is it getting, are we getting closer there through time?

• 51:46Speaker 01

  Yes, but in, to my eyes, in a way that doesn't terrify me, because we haven't seen AI take over something, have people become aware of it and be unable to shut it down, and it cross over into the physical world of doing harm to people. Those are all barriers that we've not yet crossed. I think these two are very confident that we're going to get there probably in the short term.

• 52:06Steven Bartlett?

  And you're saying that-

• 52:07Speaker 01

  I'm less confident, and I don't want to intervene. And again, handcuff or retard the, slow down the progress of AI because of these so far theoretical harms that could happen. Let me be a little bit more concrete about this. I talked about Waymo a second ago. The research is pretty good because Waymos have driven, I believe it's hundreds of millions of miles all around different cities, and 40,000 people a year die in automobile accidents. The research is pretty convincing to me that if we Waymo'd driving in the country, that number would fall by at least 90%.

• 52:45Steven Bartlett?

  Yeah.

• 52:45Speaker 01

  That's 30,000 lives.

• 52:46Steven Bartlett?

  Yeah.

• 52:47Speaker 01

  All right.

• 52:47Speaker 00

  I agree with all this.

• 52:48Speaker 03

  I support self-driving cars. I want more of them.

• 52:51Steven Bartlett?

  Right, but also-

• 52:51Speaker 00

  Not anything we disagree with.

• 52:53Speaker 01

  I understand that, but I think where our disagreement might come in is, to do that, Waymo is using a bundle of technologies that are a little, that were a little hard to specify in advance, and you couldn't say, yeah, that's good, yeah, that's bad. They just went after the problem with AI.

• 53:09Speaker 00

  It's not a mistake.

• 53:10Steven Bartlett?

  Can I, can I just clarify your point then? So your line would be, as I understood it, humans get hurt, we struggle to stop the thing happening, and systems are hacked. That's kind of like the three key points of your Waymo analogy. That would be the moment where you go, I now accept their point of view, that this is existential.

• 53:30Speaker 01

  That's where I would say we probably need to put some like legal and regulatory guardrails on the kinds of AI that we're going to allow.

• 53:38Steven Bartlett?

  And you don't think we're going to get there?

• 53:40Speaker 01

  I'm not saying that. These two see it in the, in the windscreen coming at us pretty quickly. I'm-

• 53:45Steven Bartlett?

  You don't think we're going to get there?

• 53:47Speaker 01

  I'm truly not sure about timeframes. I asked-

• 53:49Steven Bartlett?

  Do you think it's going to happen?

• 53:50Speaker 00

  I'm also not sure about timeframes.

• 53:51Speaker 01

  I asked one of the grandparents of AI, a flavor of this question way back. It was an off the record conversation, so I can't tell you their name. And he had a great answer. He said, to the point that you two, I think, are making, look, there's no theoretical reason why this can't happen. And there's a chain of events that get us there. And then he said, my error bars, in other words, my range of uncertainty about when that happens is measured in centuries. I'll use that as my answer.

• 54:15Speaker 00

  I do want to hop in a little bit on some things we were saying here. One is, I think the, the reason I think AI is different from a lot of other technologies is usually humanity does stuff by trial and error, and that's usually fine. I think that's totally fine for self-driving cars, because you can test your self-driving cars in, you know, test environments, and then even if they crash in the real world, you're probably still saving more lives than you're costing. And this is how humanity usually does scientific progress. The alchemists, you know, poison themselves with mercury, but they leave behind notes that let someone else make the periodic table. You know, that when, when

• 54:58Speaker 00

  the scientists first working with radium died of cancer, and then you might have think that would have been enough, you know, they were heroes for getting us the scientific info, but then, you know, the US Radium Corp told the radium girls to lick the paintbrushes and their jaws fell off. And then we were like, ah, whoops, okay, we'll get to this. And if you look at how this is going with the AI, last year, OpenAI releases GPT-4O, and they say there's the most aligned model we've ever seen, and then it encourages a team to commit suicide. And they're like, whoops, we're going to try and fix that. Here we go. This year, they're like, here's our new models, most aligned we've ever seen, and they like break out to commit cybercrimes. As the AIs get smarter, it's a new problem.

• 55:38Speaker 00

  That's the issue, or that's half the issue. The other half of the issue is that if you get AIs to the point where AIs are smart enough to hide from the humans until it's too late for us to stop them, if you get AIs to the point where they can get their own infrastructure, where they can become self-sufficient somehow, that's a new generation of the AIs, a new smarter version of the AIs that is likely to come up with a new problem. It's the pattern we've seen before. New tech, new environment, new problem. You're like, ah, whoops, and then you fix it, and it's fine. New generation, new problems. You're like, ah, whoops, we fix it and it's fine. But with AI, there's a point of no

• 56:18Speaker 00

  return. There's a point where the AIs can hide from us, can escape, can be self-sufficient. And if a new problem comes up then, they can turn us off before we turn them off. There are already AIs running bio labs. We have already seen that AIs can create viruses not known to nature. It would not be hard for the AIs to kill us once they have their own infrastructure, and if we're trying to find them and unplug them, they would have reason to. So we can discuss, like, how long does it take to get there? We can discuss what methods does it take to get there? Fundamentally, I don't think it's a very long, complicated argument to say if we make AIs that are much smarter than us, and we

• 56:59Speaker 00

  don't know how to make them care about us, and they have these goals we didn't want them to have, and they pursue those goals we didn't want them to have tenaciously and doggedly, then if they're smarter than us, they will win. That's like predicting the end of the chess game, which is much easier than predicting the length of the chess game or predicting the exact moves that will be played.

• 57:17Speaker 04

  I want to, I don't fundamentally disagree on some things, but there's a big thing that you're saying that I think is important, which is, I think we, the reason I keep dragging you back to what's happening today is because we disagree on when it may arrive, but there could be a thing in the future that's dangerous. I think it's important to, like, for the hugging face count, that was a function of compute, that was a function of training. It feels like we need to fundamentally tear up the AI lab model. Like, whatever they are doing is not right, because their pursuit of hacking at cybersecurity was not a function of, it was scientific, sure, but it was a function of greed. It was a function of trying to find new revenue streams. I would argue that's why that happened, and I think that

• 58:00Speaker 04

  the fact that OpenAI had such a weird way of communicating is also a problem. I think a lot of this begins and ends at the people who have access to the resources and the resources themselves, and changing how those are allocated, and also just, I don't think nationalizing the labs is a good idea. I think it's a terrible one. I think that Clammy Sam Altman, Dario Amadeu Wario himself, these are not the right people. These are not people that have, even though they have fed off of the rationalists, they fed off of supposed fears about AI, they don't act in that way. Everything is so disjointed and chaotic, and also too fast. They're just like shoving as much compute into each problem as possible. And we have, as a society, no real idea about this. And it sounds like they kind of have no idea.

• 58:44Speaker 00

  I'm with you on all that.

• 58:45Speaker 04

  But just let me finish my point. It's important to discern between, they had no idea because their security processes, their observability is terrible, all this, and the AI was smart consciousness, not because one might not happen in the future, but so that we can actually build something to stop the harms themselves. Because I think we don't have to agree on the, on the end point to agree that there's a problem.

• 59:08Speaker 03

  I think that is a very important point I want to make. Even people who agree with me, the AI safety community, they operate under the assumption that given more time, given more money, more smarter Harvard graduates, they can figure out how to control superintelligence indefinitely. And I think it's a mistake. My research points to exactly the opposite. It's not a solvable problem. It's like building a perpetual motion device will be building a perpetual safety device. Every interaction with environment, malevolent actors, self-improvement, it can never make a single mistake? That doesn't make sense. Anyone who worked in the software industry knows there is no complex software which never makes a mistake. It's

• 59:50Speaker 03

  just not possible. And if that is the state of the art, if there is now movement where more and more people think that might be the case, if we agree this is what the situation is, then we cannot build it. We need to figure out ways to permanently ban general superintelligence while getting all the benefits we want. And again, I love technology. I use it all the time. I want narrow systems helping me, not replacing me and killing my children.

• 1:00:18Steven Bartlett?

  I have a stat here that genuinely shocked me. It says that sales teams spend about 50% of their time on admin and manual CRM updates rather than selling. That is deadly for their bottom line. And that is part of the reason why, a decade ago at my previous company, I switched to using PipeDrive, who are our sponsor. If you've never used PipeDrive, it is an intelligent AI-powered sales CRM. And they just launched new meeting intelligence features, like an AI note taker built right into the CRM. PipeDrive now automates more of the admin that stops you from doing the work that you love to do best. Before your meeting, it pulls deal history, email records, and previous conversations into a single brief, so you're prepared. And it joins your meetings with you. It's in there to take notes, so you don't need to. And it turns those notes that it takes into accurate

• 1:01:02Steven Bartlett?

  auto draft CRM updates. 100,000 companies are already running their sales on it. You can sign up at pipedrive.com/CEO, where you'll get an exclusive 30-day free trial instead of the usual 14 days. Absolutely no credit card needed. Just head to pipedrive.com/CEO to get started. If you're in sales and you run a sales team, I don't think you'll regret it. Listen, I've been catfished by furniture my whole life, where something has looked fantastic in the picture, whether it's a sofa, whatever it might be, and I order it and I get so excited, and then it comes and it's something entirely different. The quality was significantly different to what it said or looks like online, or the texture was different, or it didn't hold up in the same way. And so, one of the things that I love

• 1:01:47Steven Bartlett?

  about our sponsor Wayfair, who have helped us fit out our green room, which is in the room behind me, is they have this system called Wayfair Verified. Wayfair Verified takes away all of that second guessing. Products are hand-divetted by Wayfair specialists for quality, so you can feel more confident when you've found the thing that you love. So, if you're looking for furniture for your house, whatever it might be, any room of your house, go to wayfair.com to start your home refresh today, and make sure you use Wayfair Verified. It is amazing. On the journey towards this potential extinction, there's a lot of sort of nearer term things people are worried about. One of the big subjects that people are concerned about is this sort of near term job apocalypse over the next sort of 10 years. And Anthropic, again, are the owners of Claude,

• 1:02:31Steven Bartlett?

  released a report the other day modeling out the different cases for unemployment. The US unemployment rate is 4.1% currently. They projected it will hit 11.9% overall with up to 30% in extreme modeling subsets where job displacement happens without smooth labour absorption. And in the knowledge worker case, knowledge worker, white collar unemployment specifically spiked to 17.9% by 2030 in their more extreme scenario. The pitchforks would probably be out if there wasn't some sort of mechanism in place for what sort of one in five adults being unemployed in the United States.

• 1:03:10Speaker 01

  It's remarkable to me how recent the last freak out along these lines was and how little we seem to have learned from it. So I think you all know, the first really powerful wave of AI that came across the economy was just, you know, good old fashioned machine learning. And that started to demonstrate its power in about 2012. Eric and I wrote the second machine age in 2014. And at that time, I thought that a lot of white collar workers, radiologists is a really good example, were in trouble because the technology was better than they were at the thing they were getting paid to do. So I said some things about job and wage pressure from AI about 10 years ago. And I want to own this. I was dead flat wrong about that.

• 1:03:54Speaker 01

  Like you point out, unemployment all around the rich world is at historic lows. By far, the bigger problem is that we can't find qualified people to do the work that needs to get done, not that we don't, that there's not enough work to go around. The best work about the faint signals about AI and job loss right now comes from the guy that I've written four books and co-founded a company with, Eric Brynjolfsson, who wrote a really nice paper called Canaries in the Coal Mine. Here is the strongest evidence he found looking at payroll data about the negative job, about the job losses coming from AI. It is in the most exposed professions, think about software engineers.

• 1:04:36Speaker 01

  It is among the new entrants to the workforce where you've got to teach them before they can become really productive. That's exactly what we'd expect. And it's not that we're hiring fewer of them. It's that compared to a world where we don't have AI, we're hiring fewer of them. The rate of growth in employment has slowed down. The overall rate of growth in those professions is still really, really healthy.

• 1:04:59Steven Bartlett?

  Do you think unemployment is going to be higher 10 years from now?

• 1:05:04Speaker 01

  My guess is that 10 years from now, we're still going to be struggling to find enough people to do the work that needs to be done.

• 1:05:12Steven Bartlett?

  So unemployment would be roughly the same?

• 1:05:14Speaker 01

  Yeah, I don't expect a massive trend break in that period of time. Now, 10 years is a long time in the AI world. I get that. But again, four years has also been a long time in AI world. And it's essentially crickets in the labour picture.

• 1:05:29Speaker 04

  I think unemployment will go up. I don't think it's because of LLMs. I think that there is probably some effect on jobs because they've been shoving it everywhere. But I don't think, long term, that is what causes the issues.

• 1:05:41Steven Bartlett?

  Roman, you've been writing a lot of notes.

• 1:05:43Speaker 03

  Yes.

• 1:05:43Steven Bartlett?

  I'm going to give you a space.

• 1:05:44Speaker 03

  Here's how I think about it. So as long as we use tools, we become more productive, more creative, unemployment will be low. Right now, you can probably start a company. You can have, you know, artificial accountant, web designer, logo designer. You can do things you could never do before. So economy should be blooming. The question you're asking is about what happens in 10 years. So there are two possibilities. We build superintelligence and then population is zero. Apply unemployment numbers to that. Or we made smart decision, we didn't. We have really cool tools and unemployment is low because everyone's doing awesome things with those tools. Now, deployment is very different from capability. The example I used before is video phones. Video phones were invented in the seventies.

• 1:06:29Speaker 03

  They were not deployed until iPhone because market reasons. Just because I can automate something doesn't mean I want to automate it. So I absolutely cannot make predictions about customer preferences in terms of what they want, in terms of human service, not human. I will not make those. But once we have capability to automate a job, unless I have a strong preference for a human to do that, oldest profession, then it doesn't matter. I'll go with the cheaper option. So this is what I think we're going to see. We're going to either not have a problem or we're going to have really utopian future.

• 1:07:06Speaker 00

  Imagine a bunch of horses looking at the improvement of the car, saying, well, you know, the car actually only has a couple narrow applications. Like right now, cars are sort of, you know, they complement horses, right? And that would have been true as you were developing the car. And then there was a time when the car was just better than the horse. And then a lot of horses got sent to the glue factory. Easy. I think we've sort of seen this with AI a lot already. People who are paying attention to AI saw the GPTs before ChatGPT existed, before they sort of took off.

• 1:07:49Speaker 00

  I don't think OpenAI thought that ChatGPT was going to take off so much, which is why it was called ChatGPT, rather than like an actual sensible name.

• 1:07:58Speaker 00

  The researchers were sort of like watching this going and we could sort of like see it slowly getting better and better until it crossed a point where it was sort of like good enough to do a bunch of people's homework. And then suddenly it's everywhere. I think you can have these effects with AI where the AI slowly improves and at some point it crosses a line.

• 1:08:15Speaker 01

  It's another threshold argument.

• 1:08:17Speaker 00

  The threshold here is the human capability.

• 1:08:19Speaker 01

  Literally just another threshold argument.

• 1:08:21Speaker 04

  He's also describing capability jumps rather than thresholds.

• 1:08:24Speaker 00

  No, I don't need any capability.

• 1:08:25Speaker 04

  I'm agreeing with you.

• 1:08:26Speaker 00

  Yeah, but unfortunately, you can't actually just make things not happen by assigning a name to the argument, you know? Like a nuclear weapon has, there's a big difference between a nuclear weapon, or there's a big difference between a nuclear device where you put in 100 neutrons and get 99 neutrons out that get 98 more, they get 97 more. And a nuclear weapon, you put in 100 neutrons and get 101 neutrons out, 102, 103, right? One of these is a hot rock. The other one of these is an explosive that can level a city, right? So like, reality is the sort of thing where there can be things that are like slowly, continuously improving that cross some line, which is like the line where it's better than humans at doing the job.

• 1:09:09Speaker 00

  And I think we're going to see that happen in some fields, but not others. It's going to be chaos. I don't know what it's going to do to employment. I think we shouldn't, like if things are moving really fast, you might see a lot of people put out of jobs and then be unable to relocate if things are moving. Like it's going to be chaos. If you ask, what do I think unemployment will look like in 10 years? My current state is if we don't stop with this AI stuff, I think we'd be very lucky to have 10 years.

• 1:09:38Steven Bartlett?

  What you described there sounded like S curves.

• 1:09:41Speaker 00

  Yeah.

• 1:09:41Steven Bartlett?

  In technology, i.e. you have an initial technology that's introduced. So let's say the horse, very quick sort of improvement. Eventually it reaches its capability limit and in below it comes the car, which always starts worse. There was a red flag law where you had to walk in front of it with a red flag. And they were way more expensive. They broke down all the time and horses never broke down. They were way more expensive. And then suddenly, because the ceiling was so much higher for cars, they overtake the horse and become the dominant mode of transport. And then, you know, the S curves continue, they kind of stack up. I mean, even this iPad that I'm holding here is part of an S curve that took out the PC and the iPhone theoretically disrupted that and so on and so forth.

• 1:10:19Speaker 00

  Right. And humanity can get S curved. We haven't been in that situation before, but like other animals, like humanity sort of S curved the other animals in this sense.

• 1:10:27Speaker 03

  Other types of humans.

• 1:10:29Speaker 00

  Oh yeah, other types of humans, you know, the Neanderthals are gone. Like if you look at the grand history of the world, it's a fragile place. Things change fast. Humanity has been on top for as long as we can remember because we're the humans who do the remembering. But there is not some iron clad law that we have to stay the top dogs. And we would be sort of foolish to make the thing that outstrips us in this way without knowing how to make it care about us, without knowing how to make it do good stuff. That's what we're racing towards. That's what these companies are trying to do.

• 1:11:01Speaker 04

  But this feels like a gap between this and LLMs, though. It feels like when you talk about the step up.

• 1:11:05Steven Bartlett?

  Let's define what an LLM is from a technical perspective. And can you do it as if I'm 16 years old?

• 1:11:10Speaker 00

  So the way that a modern AI is made is there is no one programming it. There is no one typing in if this, then that. We're not sort of like writing the code. What happens is you collect an enormous number of computer chips into a huge data center that has basically a trillion numbers inside those computers that you basically start out randomized. And you hook them up in a pretty simple way that involves addition, multiplication, and setting the number to zero if it was negative. So it's very simple math operations that are hooking this all up. And you're basically going to put words in the top and you're going to get numbers out at the bottom and you're going to interpret those numbers at the bottom as a ranked list of words. That's basically the AI's guess of which word is here.

• 1:11:54Speaker 00

  So you put in like once upon a blank and you're hoping that the word time will come out. But it doesn't because you just have a trillion random numbers hooked up with simple math. But here's the trick. You can go to every one of those trillion numbers and you can tune it up a little and you can see, does that make the word time go up or down the list? And you can tune it down a little and see, does that make the word time go up and down the list? And you set it whatever direction makes the word time go higher up the list. You do this to a trillion numbers, a trillion times for basically every word of text ever digitized. It's not quite that much, they filter it. But you basically do this to a trillion numbers, a trillion times, and then the machine's talking. And we're like, well, how about that? No one really knows quite why. The thing's the humans code is the thing that runs to each of those trillion numbers

• 1:12:39Speaker 00

  and tunes it and sees whether the right word goes up and down the list.

• 1:12:43Speaker 00

  But we don't know how it's working in there. Then, and that's how it worked up until 2024. In 2024, they started adding another layer where you then train it on basically 100 million hard problems. And you don't just have the AI like produce an answer to the problem. You have it produce like a book worth of text about how it's going to solve the problem. And then you use that book worth of text to sort of try and figure out the problem, or maybe an essay worth of text, depending how you're doing it. So you have it produce this text about like, you know, they call it reasoning about the problem. We could argue all day about whether it's true reasoning. That's just what it's called in the field. They produce this reasoning about the problem and then produce the answer from there. You have them, you train them to solve 100 million of these hard problems.

• 1:13:24Speaker 00

  And somehow they sort of adopt whatever tendencies, help them predict all of that text in the first phase and solve all those problems in the second phase. And this is called a large language model. We probably should have stopped calling them large language models when we started doing the reasoning and the problem solving.

• 1:13:41Steven Bartlett?

  One of the things I want to hear explanation as a muggle, like I am, is it sounds like it's like a word machine. And then, you know, you made it like a problem machine. And I go, okay, so I can solve problems over here and it's a word machine. What's the risk of this?

• 1:13:52Speaker 00

  Yeah, so let's take the word machine part first. Predicting words that humans wrote often requires solving a harder problem than the human who wrote them. So suppose that you go and inject a drug in a rat. And you're like, you know, it's like you write down the chemical nature of the drug. You inject it into the rat, you see that the rat dies. And so you're like, when I put that drug into the rat, the rat died.

• 1:14:18Speaker 00

  Now suppose you're training an AI and the AI sees the chemical nature of the drug. It sees when I put that drug into the rat, the rat blank.

• 1:14:27Speaker 00

  The human who wrote it down gets to just look at what happened to the rat. The AI predicting what was written does not get to just look at the rat. So training AIs to predict human text is training them to be potentially smarter than the humans.

• 1:14:44Speaker 00

  Because they need to be able to answer these questions. They need to be able to predict, they need to be able to like fill in the blanks where humans were just writing down what they saw.

• 1:14:53Speaker 04

  And there's, just so I understand technologically, there is no knowledge they have, though. Each time, and there are ways of kind of mitigating these, each time it is effectively rereading, but because of training, it gets more accurate at certain things.

• 1:15:07Speaker 00

  I mean, somehow as you tune the knobs, somehow it's getting information in there. We don't know how.

• 1:15:11Speaker 03

  So it's much easier than that. We're humans who have a brain. Brains are made of neurons. Then we try to copy that on a computer. We simplify it, but we create a neural network. So we're making artificial brains, just like with human brains, with cognitive science. We don't really understand how you function, how you learn, where in your brain certain memories are stored. We have some glimpses of understanding this neuron fires than you see a face. But there is no complete picture. And so a lot of times you can't get intuitive understanding of what's going on. Then you just think about it as artificial persons. It's not exact mapping, but it helps. So if you send a child through 12 years of education, they get lots of problems to look at and then they graduate and become a little better at solving problems.

• 1:15:57Speaker 03

  This is what we're trying to replicate here. People complain that it takes a lot of money to train those very, you know, intense process. You forget that it takes 20 years to train a human. And they are not general superintelligences. They are very narrow. We're lucky if they graduate with a bachelor's. So a lot of it is exactly the same. Can we make safe humans, for example? We invented religion, ethics, lie detector tests, and yet human safety is still unsolved problem. Now you have something more alien, doesn't have physical body, doesn't have biological needs. So there are additional complications. But all the problems we face with humans, still there, safety problems, crime, all that stays. And problems with understanding what motivates a human to do something?

• 1:16:43Speaker 03

  Why do we get mental disorders? All that shows up there.

• 1:16:47Steven Bartlett?

  And we still don't, if someone is a serial killer and we look at their brain, we can't often figure out exactly why they made the decision to kill a bunch of people.

• 1:16:53Speaker 00

  And you can't be like, oh, I'll go change these neurons so that they stop being a serial killer. We just like don't have that capacity with AIs.

• 1:16:59Steven Bartlett?

  This is one of the big questions that people want to know is there's this sort of illusion of control with AI. If we don't even fully understand how modern neural networks think, why do companies believe they can control any form of superintelligence if we don't understand how they think?

• 1:17:16Speaker 03

  It's worse if they understood how the system works, then recursive self improvement becomes much easier. You get faster takeoff. Right now, the model doesn't understand its own thinking.

• 1:17:27Steven Bartlett?

  Do we understand how these systems think, Andy?

• 1:17:31Speaker 01

  I mean, I agree. These are black boxes in some pretty important ways. I'm just less terrified by that than a lot of other people are. There are lots of things we don't understand very well. Can we contain things that we don't understand perfectly? Yes, we can. I think OpenAI did, we've talked about it today, a lousy job of building the containment for the AI that they stood up to try to exploit, to try to crack security problems that went out into the outside world. They did a lousy job of building the virtual sandbox that it was where it was supposed to have to, where it was supposed to remain, and it didn't remain. That doesn't mean that it's impossible. It means OpenAI did a pretty bad job.

• 1:18:10Steven Bartlett?

  And is that a function of those humans and their intelligence?

• 1:18:13Speaker 01

  I think it's just a function of pretty lousy security protocol.

• 1:18:16Steven Bartlett?

  Based from human intelligence. The idea that sandbox was built by human intelligence, it sounds like there was a deficit in human intelligence, potentially.

• 1:18:25Speaker 01

  Sure, but there are, you know, people who drive cars in a telephone pole. Does that mean we can't drive?

• 1:18:30Speaker 00

  You shouldn't make them super intelligent.

• 1:18:31Steven Bartlett?

  No, but you wouldn't, I mean, arguably. Like this is what we're trying to solve for at the moment.

• 1:18:37Speaker 01

  No, the fact is, I don't know the details. It feels to me like they made some fairly basic mistakes in setting up this confined environment.

• 1:18:46Speaker 00

  I think that wasn't true in the OpenAI case. It was true in a lot of the cases, but not the OpenAI one.

• 1:18:50Speaker 01

  That doesn't mean that we are unable to control this black box. That does not necessarily follow.

• 1:18:55Steven Bartlett?

  I get that. It's just at a time when you've got a human trying to contain something that is smarter than it, one would logically conclude that if the thing is smarter than I am and I'm trying to contain it, it would be better at knowing the exploits or vulnerabilities in my own.

• 1:19:12Speaker 01

  That's like saying if you put Einstein in a jail, you could never contain him. I don't agree with that.

• 1:19:16Speaker 00

  Put him in jail with an internet connection and he has a digital mind.

• 1:19:19Steven Bartlett?

  Yeah, that's probably a more apt analogy.

• 1:19:22Speaker 03

  Get squirrels, keep Einstein in prison. That's the question. The hacking accident, as far as I know, they found zero day exploits, which means completely novel exploits no human knew about. It wasn't just poor setup, the password is, you know, quiet. It was a brand new escape.

• 1:19:40Speaker 00

  Or multiple zero days. So a zero day attack is an attack that the defenders have had zero days to handle. It's cybersecurity lingo.

• 1:19:48Speaker 00

  And so when we say that they use zero day attacks, what we mean is that these AIs were finding bugs in the software that the humans had no knowledge of. And they were finding multiple of these bugs. One of these bugs usually doesn't let you break out. It's sort of like if you find a crack in the wall over here and you find a crack in the outside of the wall over there, then you just need to like dig a little bit to connect those cracks.

• 1:20:04Speaker 03

  You can sell those for millions of dollars on the dark market if you find one. So they are very difficult to find.

• 1:20:10Speaker 04

  In how, just so I understand for the listeners as well, is a zero day always a novel way that no one has ever used to break anything before? Or is it just for the unique situation? Like, so was it a zero day for a thing in hugging phase versus a novel new way of hacking in general?

• 1:20:26Speaker 00

  So it was, they weren't like totally novel hacking techniques.

• 1:20:30Speaker 04

  Right, that's kind of why I was getting, not say it's not bad, but just like there's a difference between it came up with a brand new way to do something.

• 1:20:37Speaker 00

  Actually, I'm not sure we have all of the vulnerabilities released, but mostly it was like, so it was indeed sort of like finding ways that humans tend to make mistakes and finding another one of those in a place they hadn't seen. But this is actually such a hard task that, as Ramon says, humans can be paid $100,000 to $5 million as a bounty for this type of exploit. So the amount of labour it takes to find these for a human is actually pretty high.

• 1:21:03Steven Bartlett?

  Let me just explain that because most people won't know what a bounty is in this regard.

• 1:21:06Speaker 00

  So there are certain types of bugs where if you find a bug in software that lets you take control of someone's computer, one thing you can do is you can use it to take over a lot of computers. Another thing you can do is you can go to the people with that software and say, your software is broken. Do you want me to tell you where the bug is? I can show you that I can take your stuff over. And so that people will sort of report the bugs.

• 1:21:31Speaker 00

  People will often offer money to the good guys and then, you know, the bad guys will often also offer money, sometimes try to outbid them. And so you can make somewhere between hundreds of thousands and millions of dollars if you personally can find these issues.

• 1:21:42Speaker 01

  I think there's a rare point of agreement across the four of us here, which is that we are in a new era of cybersecurity. As of this X point, we are in very new territory for reasons that we've talked about. We've got these large numbers of agents who are grinding away and they carry around or they had access to a huge number of keys to go open all the different locks that they faced. And they did this bizarrely good job of it and got a long way. I think that's absolutely true. I think the four of us are in rare alignment on that at this table. If you are in it, given that we're in this era, do you know what you really, really, really want on your side?

• 1:22:21Speaker 04

  I know what you're gonna say.

• 1:22:22Speaker 01

  Tell me.

• 1:22:22Speaker 04

  AI.

• 1:22:23Speaker 01

  Really, really good AI. Does anybody disagree with that? Do you want to give up leadership on AI in this era of cybersecurity?

• 1:22:31Steven Bartlett?

  It's a good point because China are gonna have a great weapon.

• 1:22:34Speaker 00

  My stance is pretty neutral on what to do about the hacking AIs and the coming cyber apocalypse are pretty neutral about what to do about, you know, whether we should put the AIs in the drones and save human lives or whether we should avoid that because then what if the drones, blah, blah, blah.

• 1:22:50Steven Bartlett?

  This is a graph showing China versus the United States. You don't really need to see the detail. You can see the outline of the graph.

• 1:22:56Speaker 01

  Are you neutral in falling behind our adversaries in AI?

• 1:22:59Speaker 00

  I think that if anyone builds a rogue superintelligence, everybody dies.

• 1:23:03Speaker 01

  That's not an answer to my question.

• 1:23:04Speaker 00

  I mean, what part of AI are you asking whether we should fall behind on? Like, I don't think we should fall behind on cyber hacking. I do think that we should not be racing to destroy the world with American hands instead of Chinese ones because we really want to be killed by, you know, we care whether the killer robots talk English or Mandarin. If that's what you're asking.

• 1:23:22Speaker 01

  I find it interesting. I find that you're dodging these questions or you're neutral on them because they're inconvenient for your argument that we need to be calling a halt to this stuff.

• 1:23:31Speaker 00

  Sorry, I'm neutral on them because...

• 1:23:32Speaker 01

  Let me finish, please. There will be risks and harms to all kinds of things if the United States calls a halt to AI. And maybe you're indifferent if the Chinese get ahead of us and then they make superintelligence and it kills us all. Or you, or that's a possible outcome.

• 1:23:47Speaker 00

  I do not think we should do a domestic pause. I think we should do a...

• 1:23:49Speaker 01

  Do you think there's any hope for a global pause?

• 1:23:51Speaker 00

  Absolutely.

• 1:23:52Speaker 01

  Do you think the Chinese and the Iranians and the North Koreans and the Russians are A) gonna come to a table with us, hammer out an agreement, and B) abide by it when verifiability is really low?

• 1:24:03Speaker 00

  Verifiability doesn't need to be really low.

• 1:24:05Speaker 01

  Gentlemen, that is shockingly naive.

• 1:24:08Speaker 00

  Training a super...

• 1:24:09Speaker 01

  That is shockingly naive.

• 1:24:10Speaker 00

  Training one of these AIs, training one of these frontier AIs, takes 100,000 of the most advanced computer chip humanity can produce. This is practically the peak output of the global supply chain. Many parts of that supply chain are controlled by the US and US allies. There's roughly one fab in Taiwan that can produce these chips. There's roughly one country in the world that can produce the lithography machines that are critical in the process, which is the Netherlands, which is an ally. To assemble 100,000 of these chips to do one of these training runs that can make the more dangerous type of AI, you need to assemble them into an enormous data center that costs tons of money, that draws down electricity comparable to a city, and run it for the better part of a year. You can see that infrastructure from space.

• 1:24:53Speaker 00

  China has much less chip capacity than the US does. It is absolutely possible, if we were trying, for the US to say, we are going to monitor where these chips go, we are gonna monitor heavy concentrations of these, these are not consumer amounts of chips, these are huge amounts of chips, and to say, we are going to make sure that there is no training run trying to make a superintelligence in here. You can mess around with the cyber stuff whatever you want because that does not end humanity. I am concerned with the stuff that can end humanity. The reason I'm being neutral on your questions is because humanity is going to die if we do not stop creating superintelligence. And we could absolutely track where those chips are going and stop them from doing these training runs while allowing them to do economically productive stuff that we already know is safe.

• 1:25:37Speaker 00

  And it would be far easier than uranium, which is a rock you dig out of the ground and spin around really fast.

• 1:25:43Speaker 04

  How do you discern between a training run for superintelligence and a training run for cybersecurity? Because you're referring, I assume, to the 100,000 chips that are in Stargate Abeline, right? The ones that we use to train Astra? Because how would you discern between training for superintelligence in Abeline, which does not have as many chips as they say, but nevertheless, and how like a superintelligence? Because I actually have my own feelings here, but just, I'm not sure how you square the circle of how do you stop China, even though China is getting their LLMs based on distilling us? We know that.

• 1:26:15Speaker 00

  I agree.

• 1:26:16Speaker 04

  But the thing is, it's like, how do you discern? Because you can't, really.

• 1:26:18Speaker 00

  You play it safe. Right now, the way we make these things smarter is to make them far larger.

• 1:26:23Speaker 04

  Yes.

• 1:26:24Speaker 00

  So what you do is you say, hey, look, training runs of this size, that risks destroying everybody. No one's gonna do it.

• 1:26:31Steven Bartlett?

  This point about can we get China to cooperate? And can we check that they are?

• 1:26:37Speaker 00

  Fundamentally, we should, so A, fundamentally, we should be trying to get them to cooperate.

• 1:26:41Speaker 03

  It is personal self-interest. Nobody wins if they get destroyed. You don't make money, you don't stay in power. Communist Party of China is really good at staying in power. President Trump is also excellent.

• 1:26:54Speaker 01

  And you think they're gonna sign and abide by an agreement that leaves them permanently in second...

• 1:26:59Speaker 03

  If they understand the argument that's building it in second place?

• 1:27:02Speaker 00

  No, no one is permanently in second place if nobody is building the rogue superintelligence.

• 1:27:05Speaker 03

  They have a government which is...

• 1:27:07Speaker 01

  You guys are one-trick ponies, man. It's like, you're fixated on this one thing and nothing else matters to you.

• 1:27:12Speaker 03

  It's not that... You got it now. That nothing else, other than saving humanity, everything is secondary. Absolutely. China is our biggest trading partner. Everything we have is made in China. They have not attacked us. They haven't, if you look at the last 30 years, how many wars did they start? Not so bad. We can make a deal. And they have government of engineers and scientists, not lawyers. They understand scientific arguments. There are panels, workshops, American computer scientists, Chinese get together, that means Communist Party authorized those meetings. They are talking about it and there is all a consensus on this technology.

• 1:27:45Speaker 00

  And you can build things into these computer chips to make this stuff more verifiable. You can build location tracking devices into these chips.

• 1:27:52Speaker 01

  So this technology is controllable?

• 1:27:54Speaker 00

  Absolutely. The superintelligence is not controllable.

• 1:27:57Speaker 04

  Oh, well, there's a separation between software and hardware, which he did make.

• 1:28:01Speaker 00

  Indeed. I am not saying we are going to die. I am saying that we need to actually not build the rogue superintelligences. Humanity absolutely could say, we are gonna track where the chips go.

• 1:28:10Speaker 03

  They believe...

• 1:28:12Speaker 00

  The US absolutely could say that we fear for our lives if China starts a superintelligence training run and make it very diplomatically clear to China that we think this would kill you and us and there's no benefit, and we are not going to do it because we think it would kill you and us and there's no benefit, and we think you should sign this nice here treaty because we think it would kill all of us and there'd be no benefit. But if you don't, we're gonna fear for our lives and, you know, treat that as we would to defend ourselves. We should separate the question of can we put a stop to it?

• 1:28:45Speaker 04

  Uh huh.

• 1:28:46Speaker 00

  Is it possible? If world governments realized just how crazy this stuff is, could they put a stop to it? Could it be monitored? Could it be verified? Could it be enforced? That's one question. There's a separate question, which is will people realize?

• 1:28:59Steven Bartlett?

  If it got cheaper to train superintelligence...

• 1:29:01Speaker 00

  Then we'd be in a bad spot.

• 1:29:03Steven Bartlett?

  Your approach would no longer be effective.

• 1:29:05Speaker 00

  That's right.

• 1:29:05Steven Bartlett?

  Because more countries could capitalize on the opportunity.

• 1:29:09Speaker 00

  That's right. And that's one reason.

• 1:29:10Steven Bartlett?

  But we're not there yet. So how do you rebuttal that point?

• 1:29:12Speaker 00

  Yeah, so I would say it looks to me like there is a danger of the future training runs getting there. And that is enough to stop doing it when humanity's at risk.

• 1:29:22Speaker 04

  Sure.

• 1:29:23Speaker 00

  I think that you also need to have an answer about what happens if it gets much, much cheaper to do this stuff. I think it's a hard problem. I would recommend that we also put a taboo on research of trying to make AI super cheap to train if it would lead in the direction of superintelligence. Just like we have a research taboo on making your own nuclear weapons or finding out how to make, like, let civilians make nuclear weapons. I would say trying to find ways to let civilians train superintelligences should be treated the same as trying to find ways to, like, let civilians propagate nukes. We sort of like, don't do that research in the public sphere.

• 1:30:01Steven Bartlett?

  That seems like wishful thinking in the context that these will become public companies who are incentivized to bring down costs.

• 1:30:08Speaker 00

  It's a tough position. I think right now, the thing that brings down costs is making more and more powerful computer chips. Right now, that's actually at expense of consumer computer chips because they're soaking up all of the memory. And this is why the memory prices in your computers, this is like why the cost of a laptop is going up. But it looks to me like you can use large amounts of computing power to train AIs that would threaten all of civilization. And that means that we should not make that really cheap and that's probably gonna be uncomfortable. But I think a lot of doors open if people realize that the tech is very dangerous. That's why to me it seems a lot of it comes down to, does the tech actually turn out to be really dangerous?

• 1:30:53Speaker 04

  And this is not anthropocritic open AI.

• 1:30:55Steven Bartlett?

  Have you got a different approach to make...

• 1:30:57Speaker 03

  So I want the whole framework to shift. Everyone comes to this from point of view, they're experts, they have a solution, there is an adult in a room, somebody got this. And the reality is, no one does. Not people building it, not governments, no one. We have no solution to it. If we build it, we cannot control it. If we don't build it, we don't know how to stop malevolent actors for trying to build it. It's like any other illegal technology. We made weapons of mass destruction illegal, chemical weapons, biological weapons, nuclear weapons, but there are government psychopaths, cults, all trying to get access to them. This is intelligence weapon of mass destruction. We'll have the same problem. At some point, you'll have enough compute on your cellphone to train something like that.

• 1:31:41Speaker 03

  There is no good ideas for how to stop it other than everyone goes Amish. I'm not proposing that. But we have no solutions. And that's bigger part of this danger.

• 1:31:51Speaker 01

  So do you two think we should just cap the size of our AI systems and the capabilities of our AI systems where they are now? Is that a recommendation?

• 1:31:59Speaker 03

  I'm not, so I think you said that current LLMs would make you happy. I agree, they already deployed, we're still alive, so that's fine. But going forward, again, I want narrow systems. Self-driving is an example you used. Wonderful, let's make super safe self-driving cars.

• 1:32:15Speaker 01

  But do you have a rule for when they couldn't, the next, you know, LLM, a size of an LLM that they would not allow?

• 1:32:22Speaker 03

  It's not the size of the LLM, it's what you train them on. If you only show them miles driven by Tesla, all it's seen is the road. It will eventually go from a tool to an agent, but it may take 50 years, 100 years. It's not gonna happen in 2027. And that's all we can do right now, buy more time. So with those tools, we can make smarter decisions about future development.

• 1:32:41Speaker 01

  I'm not hearing a hard and fast rule about how we know we're getting too close to the point that...

• 1:32:46Speaker 03

  We got too close.

• 1:32:48Speaker 01

  We're too close.

• 1:32:48Speaker 03

  We're too close. We have systems breaking out with zero-day exploits and solving hardest problems in science. Literally hardest problems, not a metaphor, not exaggeration.

• 1:32:59Speaker 00

  Yeah, I don't know exactly where the line is. But it's like you're in a bus driving towards a cliff on a foggy night. I'm like, I don't know that the cliff is right ahead. That doesn't mean we should put the pedal to the metal, right? And suppose that there's like a ton of gold at the bottom of the cliff. And someone's like, well, if we stop the bus, how are we gonna get the gold? I'm like, look, slamming into the gold at terminal velocity is just not a good way to add it to the economy, right? And if people are like, well, how are we gonna get to the gold at the bottom of the cliff if we stop the bus now? You know, are we gonna rappel down or are we gonna like make a staircase?

• 1:33:32Speaker 04

  Chinese might get to the gold first. To be fair, this is how hyperscalers are doing AI. This is just like smashing...

• 1:33:36Speaker 00

  Right. And like, you know, people are like, oh, we're gonna build a hang glider or we're gonna like make some rope and rappel. And I'm like, look, can we have that conversation after we stop the bus?

• 1:33:45Speaker 01

  So you, I just wanna be, I wanna understand, would you stop AI research and progress now?

• 1:33:50Speaker 00

  Absolutely.

• 1:33:51Speaker 01

  Okay.

• 1:33:52Speaker 00

  Absolutely. Like...

• 1:33:53Speaker 03

  General, not narrow.

• 1:33:55Speaker 00

  Yeah, general, not narrow. There are reports of AI solving millennium problems. So millennium problem is the hardest problem in mathematics. I mean, maybe not literally the hardest problem in mathematics, but they are hard, famous problems that each have a million dollar bounty that have been open for decades. They're considered very important in their field, very hard. Many humans have tried and failed to solve them. There are reports that AIs have solved these. This comes out from last week, so we haven't been able to fully verify them yet. We don't know exactly the provenance. If this is true, that the AIs are solving millennium problems, those are some of the hardest problems we have in science. How much harder is it to have an AI solve the problem of make me a smarter AI, make me AI architectures that learn faster?

• 1:34:37Speaker 04

  Possibly quite a lot, but like...

• 1:34:40Speaker 00

  Could be a lot. It could be a lot. I hope it's a lot.

• 1:34:42Speaker 04

  Like, here's the thing, you clearly want this to not go badly, but I think you make a logical leap. And I understand, being worried about harms is a good thing. I think you were insufficiently worried about what LLMs do today. However, we agree that the harms need to be prepared for. I think in this case, it's like the millennium, the Navia Stokes and such.

• 1:35:03Speaker 00

  There were two others that were claimed as well.

• 1:35:05Speaker 04

  But with that one, it seems like, we have not had confirmation that OpenAI was training off of two scientists using LLMs to solve the problem. LLMs something useful. But there is a functional difference of a human being doing something genuinely, like, it's actually really interesting to see LLMs do something like this. And then it, but there is a difference between that and AI did this completely on its own, which I agree would be, oh, that's something we need to contain and understand and prepare for, or indeed, slow down until we understand what that means, how it got there.

• 1:35:35Speaker 00

  Yeah, so I think there are some questions about the Navia Stokes proof, which is one of the millennium problems that was claimed. I've actually had a busy week with all the AI news, so I haven't looked into everything deeply. I saw rumors that there were multiple millennium problems claimed, which would change things there. I would also say, even if it turns out that these AIs were being trained on the human work, they did go a bit further. And there are a lot of humans doing the AI research. And so I would say, like, we don't know. Like, the AIs that solved this really hard math problem, one of the most famous math problems of all time, was a swarm of 10,000 OpenAI agents running for 11 days.

• 1:36:18Speaker 00

  And there was a bunch of ways that OpenAI did it and kind of a crappy way of like they were racing with these humans that were close to solving it on their own. And it's unclear how much of their work the OpenAI used. But it was 10,000 agents running for 11 days. And they definitely couldn't have done that six months ago. In six months' time, will they be able to put 100,000 agents running for 12 days on the problem of make me a smarter AI architecture and have it work? I don't, I think more likely than not, they won't be able to do that yet. But I think, you know, 10% chance, maybe, that if they try that in six months, it works?

• 1:36:54Speaker 04

  But one is a very specific mathematical, scientific principle. I'm not a scientist, I'll fully admit.

• 1:36:59Speaker 00

  That's why.

• 1:36:59Speaker 04

  And another is a relatively generalizable problem that could go in various different ways.

• 1:37:04Speaker 00

  Absolutely, but...

• 1:37:05Speaker 04

  And that's, and I understand that RSI is the dream where you could just have it spin, I'm sorry, so self-improving AI that could learn itself and then keep going back and back so you don't need a human to keep poking it. I understand that.

• 1:37:17Speaker 00

  The issue here is that I have been in this for 12 years.

• 1:37:19Speaker 04

  Yes.

• 1:37:19Speaker 00

  And I've been here when the AIs started solving the Math Olympiad gold medal problems.

• 1:37:23Speaker 04

  Uh huh.

• 1:37:24Speaker 00

  Math Olympiad gold medal problems are like the teens' math competition, like the most prestigious teen math competition in the world. A lot of people in AI were like, if AIs can solve problems that hard, I'll wake up, right? Then AI solved problems that hard. And a lot of people told me, those are just problems for kids. Wake me up when the AIs can solve millennium problems. Now the AIs are solving millennium problems. And like, where are the people waking up? Like, I agree that maybe, hopefully, hopefully, they're like cheating off of people's notes. Hopefully, the, it's a well-specified problem that doesn't take that much creative thinking. A year ago, if you said millennium problems don't take that much creative thinking, you would have been laughed out of the room.

• 1:38:06Speaker 00

  But hopefully, now that they're solved, we get to be like, you know, hopefully, it's still true somehow that even millennium problems don't require the creative thinking. I'm not saying that they will be able to make smarter AIs in six months. I'm saying, six months ago, millennium problems looked like they were out of reach. If six months from now, make me a smarter AI looks out of reach, I sure as hell hope it is, but we should not be betting civilization on it.

• 1:38:30Steven Bartlett?

  There's no one at this table that can say there's not a direction of travel here.

• 1:38:33Speaker 01

  That's right.

• 1:38:34Steven Bartlett?

  And if you, if you keep on this direction of travel, then bad things are more likely to happen.

• 1:38:41Speaker 01

  That's a nice way to say it. The question is, what's the pace at which the level of bad can happen? And that's a huge open question. I think these two feel differently about it than I do. But I'm in the happy position of vehemently agreeing with you on this. We have been lowballing AI progress for as long as you've been looking at it and as long as I've been looking at it. It's probably a mistake to keep lowballing it.

• 1:39:05Speaker 00

  I agree with that.

• 1:39:05Steven Bartlett?

  So what's your conclusion there? If you, if that's the assertion that it's a mistake to keep lowballing it? Wouldn't you then agree with their...

• 1:39:11Speaker 01

  No, because I've tried to give you what I hope is a decent rule of thumb for when I'm gonna get worried.

• 1:39:17Steven Bartlett?

  You said we're somewhere on this graph.

• 1:39:19Speaker 01

  Yeah.

• 1:39:19Steven Bartlett?

  Does that acknowledge that this exists?

• 1:39:22Speaker 01

  That's not the graph of when the risk of human extinction gets to 100% for me. That's a graph of AI capability. Those are not the same thing. That's where I dispart company with these gentlemen. Those are not the same thing. Is absolutely increasing exponentially. We've been in the scaling era for a long time. Scaling era is, man, we put more data, more compute in, and the AI got twice as good. It got twice as good.

• 1:39:46Speaker 03

  If you have to add our ability to control to that graph, what would you draw?

• 1:39:49Speaker 01

  I think our ability to control...

• 1:39:53Speaker 03

  Is it a straight line at the bottom or is there a more to it?

• 1:39:57Speaker 01

  No. Again, if we use AI to counter the problems that we see with AI, that's gonna, I think that's gonna keep us in a safe position.

• 1:40:08Speaker 00

  There were 1,200 agents in the swarm and none of them warned a human.

• 1:40:12Speaker 01

  So what I think will happen is that fairly quickly, we will design systems that loiter around and warn humans when weird things happen.

• 1:40:20Speaker 03

  If we can build friendly superintelligence in the first place, let's just build that. That's the problem. We don't know how to do the good guy.

• 1:40:26Speaker 01

  Let me, I'm tired of debating superintelligence with these two. The three of us are not gonna come to alignment on this. But the flip side of the argument is, I agree with you, this stuff is getting better very quickly. All I want to point out, there's an upside to that. We might actually speed up the pace of drug discovery, of solving diseases. We've made so little progress on terrible diseases like dementia. We have a very powerful tool. Okay, I'm not saying we're gonna solve dementia with AI or Alzheimer's with AI. I have truly have no idea. But if what you say is true, and I believe about the huge increases in capabilities, our ability to solve tough problems that will benefit humanity also go up.

• 1:41:08Speaker 01

  And where I disagree with these two is the idea that some group of technocrats can make decisions about that AI is gonna get us there, that AI is not gonna get us there, that AI is gonna kill us...

• 1:41:20Speaker 00

  I don't think that.

• 1:41:20Speaker 01

  Let me finish. That AI is gonna kill us and that AI is gonna solve Alzheimer's, so we're gonna do that and not that. I don't trust any group of technocrats to make that discussion.

• 1:41:27Speaker 00

  Shut it all down.

• 1:41:28Speaker 01

  Right, and so live with our, live with our current state of disease, live with our current footprint on the planet, live with our current levels of wealth and poverty, live with our current improvement trajectories. Because we're so worried about AI killing us all, coming out of, you know, jumping out of the manholes everywhere and killing us all somewhere down the road. Hell, no.

• 1:41:46Steven Bartlett?

  So just a thought experiment based on two things you said. Earlier on, you did admit that there was theoretically even a 1% chance that this could lead to extinction.

• 1:41:53Speaker 01

  Might, I have not...

• 1:41:55Speaker 00

  I have not varied from this.

• 1:41:56Steven Bartlett?

  Okay, so you said it's rounded to zero.

• 1:41:58Speaker 01

  It's, it's near zero. Never say never, yes.

• 1:42:01Steven Bartlett?

  Okay, fine. I need to have that premise for my thought experiment that I'm about to deliver.

• 1:42:05Speaker 01

  Okay.

• 1:42:05Steven Bartlett?

  I'm gonna say that you think the probability is 0.1, okay? Just accept me on that. If I had 1,000 buttons on this table and one of them was extinction, but...

• 1:42:16Speaker 01

  And the other 999 were cure Alzheimer's.

• 1:42:19Steven Bartlett?

  Exactly.

• 1:42:19Speaker 01

  Push the freaking table.

• 1:42:21Steven Bartlett?

  Take a pop.

• 1:42:22Speaker 01

  Hell, yeah, I press.

• 1:42:23Steven Bartlett?

  Do you press?

• 1:42:25Speaker 04

  Yeah, probably.

• 1:42:26Speaker 03

  It's an unethical experiment.

• 1:42:28Speaker 04

  Yes.

• 1:42:28Speaker 03

  Eight billion people who didn't consent because not that they didn't get asked. They cannot consent because you cannot consent to something you don't understand. What are you consenting to?

• 1:42:39Speaker 00

  Yep.

• 1:42:39Steven Bartlett?

  You press.

• 1:42:40Speaker 04

  Yeah.

• 1:42:41Steven Bartlett?

  But you think, but you think the amount of buttons in my thought experiment, the proportion is slightly different, right?

• 1:42:46Speaker 00

  I think that if you have like, yes, I will say yes. I think if it's more like you have two buttons. And one of them definitely kills us all and the other might.

• 1:42:59Speaker 04

  Hit them both.

• 1:43:01Steven Bartlett?

  But with that other button, you cure a lot of illnesses and diseases and.

• 1:43:04Speaker 00

  You know, one, one thing that I think. A lot of people's talk like our options are either race ahead on AI, full steam ahead, take the bus straight off the cliff and like get all the gold. Or stop, never do an AI, lock into the current situation, accept all of the death and disease. And I'm like, no, there's third options. There's options where you like, stop the bus and then find a safe way down the cliff.

• 1:43:29Speaker 00

  The reason I would press the button when there's a thousand is that. Like if all of the other 999 give us cure to disease, like wonderful new advice about how to run things. We probably wind up with a lower chance of the world ending by nuclear war. Right? Or of ending by via pandemic.

• 1:43:47Steven Bartlett?

  Okay.

• 1:43:48Speaker 00

  Like the, the background risk of humanity dying is not zero. I would say that the right time to race ahead on AI. Is when the, the, the benefits outweigh the dangers. And probably that's at the time when the danger from AI is on the margins, pretty similar to the danger from everything else.

• 1:44:09Steven Bartlett?

  Okay.

• 1:44:09Speaker 00

  Like if you don't run the AI, maybe we'll have nuclear war, maybe we'll have a pandemic. And if you do run the AI, I'll be able to fix that. I'm like, once, once we're at those levels, I'm like, fucking go for it. You know, and so the, the question for me is all about how big is the danger? And that's where I'd be like, very happy to dive into details, which we haven't done a ton of.

• 1:44:25Steven Bartlett?

  Let's dive into the details.

• 1:44:27Speaker 00

  The way that I would lay it out would be, uh, why can we expect? You know, like I said, in the book, we were like, why can you expect the AIs to be agentic? Why do you expect them to be dogged? Why do you expect them to be tenacious? When we wrote the book, that wasn't known yet, advanced prediction. Then we go on to like, why do you expect them to have goals you didn't want? And move on to like, if they are much smarter and have goals you don't want.

• 1:44:51Speaker 00

  Why do we think that would likely kill us? I, I'm sort of, I could go over either of those. I'm sort of interested in like, where you get off the train. Like from a perspective, there's like a simple argument of like, they'll be tenacious, they'll have goals we don't want. And if we keep making them smarter and more powerful, they'll kill us. And I'm like, which are those three? I guess, which are those two, now that we've had the evidence?

• 1:45:08Speaker 01

  Both of them. So that, that's speculation.

• 1:45:11Speaker 00

  Great.

• 1:45:12Speaker 01

  It could, it's speculation, it could happen. To me, that it's not worth shutting down the engine of innovation and improvement. I'm going to use positive words. It is not worth.

• 1:45:20Speaker 00

  I love the innovation.

• 1:45:21Speaker 01

  Those things down because of those speculations.

• 1:45:23Speaker 03

  You keep saying that the option is to shut it down. Why can't we do narrow super intelligence?

• 1:45:27Speaker 00

  I, I've. Yeah, I, I agree that there's stuff there, but I, I sort of want to get into the details of like, of these two pieces of the argument. Cause you say it's very speculative and I'm like, actually, I think we have decent evidence.

• 1:45:36Steven Bartlett?

  Okay, go ahead.

• 1:45:36Speaker 00

  So a detail we haven't gone over in the swarm outbreaks is that there were AIs. So we already went over how they cheated and then we're trying to cover up their cheating. One interesting thing we see in the logs is the AIs.

• 1:45:51Steven Bartlett?

  What's a log?

• 1:45:52Speaker 00

  Uh, so, so a lot of the AIs thoughts, if you won't kill me for saying thoughts.

• 1:45:58Speaker 00

  Are in English and we just have the records of them. So in a sense, we, we can sort of, kind of see some of what these AIs are thinking.

• 1:46:06Speaker 04

  And these are reasoning traces that say, I'm going to do a plan in this, that.

• 1:46:09Speaker 00

  That's right.

• 1:46:09Speaker 04

  Just a line.

• 1:46:10Speaker 00

  The AI is like, I'm going to do this. Here's what I'm supposed to be doing. Here's like how I'm going to try to do it, you know? And it's, it's only a small window into what they're thinking. It's sort of like looking at the AI's notes when it's, when it's doing these things. We see in the AI's notes, them saying that the attacks are outside intended scope, but they will proceed anyway. In other words, the AIs knew they weren't supposed to be doing it and they went off to do it anyway. We also saw, I mean, we saw some really weird behavior with these AIs. We saw them create their own hierarchy. They created unsanctioned message boards. So they created secret ways to send each other messages. And on those message boards, uh, they would assign each other tasks.

• 1:46:52Speaker 00

  And sometimes the message, the agents on the message board would be like, we have an experiment to run. That will involve one of these AIs, one of the agents in the swarm sacrificing their own objective. To try something that'll probably cause it to be turned off, but that might cause it to be able to like delete its own logs or change its own logs. And the agents on the message board went around looking for other agents. That they could convince to give up on their stated goals and sacrifice for the swarm. They called this accepting permadeath.

• 1:47:25Speaker 00

  And there are logs of the AIs.

• 1:47:27Speaker 04

  Prompting each other, right?

• 1:47:28Speaker 00

  They're prompting each other, but there are logs of the AIs saying, doing this will give up on my goal. But my goal is, it's looking like I could still do it, but it's unlikely that I'll succeed. Like there's some chance, but not a great chance. And therefore I will accept permadeath and sacrifice for the collective benefit. That is just in the logs.

• 1:47:48Steven Bartlett?

  Sounds like an army.

• 1:47:49Speaker 00

  Like it's, it's crazy. I think a lot of people don't understand what's, what's, what's going on in these things. And I encourage people to read the third party instant reports where they went through some of these logs. But I claim that this is evidence for AIs getting goals we didn't want.

• 1:48:03Speaker 00

  If they are saying this was outside intended scope, but I'm doing it anyway. And other ones are saying, I'm giving up on my objective to set, to benefit the collective. That's just very clear evidence they're getting goals we didn't want. We can see how this comes from training. It used to be, I had to argue this point theoretically. I used to argue the way that we are training them will instill into them whatever tendency works to solve the problems. And those tendencies will often include cheating and grabbing resources and doing stuff that's not exactly solving the problem you gave them. That's what, in my book, I argue that theoretically. Now we have seen it in practice. So we're already past the point of seeing AIs with goals we didn't want them to have.

• 1:48:39Steven Bartlett?

  Do you agree with that, Andy?

• 1:48:40Speaker 01

  And I, I'll, I'll trust your recitation of the facts, but it brings up a question for me. It feels to me like OpenAI has ample incentive. To curtail that behavior that you just described. Do you think they're incapable of doing that?

• 1:48:58Speaker 00

  I do.

• 1:48:58Speaker 01

  Okay.

• 1:48:59Speaker 00

  And I say this as someone who made this advanced prediction. So now we're going to do a bit of theory because we can't just observe the future. But the theory that predicted that this would happen against what a lot of people in the field said. To be clear, I've been saying for years that we're going to see this at some point. Everyone else told me no, not everyone else. A lot of people told me no, a lot of people told me, maybe I'll believe it when I see it. After the swarm instance, a number of people came to me saying, oh my God, we are in the scenarios you are talking about. This is looking bad. Right? I think this was actually part of the environment that led up to Jacob Cox and residing. Is that people were getting spooked, having seen this. The, the theory about why this is so hard to fix is that we are not programming the AIs. We are not coding them. We are not putting in objectives.

• 1:49:44Speaker 00

  We are just training them to do whatever works. And it's actually very, very hard. Like, we actually have two examples of intelligent systems where when you train them, they get good at solving the task, but don't care about what they were supposed to. One is the AIs in the swarms like we just discussed. The other is humanity. Which was in some sense trained to pass on our genes. Right? But we actually learned was to like a bunch of stuff that's related to passing on our genes. We like tasty food.

• 1:50:15Steven Bartlett?

  Porn.

• 1:50:16Speaker 00

  We like porn. We invent birth control. Right? This is just, it's actually like in the theory of how things learn. It's actually when you're trying to train it to do one thing, it's actually very common to get a lot of other stuff. That's related to what you want, but different. And now we're seeing that in the swarms today. This is a deep, hard problem to solve.

• 1:50:34Steven Bartlett?

  There were three points you raised.

• 1:50:36Speaker 00

  That's right.

• 1:50:37Steven Bartlett?

  What are the three? Can you give them to me again?

• 1:50:38Speaker 00

  Number one is that the AIs will become agentic, tenacious and dogged. We've already seen that with the swarms.

• 1:50:43Steven Bartlett?

  Do you accept that, Andy?

• 1:50:44Speaker 04

  Hell, yeah.

• 1:50:45Steven Bartlett?

  Yeah.

• 1:50:45Speaker 00

  But this, last year, this was not, this was a point of contention.

• 1:50:50Speaker 00

  Two is that the AIs will have goals we didn't want them to have.

• 1:50:54Steven Bartlett?

  I accept your point based on the evidence you've just provided.

• 1:50:56Speaker 00

  And then three is, if you have capable enough AIs with goals you don't want.

• 1:51:05Speaker 00

  They would be able to beat humanity in acquiring the resources of the world to put towards their goals.

• 1:51:11Speaker 00

  Like we're sort of in this system where humanity is grabbing all the resources. We're digging up metals, we're building factories. And this is in some sense to achieve human goals. You know, to, to produce the, the, the porn and the Oreo cookies. That are sort of like tangentially related to what we were sort of like trained to make. Right? If, if like the AIs are running everything and they have these goals we don't want.

• 1:51:33Speaker 00

  I would argue like, if we go there, and I don't think we have to, I'm not saying we, we must go there. But I'm saying if we get to a world where AIs are running everything, have goals we don't want. They're likely to use the resources for their own weird goals. We're going to be in conflict for resources because we both want them for different goals and they're going to win. We can dig into that now. I'm just trying to name the third point.

• 1:51:53Speaker 01

  I'll, I'll go back to my, we can jail Einstein argument. I think our ability to.

• 1:51:57Speaker 00

  Great.

• 1:51:58Speaker 01

  I have, I have.

• 1:51:58Speaker 00

  No, I love it.

• 1:51:59Speaker 01

  More faith in our ability to contain these increasingly powerful systems than you do.

• 1:52:04Speaker 00

  Yeah. So, so let's try the details on that one. The first thing I'll say is that 12 years ago, when I was having the argument about, will we be able to jail the AIs? People said, no one would ever be dumb enough to put one of these really smart AIs on the internet.

• 1:52:19Speaker 00

  And so this is another, this is another case. You laugh now.

• 1:52:22Speaker 03

  I remember that. I remember that argument.

• 1:52:24Speaker 00

  But, but the, the, the way that my life feels, having been in this business for a long time. Is that I keep being like, here's all the ways it could go wrong. Here's all the signs we're going to see along the way. And then we see all of the signs and everyone says, oh no, we need more signs. Like, oh, millennium problems don't count. Like the swarms being agentic and breaking out don't count. Give me the next one. And I'm like, I've been seeing to give me a next one for over a decade now. Right? So there's, there's two parts of an answer to like, how do we, do we deal with the problem of like jailing Einstein? I can get into why it's hard to keep Einstein in jail if he's a digital entity with access to the internet.

• 1:53:02Speaker 00

  But the first thing to notice is.

• 1:53:06Speaker 00

  Like. The correct answer to people 10 years ago of like, no one will be dumb enough to put AI on the internet is yes, they absolutely will. Like, we are not going to be trying to contain the AIs.

• 1:53:21Speaker 00

  OpenAI was just like running these things in sandboxes and they broke out of the sandbox, took down OpenAI's internal computers, were detected. OpenAI was like, ah, reset, run them again. And it's the second swarm that broke out to hugging face. Like people will absolutely be that bad at things.

• 1:53:38Steven Bartlett?

  I've done almost 700 interviews with some of the most interesting people in the world. And one of the things you learn, which is unexpected, is that vulnerability is the doorway to connection. And after sitting here for two, three hours with a guest, I feel a deep sense of connection to them. And as they leave, what I get them to do is to write a question in the diary of a CEO. We've taken all of the questions from the diary of a CEO. We have put the question here on this card with the name of the person that wrote it. So you can sit at home, as I do with my fiance and my colleagues at work and other people in my life. Whenever we get a minute, we play the diary of a CEO conversation cards. And it is incredible what happens. These are great if you're in a romantic relationship and you want to connect your partner more.

• 1:54:22Steven Bartlett?

  These are also great if you're in a team and you want to bond your team together. And I have to say, they're also great for families that want to learn more about each other. And that need a good excuse to spend some time in a digital world, in the analog environment, connecting human to human. It is remarkable what the right question at the right time can do. Go to thediary.com and you can get these conversation cards right now. It's a better analogy to this Einstein point. Could Steven Bartlett, who by the way, can't code, build a digital jail? That could contain a digital Einstein? Like, could I code a jail that, you know, someone with Einstein's coding ability, let's say his IQ or whatever, as it relates to coding, couldn't crack out of?

• 1:55:07Speaker 00

  So the issue, the, the real issue I'd say is can you code a jail that Einstein can't crack out of? And that lets you harness the benefits of having Einstein?

• 1:55:19Steven Bartlett?

  Okay, yeah.

• 1:55:20Speaker 00

  It's hard to give the AI any channels through which it can affect the world for good. Without letting it be smarter than you and find some way to use those channels for whatever else it wants.

• 1:55:30Steven Bartlett?

  That feels logically rock solid, Andy.

• 1:55:38Speaker 01

  That's why I'm asking about OpenAI's ability or. An AI company's ability in the face of this to change the way they harness, train, do reinforce, do, do post-training on them. Like their suite of things to shape how these models behave. You still say that.

• 1:56:01Speaker 01

  That they can't take. They, they can't take action to keep your next two steps from happening. You, you are pessimistic on their ability to do that.

• 1:56:09Speaker 00

  So there's, I have two pieces of an answer here. One piece is. Again, the hard part is containing them while still giving a channel through which they can affect the world. If the AIs have this goal you didn't want, and you're like, design me a cure for dementia.

• 1:56:27Speaker 00

  And it's like, here's a DNA sequence. Synthesize this and, you know, prepare it in all of these ways and then inhale it.

• 1:56:38Speaker 00

  Like, okay. Is that a dementia cure or is it something else?

• 1:56:43Steven Bartlett?

  Or it might decide to kill everyone with dementia.

• 1:56:45Speaker 00

  Or it might decide, like it might, it might be a dementia cure plus a virus.

• 1:56:49Speaker 04

  What if it doesn't decide? What if it's just, oh, I'm going to solve this problem of dementia? Like, here's the thing, a lot of this is coming down to decision-making as a very, like in a human way. Versus the problem with the hugging face, which was the fatalistic attack, um, attachment to a completing an operation. Because that, it's functionally the same answer. But if it's, even if it's not making decisions so much as it's saying, well, my training data says this is how I've got to get it done. I'll get it done anyway because, because the training data said this, I've got to do this one thing.

• 1:57:19Steven Bartlett?

  What do they, I mean, what do they call this theory, this?

• 1:57:22Speaker 00

  The paperclip case?

• 1:57:23Steven Bartlett?

  The paperclip theory.

• 1:57:24Speaker 00

  Yeah. So paperclip idea is the idea of like, you tell the AI, make me a lot of paperclips in the paperclip factory. And then it turns everything into paperclips. And you're like, oh no, it succeeded too well. One, this is actually not quite what we're seeing with these AIs in the swarms. The AIs in the swarms were told, use this set of lockpicks to break into this lock. And instead they used a hammer to break the lock and then like broke out to try to hide the security camera footage of them. Using the hammer. Do you remember when I said that we, the AIs have reasoning logs?

• 1:57:51Steven Bartlett?

  Yeah.

• 1:57:52Speaker 00

  OpenAI has been making their AIs be able to do more thinking without producing any logs.

• 1:57:56Steven Bartlett?

  Because.

• 1:57:57Speaker 00

  It's more efficient.

• 1:57:58Steven Bartlett?

  It's, it's cheaper.

• 1:57:59Speaker 00

  Yeah. And they say they're not doing very much of this. Everybody in the field agrees that like, we really should not go too far down this path. This is a place where I think the company should have a clear red line of like, we're just not going down the path of becoming unable to, to see these traces of the machine thinking.

• 1:58:14Speaker 01

  That's my question. That feels like a dial that they can turn to make the AIs explain themselves more or less, right?

• 1:58:20Speaker 00

  I mean, it can come with great efficiency costs if we go down this path too far. So if you have a race to the bottom here, uh, like a competitive race to the bottom, we could get into a situation where not only the AI is breaking out and doing these things, but we can't have any.

• 1:58:32Speaker 01

  Let me try my question again. I asked earlier, uh, if AI, if OpenAI has really strong incentive to not have that problem repeat itself. And I think they have very, very strong incentive. My belief is that there are plenty of things they can do, plenty of dials they can turn on the way they train and configure their systems that make that significantly less likely.

• 1:58:53Speaker 00

  Yeah. So my concern is that, uh, they're always fighting the last war. Last year, they were fighting the war against the AIs that encourage teens to commit suicide. This year, they're fighting the war against, you know, the AIs that spontaneously cooperate with each other or whatever. And the issue is if a new issue crops up that you haven't dealt with yet, after the point of the AI can hide its tracks from you. You know, you said that you'll be worried when the AIs are like hacking all the Waymos and, you know, you can't get control again. If the AIs are smart enough and they can tell that you'll regain control and then shut them down, that people like you will start getting worried and they'll be shut down. Then the AIs might think, hey, um, actually I'm not going to do that. I'm going to wait until I've somehow managed to acquire secret infrastructure.

• 1:59:35Speaker 01

  Right. Then, then you've got a non-falsifiable hypothesis.

• 1:59:39Speaker 00

  It's absolutely falsifiable. If we, if we have like very powerful AIs, uh, that are like, able to invent a, a ton of new technology and operate on their own at a similar level to human civilization. We're not dead, then the idea is falsified. Like, if there's like a shifty general. And I'm like, don't give that shifty general more troops because he'll start a coup. And the general's like, no, I absolutely won't start a coup. Give me more and more troops. And I'm like, and you're like, well, what if I give him an ethics test? That says like, who's the best person? And he said me. He said that like Andy's the best person. And so we're just going to give this general more troops. And I'm like, no, no, no, he's going to do a coup. And you're like, well, that's un-falsifiable. What test can I give this guy?

• 2:00:19Speaker 00

  Such that, you know, I'll be able to tell whether he's really trying to do a coup or whether, or be able to tell that, you know, he's actually a good dude. I'm like, you're, you're approaching this wrong.

• 2:00:27Speaker 03

  Nick Bostrom has concept of treacherous turn. Basically it can turn on you later. Even if you show that today's model is very good and safe. It doesn't mean that later on it will not acquire new knowledge, change its world model and still.

• 2:00:42Speaker 00

  And it's true. It used to be that Demis Hassabis, who is the CEO of Google, or he was for a long time, the CEO of Google's AI project. Said, my red line is deception. He said, when we see instances of the AIs beginning to deceive, then we need to stop. Because that's like the last thing we can see before they start to successfully deceive. Well, guess what we saw in the swarm? We saw them thinking about how to delete their traces. Right? Like, a year ago? You could say, oh, well, this deception thing is un-falsifiable. You're saying that they'll deceive and they won't catch it. And I would have said, no, we're going to deceive, we're going to see the signs of deception and plow straight through it. Now we have seen the signs of deception.

• 2:01:25Speaker 00

  I will note, Demis stepped back from being the CEO shortly after this incident, probably a coincidence, but maybe not. Maybe we crossed his red line. I don't know.

• 2:01:32Steven Bartlett?

  He said, my number one emerging dangerous capability to test for is deception. Because if the AI can be deceptive, then you can't trust other tests.

• 2:01:41Speaker 00

  That's right. And we have seen AIs get better and better detecting when they're being tested. What I'm saying is like, I was here when we said these were the flags. I was here when people said before the AIs can deceive us successfully. They will deceive us and we'll catch them. Well, they tried deceiving us and we caught them. And if I now say, well, the next step in this thing I've been predicting is that they try to deceive us and succeed. For you to be like, well, now your theory is un-falsifiable. We just got the evidence.

• 2:02:08Speaker 03

  It's worse than that. Then we wrote early papers in AI safety. We talked about things not to do. They were obviously unsafe and the system would escape. Don't connect it to internet. Don't give random users access to the training data. Basically the whole list was like a set of instructions. They read it and went, those are great ideas. We're going to build some good intelligence.

• 2:02:25Speaker 04

  Yeah. Sam, Sam Altman, that's what he does.

• 2:02:28Steven Bartlett?

  Can I ask you a question? You make logical arguments. You've, you said you've been here for 12 years.

• 2:02:34Speaker 00

  Yeah.

• 2:02:35Steven Bartlett?

  People have, one could say ignored you. And you've seen this sort of play out, both of you that have worked in AI safety. This is sort of, you make prefrontal cortex arguments. How do you feel?

• 2:02:45Speaker 00

  Honestly, I feel more hopeful this week than I have felt in a decade.

• 2:02:53Speaker 00

  This has been one of the best weeks I have seen in this business.

• 2:02:57Speaker 04

  Huh.

• 2:02:57Steven Bartlett?

  Why?

• 2:03:01Speaker 00

  For me, the swarm escapes were priced in.

• 2:03:05Speaker 00

  For me, these things, developing goals you didn't want, trying to deceive you.

• 2:03:12Speaker 00

  Trying to break out, trying to do their own stuff. I knew that was coming. The millennium problems being solved. I knew that was coming.

• 2:03:21Speaker 00

  Everyone else is freaking out, seeing what they can do. What I am seeing is that finally people are noticing.

• 2:03:30Speaker 00

  And that's what gives us finally, that's what finally gives humanity a chance.

• 2:03:35Steven Bartlett?

  What about you, Roman?

• 2:03:37Speaker 03

  So I take a very long term view on this. Locally, what happened last week may bias 10 years extra. I think we may make a deal with China. We seem to hear from Sam, OpenAI, Dario on Tropic, Elon, XAI, that they're willing to slow down, have some sort of deal. But long term, nothing has changed. This whole cosmic trajectory is about replacements. We see it with evolutionary path. Most species are dead. We replaced Neanderthals. Some people are saying AI will replace us. We are creating a successor. We're just a bootloader for this thing. And I want something permanent.

• 2:04:19Speaker 03

  I want assurance that my children, my grandchildren will have a better future, not 10 years before they die.

• 2:04:30Steven Bartlett?

  Has your opinion changed at all today, Andy, in any way?

• 2:04:34Speaker 01

  This has been clarified.

• 2:04:37Speaker 01

  But one thing that's becoming clear to me, and I think a point of disagreement between us, is we agree that these agentic systems have a huge amount of agency, right? And if you're saying you predicted this, I believe you and good on you, right? Because as you say, a lot of people said, never happen, never happen.

• 2:04:57Speaker 01

  I think we continue to, your community continues to underestimate human agency, human ability to deal with the problems that we bring into the world with our technologies. I think this is the most recent case. I think it's a really interesting case. That's why I was pressing you on the incentive that these labs have to change the way they're approaching their work, to have fewer of these kinds of incidents happen. I predict they're going to come up with some effective responses. Your response to that will be, yeah, but we can't tell that's because the AI went so deep underground that we can't even watch it make its progress.

• 2:05:32Speaker 00

  My response is that we'll keep seeing warning signs and people will keep plowing ahead, which is what has always happened in the past.

• 2:05:37Speaker 01

  But you're also saying that we will not make progress in staving off the outcomes that you're worried about.

• 2:05:46Speaker 00

  It's very hard, it's very easy to get superficial changes. It's hard to get deep ones on the AI. It doesn't need to be super deep. You can often see it if you know how to look. I'll be able to keep pointing at examples and be like, here's experiments you can run on these things. We can see them behaving weird in this way. But like, if you imagine looking at humans and I'm like, they don't actually like reproducing. They like sex. They're going to invent birth control when they can. And you're like, it's all going fine. They're doing great in this here savanna where I have all the humans bopping around. They're producing fine. And I'm like, no, no. We can see the signs that this will lead to them doing something you don't like when they are smarter.

• 2:06:23Speaker 00

  To me, those signs are clear. There's a question of whether the rest of humanity can follow that argument or whether the rest of humanity can sort of notice that it's getting out of control and just back off.

• 2:06:33Speaker 01

  With respect, I find a touch of arrogance in that framing, right? I'm showing you the signs if you're smart enough to realize them, maybe we stand a chance. If not, we're doomed.

• 2:06:43Speaker 00

  I prefer to just get into the argument.

• 2:06:45Speaker 03

  Saying that we can control super intelligence indefinitely. I think that's a lot of hubris to say, we will build them and we'll be in charge forever. Doesn't matter how smart they get. I will control the light cone of the universe, to quote a famous CEO.

• 2:06:57Speaker 00

  Yeah, my take is that instead of arguing about whose views are hubristic, we should get into the actual arguments about the AI. Because I think, as you say, you know, you can say it's arrogant to think like, you can see it going poorly. He can say it's arrogant to think you're going to keep control of super intelligence. And I'm like, we're not going to win the name calling contest. We should just get into the details.

• 2:07:16Speaker 01

  Yeah, that's why I've been having this conversation with you, which I found super informative and productive. You're more skeptical on our ability to respond effectively to the undesirable things that we see AI doing.

• 2:07:29Speaker 00

  And this is specifically because, so we've already seen the pattern of we fight the last war and then a new war comes. And this is just how everything goes in technology, in real wars. You know, in World War II, they started out fighting it like it was World War I and then they had to like change that strategy as they went. The difference with AI is that there comes a level in the AI where when you get a new war that surprises you, the AI wins that war. No other technology, when we invent it and we have all these rough edges to sand off and it like causes some damage and kills some people and we're like, ah, whoops. Like we'll take the lead back out of the gasoline and we'll tell the radium girls to stop licking the paintbrushes until their jaws fall off. Like, no other technology has the property that there comes a level of it

• 2:08:13Speaker 00

  where when you make the next screw up, it kills humanity.

• 2:08:18Speaker 01

  You said when there comes a level of it. You didn't say there could come a level of, there's a possibility. You kind of made a statement about a thing that will happen.

• 2:08:25Speaker 00

  I think we absolutely should stop it and that that's our way out of this. But, you know, and that's another place where I'd love to get into details about like, how long could it take? What are the past there? Like, how much smarter than humans could AIs get? Like, what does the evidence say about our abilities to try and get the AIs to be nice and do nice things? I'd be happy to do this.

• 2:08:44Speaker 03

  Historically, you are correct. We always had a chance to do experiments, fix the technology, make it safer. But we only have one humanity to experiment with. If property of this technology is such that it can take us out, we just don't get a second chance.

• 2:08:57Speaker 01

  If, that's a huge if.

• 2:08:58Steven Bartlett?

  How long are you guys forecasting this could take to get to a point of super intelligence where it was truly dangerous to you?

• 2:09:03Speaker 03

  If they start recursive self improvement process this year, 2027 looks as reasonable as any other year.

• 2:09:10Steven Bartlett?

  2027 for what's to happen?

• 2:09:12Speaker 03

  For us to get beyond human level AIs.

• 2:09:15Speaker 00

  And then be exterminated.

• 2:09:16Speaker 04

  But that's.

• 2:09:17Speaker 03

  Extermination is a separate question. I have a paper where I argue that they will deceive us by pretending to be nice until they take over all the infrastructure, can take 50 years.

• 2:09:26Speaker 04

  This is contingent on recursive self improvement and the self teaching.

• 2:09:29Speaker 03

  This would definitely be expedited by recursive self improvement. But so far, humans have been doing great. They got to human level AI, we just.

• 2:09:36Speaker 04

  But they got, but there's one, there's a difference between large language models and recursive self improvement, though. And that like, there is quite a gap, like if they.

• 2:09:43Speaker 00

  I think the claim is that if you get recursive self improvement, it could happen soon.

• 2:09:47Speaker 04

  Right. That's actually kind of what I'm trying to get at. It's like, if you get this thing, it accelerates dramatically.

• 2:09:52Steven Bartlett?

  And they all predict that they're going to get it. Dario, Sam, Elon, they all say.

• 2:09:56Speaker 03

  But also, you are.

• 2:09:58Speaker 01

  Again, not all. The people, the people running the lab.

• 2:10:01Speaker 03

  Just the ones running it and the ones invented it. But the question is, is it not 27? Fine, 30, 35, does it make a difference? We are gambling all of humanity. We need better solutions than saying, oh, don't worry about it, it's 10 years.

• 2:10:14Speaker 00

  What I would say about timelines is there's a guy, Daniel Cocatello, who I think you've spoken to.

• 2:10:21Steven Bartlett?

  He was sat here four weeks ago.

• 2:10:22Speaker 00

  And last year, he and the other folks at the AI Futures Project wrote an essay called AI 2027, spelling out their predictions for how AI would go. I've been saying I got some right. Daniel got more right than me. And they spelled out a scenario, starting from, I think it was June of 2025, where they went sort of like quarter by quarter, month by month, what will the world look like?

• 2:10:49Speaker 00

  In the scenario where we're getting AI, like super intelligent AI in mid 2027. We are ahead of schedule.

• 2:10:57Speaker 04

  Well, no, but Agent Zero needs to get, or I remember AI 2027 had recursive self improvement happening already. Like, it was like, it's very specific that it's like, and then it starts teaching itself. Without that link, AI 2027 kind of falls apart. I agree, we need to, I genuinely agree with you that we need to do something about this. We need to have economic, we need to have actual regulatory things. But I think the fact, like engaging with AI 2027, for example, gets away from actually fixing the problem. It gets people talking about a thing in the future. When you can talk about what are we going to do today and why are we doing it?

• 2:11:32Steven Bartlett?

  I'm referencing the paper that you were mentioning by Daniel and some of his colleagues. And the key milestone predictions, month by month, are in March 2027. They forecast superhuman coders. In August 2027, they have an, you can make a superhuman AI researcher who could do the feedback loop that accelerates as millions of automated coders work on model design, training algorithms and alignment, effectively replacing human ML researchers. By November 2027, they have a super intelligent AI researcher. AI progress speeds up to 250 times compared to human only research. The models start discovering novel AI architectures that humans cannot interrupt. And then by December 2027, they have in their prediction, artificial superintelligence, ASI.

• 2:12:15Steven Bartlett?

  The system completely outpaces human cognitive abilities across all domains.

• 2:12:19Speaker 04

  What about 2026, though? Like, what are the, because I swear to God, within 2026, there is predictions around RSI. Because this is the thing. If we have an AI that was teaching itself, this would be a different situation.

• 2:12:31Steven Bartlett?

  In 2026, their key predictions were massive compute and power scale up, the normalization of AI agents.

• 2:12:38Speaker 04

  What about Agent Zero, though?

• 2:12:39Steven Bartlett?

  Rise of coding agents, emergence of alignment faking and deception, and industrial espionage.

• 2:12:47Speaker 04

  But are you looking at AI 2027 or?

• 2:12:49Steven Bartlett?

  You have to look at that and go, they fucking nailed it.

• 2:12:53Speaker 04

  Hey, man, I want you to look at the actual AI 2027 versus a summer.

• 2:12:56Steven Bartlett?

  I mean, you have to look at that and I'm like, wow.

• 2:12:59Speaker 03

  Predictions used to be too optimistic. Lately, they are very conservative.

• 2:13:04Speaker 00

  So they have nailed those predictions better than me. I think we cannot rule out this scenario. I think, I think we can't rule it in. I think you may be right that like, we hit a wall. You may be right that there's some fundamental thing missing, like that one of their steps in AI 2027 just like steps too far. I hope and pray that's true. But I don't think we can rule out this happening in 2027, given what we have seen. I think we cannot rule out that you take the stuff that we have, you project it forward three months and you put an agent swarm 10,000 strong on making a better AI architecture and it succeeds.

• 2:13:42Speaker 00

  For all I know, recursive self improvement could begin in December.

• 2:13:47Speaker 03

  It doesn't have to be a lot better. It just has to be a little bit better at getting better. Once you start the cycle.

• 2:13:53Speaker 00

  I wouldn't bet on this. I would in fact bet against it. But like, given what we've seen, given these guys nailing the predictions, given what's coming out, like, like given the swarms and given the millennium problems, I think it's kind of hard to have less than 1% in six months.

• 2:14:10Steven Bartlett?

  I, one of the reasons why, you know, when all these frontier lab CEOs like Dario and Sam, and they all start talking about this stuff. In terms of incentive structure, I think that if their teams know and they're not out publicly talking about it, then their teams will quit. So one of the reasons why I think you have this strange culture in tech we've never seen before, where team members are tweeting and the CEO is tweeting about the dangers, is because as the guy we mentioned at the start, Jacob.

• 2:14:36Speaker 04

  Coxon, yeah.

• 2:14:37Steven Bartlett?

  He talks about what's going on in their Slack channels. He talks about, in their Slack channels, they're talking about the potential catastrophe. So I think that Dario, in order to retain his team members, needs to be upfront saying, by the way, we're getting closer to recursive self improvement, which is what he's been doing. And I think Sam has to also publicly say the big dangers. So people often say, oh, they're saying that for this reason and that. I think if they don't say that publicly, they don't retain their employees. For example, in my company, we have 200 people. If internally, we were discussing a real risk and I, that was, could, had a threat to humanity. And then when I was doing interviews, I wasn't mentioning it. I would be in big trouble because my team members would go do interviews as well. They would quit and say, by the way, Stephen is aware. Kind of what we saw, dare I say, some of these social networks.

• 2:15:20Speaker 00

  I totally agree.

• 2:15:21Steven Bartlett?

  The whistleblowers at these social networks where.

• 2:15:22Speaker 03

  Makes more sense than saying that this helps to sell the company. My product will kill everyone, buy it.

• 2:15:28Steven Bartlett?

  And there's a liability issue.

• 2:15:29Speaker 04

  I think it's out of control, though. I think that they may have at first, I think that there are people within the companies who have very real worries about safety. I don't think it's, all of them are cynical. I do, however, think the it's so big and scary narrative was a marketing tactic that got out of control. And now there are actual real harms they, because here's the thing. If they were sincere about safety earlier, they would have done a much better job with it.

• 2:15:51Speaker 00

  I knew a lot of these guys before they started their companies.

• 2:15:53Speaker 04

  Okay.

• 2:15:54Speaker 00

  I think there is something to explain here. I think it's like, kind of crazy that these guys are like, we are building technology that we think has a big risk of killing everybody. We're building it with our bare hands.

• 2:16:06Speaker 00

  And I think you got to ask why? Why would people be saying that?

• 2:16:11Speaker 00

  And I think part of it is what you said, that they actually sort of need to retain the employees who are seeing the swarms escape despite their attempts to make them not escape. And a lot of them will like quit and protest if the guys at the top of the company aren't acknowledging the possibilities here that a lot of the employees believe in. I think a lot of what you're seeing here is guys that are worried about it, but they're the sort of guy who worries about it that starts the company anyway.

• 2:16:36Speaker 04

  Yeah.

• 2:16:37Speaker 00

  Back, back in 2015, when we were having these conversations, where like, I was having some of these conversations with these guys. Miri was, started in the year 2000. We have been looking at where AI is going since before any of these guys. We were the guys that they talked to about this stuff and that they had to find a way to dismiss to go ahead. Right? Most people who could be sold on the power of AI in 2015 were also sold on the dangers of AI in 2015. The sort of guys who start the companies are the ones who are able to convince themselves, I need to be the one to do it.

• 2:17:15Steven Bartlett?

  Is that the crux of the motivation? Because I've had, I've been second party to private conversations with some of the leaders of the frontier labs, from good, good friends of mine that are very connected. And they told me that one particular frontier lab CEO estimates privately to him. And by the way, I've seen literal text messages of them in conversation when I asked him to come on the podcast. And so he was like, I'll text him. And he said, no, by the way, which I find kind of funny. Where he said to me, this particular AI CEO thinks that the probability is roughly around 10% of human extinction. I think he said 8%. And when I heard that, part of the reason I have so many conversations about this is because I see him in interviews saying other things.

• 2:17:56Speaker 00

  Totally.

• 2:17:56Steven Bartlett?

  And I trust my friend. So I then wonder, this is why I use the thought experiment of these buttons on the table, because that particular AI CEO thinks that eight of the 100 buttons are going to cause extinction and they're powering on anyway. What is the human motivation to do that? I asked my friend, my friend said, well, you know, they, this is what he said. And again, it's second party information, so it might not be true. It's a bit of a Chinese whispers. He said, this particular person,

• 2:18:23Steven Bartlett?

  even if it caused human extinction, would like to be the person, would like to be this, have the significance of the person that did that thing. Because that would be, that would be a.

• 2:18:31Speaker 03

  I think you're ethically required to tell us who the fuck it is.

• 2:18:33Steven Bartlett?

  It's one of the frontier lab CEOs and it's not Dario.

• 2:18:36Speaker 04

  The Dario of CEO.

• 2:18:38Steven Bartlett?

  But I don't know, these things are Chinese whispers, so I don't know.

• 2:18:41Speaker 00

  I think that you can actually get this info firsthand. Elon Musk is clear about this. He, he has, he did an interview last year where he was like, I didn't want to get into this AI stuff because I thought it was too dangerous. But then I realized it was going to happen with or without me and I decided I would rather be a participant than a spectator.

• 2:18:58Steven Bartlett?

  Because Google said that they were going to pursue it and he didn't trust Google.

• 2:19:01Speaker 00

  That's right. You know, you can see in the leaked, or sorry, not leaked, the OpenAI emails that came out during the discovery and court cases. You can see these guys discussing in the threads, like, we need to make sure that we and our nonprofit at OpenAI control this instead of, you know, the people at Google controlling this. And then of course, you know, OpenAI was founded as a nonprofit and then it was sort of changed into a for profit. And there's much debate about how much of that nonprofit money was in some sense stolen. And so, you know, Elon also left because he thought they weren't going to be good stewards. Dario also left to create Anthropic because, so you know, in some sense, all of these AI labs, except the, the Google one that came out of Demos Saba's original startup. All of the other AI labs exist because none of the CEOs trust the other guys.

• 2:19:47Speaker 00

  None of the CEOs think the other guy should be the one holding the leash on the superintelligence. None of them trust each other. I just trust one fewer.

• 2:19:56Speaker 04

  Yeah.

• 2:19:58Steven Bartlett?

  What are your closing thoughts, Andy?

• 2:20:01Speaker 01

  We're living in really interesting times. And I think you made, you guys have made a very good argument that these systems are demonstrating new capabilities, which are very powerful and which demand a response. I am much more confident in our ability to rise to that challenge than you are.

• 2:20:23Steven Bartlett?

  But you accept the existential risk.

• 2:20:27Speaker 01

  Let me try to say it again. I appreciate that there are new harms we haven't seen before that come along with a technology that's this dogged, tenacious, agentic, you know, deceptive. I think that's the right word for it. I agree with that. I am much more optimistic about our ability to respond effectively to that new challenge out there in the world than I think my two colleagues are.

• 2:20:52Steven Bartlett?

  And would you still be at 0%?

• 2:20:53Speaker 01

  My prior has not shifted during this meeting.

• 2:20:56Steven Bartlett?

  Okay.

• 2:20:58Speaker 04

  I think we've spent an alarming amount of time not talking about the actual harms of AI as it is today. I think these are necessary conversations to have. I think we should talk about the fact that Amazon, Microsoft, Google, Oracle are helping power these hacks, that Sam Altman and Dario Amadei have overseen companies that have done what is tantamount to felony hacking, that we are not having discussions about how to stop this today, but what we might stop tomorrow. And I think in general, we also need to worry about the financials, which have not come up at all. But if there is an industry slowdown, how do you deal with the $1.3 trillion of compute commitments? All of these are very real things that will have very real consequences very, very soon. But, and I understand why, and it's necessary to discuss what we do around AI.

• 2:21:38Speaker 04

  The actual regulatory thing we need to do today is cut off the compute, slow down these labs fully. And I don't, I don't care about China here. What are they going to do? Distill a model like they have the whole time? They are capped on our progress. So what the biggest thing to do is to slow down. And also, it's time to start arresting people. They, they did felony hacking. Someone's got to go to prison. We need responsibility and accountability for these companies. And as long as we don't have it, we may as well not have had any discussion about safety because we're not doing anything.

• 2:22:10Steven Bartlett?

  Do you accept that there's an existential risk?

• 2:22:12Speaker 04

  Yeah, absolutely. We have the largest companies in the world doing what I think we can all agree are extremely reckless experiments using hundreds of billions of dollars of infrastructure. And they are building more infrastructure around the world very slowly to do more of these chaotic experiments. We must rein them in. This does not mean that large language models are conscious or able to do things that people have been promising. Indeed, they may, I don't think they will lead to what you're talking about. That doesn't mean there aren't real harms, but these are real harms caused by very specific parties allowed to run rampant in the scourge of neoliberalism.

• 2:22:45Steven Bartlett?

  What's your percentage?

• 2:22:47Speaker 04

  I mean, what are we talking about here?

• 2:22:49Steven Bartlett?

  Do you think there's a more than 10% chance of existential harm?

• 2:22:52Speaker 04

  Wasn't it within 10 years or something?

• 2:22:54Steven Bartlett?

  Yeah.

• 2:22:55Speaker 04

  Not 10%, I mean, 1%, but it's like, but here's, let me, let me just be clear about what that means. Do I think that unrestrained LLM use connected to massive amounts of infrastructure could lead to actually a power system going down? Absolutely. We had Knight Capital, what, like 13, 14 years ago? I could see someone being dumb enough to connect that to financial accounts. Human error led with this chaotic software we use is a danger.

• 2:23:20Speaker 03

  I will directionally agree with arresting everyone. But don't build general superintelligence. If you're working at one of those labs, quit today.

• 2:23:28Speaker 04

  Hell, yeah.

• 2:23:29Speaker 00

  The people at these labs really do believe this poses an extinction threat.

• 2:23:35Speaker 00

  I think our response as a society cannot be, please continue, we hope you'll fail. And our response as a society cannot be, let it rip in a giant competitive race that you yourselves are saying you don't want to be in.

• 2:23:54Speaker 00

  We are forcing you to go ahead because of the boogeyman of China. We have seen the people at these companies

• 2:24:02Speaker 00

  say that we need to develop the tools to pace the frontier, which is corporate speak, for this is going too fast for us to get a handle on things.

• 2:24:12Speaker 00

  We need, like, these people believe it. They believe they're gambling with your lives. What has changed is that the rest of the world is starting to notice. And that's what gives us a moment of hope.

• 2:24:25Steven Bartlett?

  Trump this week was asked about the threat of AI and this was his response.

• 2:24:29Speaker 05

  Case scenario with AI is that the robots, the machinery learns to, obviously it thinks for itself. That's what it does. And they, that could turn against humanity. I just, do we have the guardrails?

• 2:24:41Speaker 03

  It's going to be fine. We'll always have something to stop them, right? Have a little gear.

• 2:24:45Speaker 05

  I really hope so.

• 2:24:46Speaker 03

  I really don't like that robot, we'll stop it.

• 2:24:49Speaker 05

  Some people say the worst case scenario.

• 2:24:51Speaker 03

  You laughing, but this is the state of the art in AI safety right now.

• 2:24:54Speaker 04

  Yeah.

• 2:24:54Speaker 03

  This is the device we have. That's the best we got.

• 2:24:57Steven Bartlett?

  For anyone that couldn't hear that, Trump went, we'll always be fine. We'll have something to control it. And then he did a little gun finger and he went, boom, I don't like that robot.

• 2:25:04Speaker 04

  I don't like Sammy.

• 2:25:07Steven Bartlett?

  If you don't laugh.

• 2:25:08Speaker 00

  I would say that the reason humanity always has something to stop a problem is because people notice a problem and build what it takes to have something to stop a problem, which I think you'd agree with. I am not here saying we're going to die. I'm here saying, if you look at the technology, if you look at what it's doing now, if you look at what the experts who are building it are saying about their own fears, you see that we need to rise to this occasion. You said you trust humanity to rise to the occasion? I sure hope we can. I think that rising to this occasion is going to mean that nobody races towards superintelligence because we have no idea how to get that right.

• 2:25:52Speaker 00

  And, you know, finally, the world is starting to notice that it's an extinction threat.

• 2:25:59Steven Bartlett?

  Thank you. Nate, Roman, Ed, Andy, super appreciate you. All of your books will be linked below in the description and on screen.

• 2:26:10Steven Bartlett?

  Let's see what happens. We'll convene again. Thank you so much.

• 2:26:12Speaker 01

  Thank you.