The Point of No Return: Connor Leahy on Stopping AI Before It's Too Late

What's up, everybody? Today, I am bringing you an absolutely critical episode with Connor Leahy, a leading voice in the world of artificial intelligence safety and the CEO of ControlAI. Connor is a computer scientist who has spent years at the forefront of AI research, grappling with the real and present risks facing humanity as this technology rapidly evolves. He has a bold and, frankly, chilling perspective: pursuing superintelligence could endanger us all. In this episode, Connor breaks down why AI isn't just another tool—it's something fundamentally different, and possibly the biggest adversary we've ever created. We dig into the game theory behind international AI development, why current regulations are falling short, and what would actually have to happen for us to avoid catastrophic risks. If you've ever wondered how close we are to a true point of no return, or what it would actually take to keep the future of humanity in our own hands, this is the episode you cannot afford to miss. I hope you learn as much from this convo as I did—Connor will challenge the way you think about progress, technology, and what it means to be wise in a world racing toward the unknown. If you get value from this, I’d be so grateful if you leave us a review. That’s how we spread the word and help more people unlock their true potential. I'm Tom Bilyeu, and welcome to Impact Theory. What's up, everybody? It's Tom Bilyeu here: Want my help starting a business? Join me here inside Zero To Founder https://tombilyeu.com/zero-to-founder?utm_campaign=Podcast%20Offer&utm_source=podca[%E2%80%A6]d%20end%20of%20show&utm_content=podcast%20ad%20end%20of%20show Sign up for my AI Masterclass: AI Masterclass https://tombilyeu.com/ai-masterclass?utm_campaign=Live%20Masterclass&utm_source=podcast&utm_medium=evergreen FOLLOW TOM: Instagram:https://www.instagram.com/tombilyeu/ Tik Tok:https://www.tiktok.com/@tombilyeu?lang=en Twitter:https://twitter.com/tombilyeu YouTube:https://www.youtube.com/@TomBilyeu Tailor Brands: Check out Tailor Brands to get started with your business today: https://bit.ly/TailorBrandsSept Quince: Free shipping and 365-day returns at https://quince.com/impactpod ElevenLabs: Book your demo at https://elevenlabs.io/impactpod Incogni: Take your personal data back with Incogni! Use code IMPACT at the link below and get 60% off an annual plan: https://incogni.com/impact Ketone IQ: Visit https://ketone.com/IMPACT for 30% OFF your subscription order. Horizon.ai: Go to https://horizon3.ai/IMPACTTHEORY and request your free NodeZero demo. No commitment required. Results in hours, not weeks. Butcherbox: Go to https://ButcherBox.com/IMPACT https://butcherbox.com/IMPACT to get $20 off your first box, plus your choice of free ribeye, new york strip, or filet mignon in every box for a year — with free shipping always Quo: ​​Try for free PLUS get 20% off your first 6 months at https://quo.com/impact See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

• 0:00Speaker 00

  Let's talk about the moment your side hustle becomes a real business. Now, I see way too many founders stay stuck running their business like it's a hobby. And then they wonder why nothing ever feels real. It doesn't feel real because it's not. I've helped thousands of entrepreneurs build real businesses through Impact Theory University. And the first step in the path to building a legit business is registering it. And Tailor Brands will help you do just that in a few clicks. An LLC, a limited liability company, is an official business structure. It's inexpensive to form and very simple to maintain. It's how you open a business bank account. It's how customers know they're dealing with a real company. And Tailor Brands doesn't stop at the filing.

• 0:45Speaker 00

  Their dashboard tracks what comes next based on what your business actually needs. So EIN number, registered agent, licenses and permits, bookkeeping, business banking. I tell every founder the same thing. Just make it official. Click the link in the show notes to check out Tailor Brands and get started on your business today. This is a paid advertisement.

• 1:07Speaker 01

  Tire Discounters presents Strange But True. You can now get four tires installed with a free alignment and oil change as low as 350 bucks. Oh, it's true.

• 1:21Speaker 01

  See store for details.

• 1:22Speaker 02

  I'm a computer scientist by background. I've written a lot of software in my life. AI is very different from normal software. Most software is written with code. Line by line, you tell the computer exactly what to do. This is not how AI works. The people at OpenAI do not know what is going on inside of their AIs. Dario Amade, the CEO of Anthropic, thinks he, we understand maybe 3%

• 1:44Speaker 00

  Oh my god.

• 1:45Speaker 02

  of what goes on in our AIs. Superintelligence is not a tool. It's not a weapon. It's an adversary.

• 1:54Speaker 00

  Connor Leahy, welcome to the show.

• 1:55Speaker 02

  Thanks so much for having me.

• 1:56Speaker 00

  Dude, this is one of the most important conversations happening right now. So I'm certainly glad that you, as a voice, are out there doing your thing. As I was building a mental model for the interview. To me, it looks like you have a series of what I call base assumptions that your push to get AI development stopped or curtailed. We'll get to exactly what that is in a second, is driven off those. So I want to start the interview by asking some essentially calibrating questions to get to those answers so that then I know what you're building on top of. So my first question is, you believe that if we pursue superintelligence, the risk that that destroys humanity is so high

• 2:41Speaker 00

  that we should stop that pursuit immediately?

• 2:45Speaker 02

  Yes.

• 2:45Speaker 00

  Okay. Do you believe that the correct path is a non-development agreement between the US and China?

• 2:53Speaker 02

  Yes, specifically when it comes to superintelligence. I do not think this applies to all forms of AI. One thing that lobbyists love to do is they love to try to equivocate all AI as the same technology. This is obviously nonsense. Here's a fun fact. Did you know that you could buy uranium ore on amazon.com?

• 3:07Speaker 00

  I did not know that. That's terrifying.

• 3:09Speaker 02

  You could ship it to your house, put it on your desk. It's perfectly harmless. Natural uranium ore is very harmless. It's perfectly safe to touch.

• 3:15Speaker 00

  Okay.

• 3:16Speaker 02

  But obviously, highly weapons enriched uranium is extremely illegal.

• 3:21Speaker 00

  Right.

• 3:21Speaker 02

  This is also how I think about AI.

• 3:23Speaker 00

  Okay. Do you already know where you would draw the line?

• 3:26Speaker 02

  I have some suggestions, but fundamentally, this, in a sense, we can take a step back or maybe that's one of the questions still to come, is where, why do I think these risks exist? What do I think their shape is? And a lot of the risks I see where they come from is fundamentally that we don't understand AI. AI is very different from normal software. I'm a computer scientist by background. I've written a lot of software in my life, you know, even without agentic help. And most software is written with code. Line by line, you tell the computer exactly what to do. This is not how AI works. AI is more like grown rather than written through a technique that's called neural networks. You have these massive piles of data and you kind of have a program self-assemble itself,

• 4:10Speaker 02

  grow, learn from this data. And what comes out the other side is in lines of code that you could read or understand what's going on. It's more like billions and billions and billions of numbers. If you multiply and add all those numbers in the right order, you get ChatGPT.

• 4:24Speaker 00

  How do you begin, though, to parse whether something falls into the yes bucket or the no bucket? It isn't just that it uses a neural net presumably?

• 4:33Speaker 02

  Yes.

• 4:34Speaker 00

  Okay, so if it's not neural net, is it size of the data center?

• 4:39Speaker 02

  Fundamentally, I think it's a behavioral classification in the senses. There's obviously a category of things that are, can outcompete humanity. The way, that's what I like to think about it. I like to think about is like, if you build a system of any kind, it can be a single agent, it can be a swarm, it can be, you know, it can have many shapes. If you build a system that can outcompete humanity at all relevant tasks. Every time you start a business, you go out of business because an AI business outcompetes you. Every time you trade on the stock market, you lose all your money to an AI hedge fund. Every time you run a political campaign, you lose to the other guy who's using an, you know, AI advisor. Every time you run a military campaign, you lose to the person who's using more autonomous AI systems. If you get into a world like this, power of all kinds, economic, military, political, et cetera,

• 5:23Speaker 02

  is concentrated in the hands of non-human entities in AIs. And if we don't control these systems, if we don't understand these systems and if they don't have our best interests at heart, it's very hard to imagine that going well.

• 5:35Speaker 00

  Okay, it seems like we're already there, but the AI is a tool. So the way that I think about AI right now is if somebody has AI and they compete against somebody in the same thing without AI, the person with AI will win effectively 100% of the time. Do you disagree with that?

• 5:50Speaker 02

  I agree with that. What I disagree with is I think if a human against an AI without a human, the human still wins.

• 5:55Speaker 00

  Totally agree.

• 5:56Speaker 02

  Yes, what I'm saying is what happens when a human versus an AI alone, an AI, an autonomous AI, and the autonomous AI wins every time.

• 6:02Speaker 00

  So it's the autonomous part that's important for your definition?

• 6:06Speaker 02

  Yes, I think autonomy is very important here. When I say autonomy, I mean, you know, both in the obvious sense of the term, but also like, obviously, if you just have a human rubber stamp yes on everything, that doesn't count as oversight.

• 6:16Speaker 00

  Fair.

• 6:17Speaker 02

  So that would still count as autonomous to me, even if there's technically a human involved, but the human doesn't even understand what the system is doing and they're just rubber stamping all the decisions. Like, what I expect to happen, for example, is that there probably will continue to be human CEOs for quite a while, but they just rubber stamp what the AI CEOs tells them to do. They're just there for legal representation.

• 6:36Speaker 00

  And so for you, we would have then long passed where we, we're now beyond the point of no return and all is for naught.

• 6:43Speaker 02

  Yes, I like thinking about the point of no return much more than like, think about like the point of like extinction or doom in the sense.

• 6:49Speaker 00

  Why?

• 6:50Speaker 02

  Because that's the point that actually matters.

• 6:52Speaker 00

  Okay, so it's like stakes versus where we are on the timeline.

• 6:55Speaker 02

  Yeah, in a sense, the, what matters to me is the point of no return. Once we've hit the point of no return, where there's so much AI, it's so powerful, it's so intelligent, it's self-improving, you know, there's not just one AI, there are millions or billions of them. Even if we're not dead, even if we continue to maybe live for a couple of years or even decades, we no longer have any control over the future and there's no way for us to get it back.

• 7:19Speaker 00

  Okay, you've already slipped in a base assumption there. So I, before we get to that base assumption, because it is certainly on the list of where this sort of breaks bad, very clear definition. So autonomous AI that can outperform humans at everything, that's the one that we're worried about.

• 7:33Speaker 02

  Yes.

• 7:34Speaker 00

  We need to understand the mile marker, which is what's the point of no return? You've talked about this before, so I think we can hit this one quick, which is, we're close, you don't quite know how close, we're so close, it's dangerous, we should stop now.

• 7:46Speaker 02

  Yes.

• 7:46Speaker 00

  Okay, if I gave you a big red button and you could unilaterally stop everybody from developing any further, we can use what we already have, but we cannot progress any further. Would you hit it today? Would you hit a week, a month, a year?

• 7:57Speaker 02

  I like to say, if we as a species or as a civilization were three steps wiser than we were today, we wouldn't have done ChatGPT in the first place.

• 8:05Speaker 00

  What are the steps? What do you mean steps?

• 8:07Speaker 02

  It's just a, it's just a.

• 8:09Speaker 00

  Three clicks, three, just a little bit smarter?

• 8:11Speaker 02

  Three generations of wisdom. And if we, if we were two steps wiser than we currently are, we would have done ChatGPT, saw that it did all these crazy things that none of our scientists predicted and it got, you know, 100 million users, right? I mean, like, whoa, whoa, whoa, whoa, whoa, what the hell is going on here? And would have rolled back, tried to figure it out. And if we were one step wiser, we would stop today.

• 8:30Speaker 00

  The original question about US-China is trying to get to is, I, my belief is that you see this as a state level thing and that law enforcement is going to be the punchline. Meaning that just like with nuclear, if we found that somebody were trying to build a nuclear weapon, we would immediately stop them, that's against the law, we've got things around that already, we need to do that for AI.

• 8:53Speaker 02

  Yes.

• 8:54Speaker 00

  Okay, this, this is going to be a big part of where I look at this and I go, I think that there's a problem there, meaning you're never, because it is, we agree that this is a state level issue between the US and China specifically. Now's your chance to say there's somebody else.

• 9:10Speaker 02

  I want to say the EU, but I can't.

• 9:12Speaker 00

  Okay.

• 9:13Speaker 02

  With my heart.

• 9:13Speaker 00

  Yeah, yeah.

• 9:14Speaker 02

  I'm joking. That was a joke. I've lived in Europe for many years. I love the EU.

• 9:18Speaker 00

  They're amazing. My wife is European. I totally get it. But we really have this US versus China thing. Okay, so you put forward a trust but verify situation. Much like nuclear or something that has a very different approach to the trust but verify?

• 9:34Speaker 02

  I think there's a lot of inspiration to be had from nuclear example. I think the history of nuclear regulation is the closest to there is a great crazy new technology that will change everything. And we rose to the occasion. Now, I'm not saying we fixed everything, but you know, when I think about regulating AI and like the history of AI, I like to think back to Leo Zillard, the person who came up with the nuclear bomb. When he first did the math, he realized, oh shit, it's possible. Nuclear bombs are possible. He did the math and he was like, oh my God, it's definitely possible. So what he did is he moved to San Francisco, started a nuclear bomb startup, which split into three other startups to sell as many nuclear bombs to the public as possible. No, of course, that's not what happened. He went to the government. He went to the military. He was like, guys,

• 10:16Speaker 02

  this will change everything. We have to do something. We have to figure this out. And this is what didn't happen for AI. What didn't happen for AI is that we didn't realize that this will change our civilization. This is not a private matter. This is a national security matter. This is a matter of the organization of economic structures. This is a political question. This is a government. This is an intergovernmental question. And we just didn't do that. And by we, I mean the scientific community, like the scientific and the technical community, like people who started companies like, you know, DeepMind or Anthropic or OpenAI, they weren't Leo Zillard. They went to go make a bunch of money off of it. They didn't go to the government and were like, we need to build new institutions to responsibly

• 10:57Speaker 02

  steward this powerful technology. And so we didn't do it. You know, at the time when nuclear was first considered, the idea that you would have like stuff like, you know, the International Atomic Energy Agency and so on was unthinkable. Such a thing had never existed in the history of any legal code before. They had to invent whole new forms of law to deal with this new technology. And that makes sense. If you have an incredibly powerful new technology, you have to steward it responsibly. And that will require coming up with new doctrines, with new ways of like, what level of risk do we tolerate in various ways? Where do the risks come from? Who decides the level of risk that we take with these relevant technologies? And at the moment, we're just not making these choices at all. It's not

• 11:40Speaker 02

  even that we're making the wrong choice. It's that we're not making these choices at all. And I would love if there was like disagreement, for example, between POTUS and Xi Jinping about what the correct level of extinction risk is. You know, if they're arguing, oh, it should be this percent or that percent. I think this would be a really good world to live in, because then we could have this argument. But we're very far even from that point, that we're not even making these decisions yet.

• 12:03Speaker 00

  We're hitting pause for a moment, but there's plenty more ahead, so don't go anywhere. Let's talk about a problem the military couldn't solve with willpower. Picture a soldier making life or death decisions after 36 hours without sleep. This is high stress. There's obviously no margin for error. The military needed their people thinking clearly, even under conditions that would break most of us. So they funded a $6 million Department of Defense research contract to find the answer. What they found was ketones, the brain's most efficient fuel source. That research became Ketone IQ. Ketones cross the blood-brain barrier and fuel your neurons directly. They enhance mitochondrial function for longer mental stamina.

• 12:46Speaker 00

  I take it for interviews, before writing, before I do coaching, and I can feel the difference. Visit ketone.com/impact for 30% off your subscription order, or find Ketone IQ at CVS stores nationwide and get $5 off your first six-pack for a limited time. There's a reason Quince sells cashmere for $60, and it isn't the cashmere. It's real 100% Mongolian cashmere. It's the same stuff luxury labels charge a fortune for. Quince just works directly with ethical factories and cuts out the middleman. So there's no need for a markup and no logo tax. The $60 goes to the actual sweater, not the fancy

• 13:29Speaker 00

  name on the tag. That's why everything they make runs 50 to 80% less. And it's not just the cashmere. I ordered a few things myself. Their fleece joggers, a couple of their tees, which I wear obsessively. First thing I noticed was the quality. Soft, well-made, built to hold up. They make down jackets and wool outerwear now, too. So upgrade your stuff to things you'll actually wear. Find your next fall favorites at Quince. Download the Quince app for app-exclusive offers, or go to quince.com. Get free shipping on your order and 365-day returns, now available in Canada and the UK, too. And when Quince asks where you heard about them, let them know it was Impact Theory. It's the best way to support

• 14:14Speaker 00

  the show. I want to talk to you about how often you test your security. Most companies do it once a year. Then a whole year goes by, but you've had new hires, new tools, new permissions. People are submitting new code every week. That test may have made you secure for one day, but if nobody checks the other 364, you're in trouble. Horizon 3 closes that gap. Node Zero is an autonomous AI hacker that continuously attacks your environment at machine speed the way a real attacker would. Your entire attack surface, external to cloud to identity in one single test. 225,000 production-safe tests run with zero downtime. See exactly

• 14:58Speaker 00

  what an attacker would find in your environment before they do. Go to horizon3.ai/impacttheory and request your free Node Zero demo. Visit horizon3.ai/impacttheory. Again, that's horizon3.ai/impacttheory. No commitment required. Results in hours, not weeks.

• 15:23Speaker 00

  Thanks for sticking around. Let's get right back into the action. Okay, if Elon can be believed, he said for years that he tried to lobby the government to let them know this is going to be a real problem. This is, he referred to it as a demon summoning circle, and we're all just hoping that the demon that we summon is going to do our bidding. And he said he just could not get anybody to take him seriously. And so he became fatalistic. That's when he tries to start OpenAI, because he's like, well, if it's going to exist, I really don't want Larry Page at Google to be the only one making it because he's whatever, a specist, or that was the accusation to Elon. And so Elon

• 16:02Speaker 00

  has, again, if he can be believed, attempted to do maybe an early prototype of the very systematic way that you're going about it now. So do you think it was overwhelming confusion that led people to do that? And because I have a different take on the nuclear analogy, which is where I'm getting to. Do you think Elon wasn't good at it? People were overwhelmed, but why didn't that take hold?

• 16:32Speaker 02

  I think an average political institution building campaign takes 20 years. I don't think Elon spent 20 years on this. I don't know. Maybe he did, and I'm not aware of it.

• 16:40Speaker 00

  But why, when the guy that understood the nuclear thing, why did he get immediate attention when he went to the government?

• 16:45Speaker 02

  Oh, no. It took decades.

• 16:46Speaker 00

  Really?

• 16:47Speaker 02

  Yes.

• 16:47Speaker 00

  So he was banging and banging and banging on it?

• 16:49Speaker 02

  So he was banging for years. Then there was the Einstein letter, which led to the Manhattan Project. But until the IAEA was created and all these other things, that took decades of work by international diplomats, you know, by lawyers and by, you know, statesmen, you know, between, this was when the Soviet Union was still around, right? Imagine how hard it was to get the Soviet Union to agree to something like this. This was a decades-long project that also kept evolving, you know, throughout the 60s, the 70s, the 80s. And so this was not a thing that was like, oh, Zillard, you know, had one Zoom call and then the thing was solved. Quite the opposite. It required thousands, tens of thousands of scientists, diplomats, politicians, et cetera, working for decades. So, you know,

• 17:30Speaker 02

  I applaud Elon for having tried. But fundamentally, a thing that I think is very frustrating to tech people is that non-tech things run at a different pace, is that if you want to convince, say, 1,000 diplomats to take an issue seriously, there's kind of like a hard limit on how fast you can do that. I think it's faster than people think it is, but it's still longer than like a couple of years. It takes time and it takes effort and it takes a lot of money. I'm not aware of Elon spending a billion dollars on political advertising or on AI X risk. He could have done this. He obviously has the money. He could just spend a billion dollars on political advertising to make this issue known. He could fund 50 think tanks to work on existential risk of AI. He has the money to do it. So in this

• 18:12Speaker 02

  regard, I don't know, you know, Elon, you know, I think he's an extremely prescient and extremely competent person. I think if he really wanted to, he could. I think he absolutely could. He could fund us, you know? We're getting a lot done with the money we're doing. And if he gave us 10x more money, we would get 10x more done. So I don't know why he made the choices he made. We can psychoanalyze all day. But fundamentally, one of the things that we learned when we started doing this is that every single person in the world told us this is impossible. No one will listen, and we've tried already. And we're like, we'll see. And then we just talked to politicians, and overwhelmingly, the response we got was, oh, wow, I had no idea. No one told me. Thank you for telling me. This seems really bad.

• 18:55Speaker 02

  I don't think people have tried. Because politics is slow, it's frustrating. You have to repeat something. You know, in advertising, you usually say, to get a message across, you have to repeat it seven to 10 times per person on over this, you know, a schedule like six to 12 months. One conversation will never be enough. You'll have to talk to that person seven to 10 times until they can start to really understand these. And this is frustrating. This is annoying. It takes a lot of time. So no, I think there's a lot that can be done.

• 19:22Speaker 00

  All right. So as a marketer, one of the reasons that I think that this stalled out, one of the reasons that I think nuclear ended up with, if we tell ourselves the wrong story about what happened there, it will seem like, oh, this was just another case of we lobbied, lobbied, lobbied, lobbied. He was ignored in the beginning, but then we finally got through to people. Is that when you're marketing, when you're doing sales, you're trying to elicit an emotional response. If you can't get the emotional response, if you can't create a sense of urgency that people feel, not understand, but feel, they're never going to move. And so when I look at the nuclear question, to me, that was a failed version. And we just sort of got off lightly that nobody after what we did to Japan, that everybody saw the level of destruction and just said, whoa, we gotta chill on this and not actually move forward.

• 20:06Speaker 00

  But to me, that was by seeing the devastating power of it, by the scientists themselves suddenly asking themselves, what have I done? And sort of waking up with that on their conscience. Then people were like, okay, yo, we gotta chill on this. But the reality was, if I'm not mistaken, and please tell me if you know something I don't. But they, when they were doing the first test, they knew there was a non-zero chance that they would light the entire atmosphere on fire and kill everyone.

• 20:33Speaker 02

  Yep.

• 20:33Speaker 00

  And they still did it.

• 20:34Speaker 02

  Yes, but they did calculate the risks and it was like a billionth of a percent.

• 20:39Speaker 00

  Okay, so if you had been in that room and you run it and you see it's a billionth of a percent, and I know you don't like to be unilateral, but what way would you cast your vote? Would you say, don't do it? Or would you be like, ah, that's a tiny fraction, go for it?

• 20:52Speaker 02

  Generally, my, my vibe is, my, my rule for my personal life is that something is less risky than driving a car, it's fine.

• 20:59Speaker 00

  Okay, so that one feels less risky, even though the consequences are so catastrophic that as long as it's just an odds question.

• 21:09Speaker 02

  There's a thing where, um, obviously I'm not a physicist. So I'm not sure how much I would have believed these numbers if I was a physicist. My understanding-

• 21:16Speaker 00

  You're likely to believe high or low, or just-

• 21:18Speaker 02

  My understanding from physicists I've talked to is that the calculations they do were like very solid. Like, it was like really a weird edge case to imagine that something could have gotten wrong here, because they actually really did understand the physics they were working with. This was about the fusing of nitrogen in the atmosphere. And this was a type of science that they were very confident in. Now, if you ask me the same question about a science I don't understand, for example, if you put me there and I'm not a physicist, I probably would have voted against it. Because, I don't know, maybe the model is wrong. You know, even if the model predicts a certain thing, maybe the model is wrong. So it's also about uncertainty. I, I, it's not just about what is the number that comes out, but also how certain error that this number is the thing you care about. So I do think that, you know, I may or may have said no, I may have said yes,

• 22:03Speaker 02

  depending on how confident, again, I'm not a physicist. Unfortunately, I just don't know how good the model actually was. Like if you, like, if someone tells me, oh, there's a, you know, quintillionth chance that if you, you know, turn on a light bulb, it will create a black hole that will consume the world. I'm like, whatever dude, you know? Because quantum physics, by the way, does predict this, right? Like every time you do anything, something could quantum tunnel into a black hole or whatever, right? But it's like, who cares? So, but what I find really interesting about this example is, is that they did run the math. They did have a very concrete model. They had a very concrete reason that they wrote down for like, why they think this is an acceptable thing to do. Meanwhile, we have, you know, our current, you know, crop of tech people saying on podcasts

• 22:44Speaker 02

  that they think there's maybe, I don't know, a 20% chance that it'll kill everybody.

• 22:49Speaker 00

  One of the base assumptions I think you have is what you were just articulating, which is about, it's the unknown element of AI. You said, if I wasn't a physicist and I didn't understand it, then I might vote no, just because the unknown is a big question for me. I feel that way about Darius or anybody that says, oh, we've got a rough 20% chance. I don't think they're running math. I think they reach into the future emotionally and say, I feel like something bad could happen. And because it's vibes, we're not able to do the math equation.

• 23:21Speaker 02

  Yep.

• 23:22Speaker 00

  And so that is sort of possible breaking point, number one, of trying to use the nuclear as a example of will it work or won't it work? What I'll point to there is, okay, if we're not able to get any kind of percentage, nobody's able to run the math and we can move forward with a degree of certainty, we either do what you're saying and just shut it down now. But given that, um, because it's the US versus China, I don't think that will ever work. And so I want to see if you can dismantle this for me. The way that I look at this is that, and by the way, just so everybody knows what my bias is, I would be perfectly happy with you unilaterally hitting the button today, but I wouldn't have been happy if you'd hit it yesterday. That's a little cheeky, but I'm just trying to say like, I love what I have now.

• 24:07Speaker 00

  It's so useful, but because I can't picture my future, it scares me. And I would be much more comfortable just saying, hey, it's already helping so much. You know, we got this far, woo, we got lucky, just hit it. So this is not me saying, I just, more, more, more. I would be very fine with that button. In fact, if I had the button, I would hit it because, uh, while I agree, no one should act unilaterally, I might be a little more foolish. The reason that it begins to break down for me as a nuclear question is we couldn't stop ourselves, even though we knew that we were unleashing something that could bring Armageddon if it proliferated, and of course it did. So we've already created one technology, could not stop ourselves from doing it before we had the weapons pointed at each other.

• 24:51Speaker 00

  So when I look at this, I say, okay, I think you probably could. You can, I think the US is on a path to rejecting AI. So that feels doable. Okay, you can reject, you can get the US to reject AI publicly. If you can get the US to reject it publicly, you might be able to get China to reject it publicly. What I don't believe is that either of them can afford to reject it privately. And that just as you have said, AI just gets better at lying when you catch it doing things. It has an incentive structure that it follows. And if lying and going underground makes you more likely to get it done, then you lie and go underground. Game theory tells us, they'll both say, yeah, yeah, yeah, we're, we're not going to proliferate because they want the other side to stop.

• 25:36Speaker 00

  And then they're both going to go to their own corners because they can't trust that they actually will stop. Knowing that it doesn't matter which of them pushes forward, how do we actually, not just the, the agreement, but how do we actually get them to stop?

• 25:50Speaker 02

  So there's a bunch of points here, many points. So let's, let's try to disentangle them a little bit. There's the game theory question, there, uh, like the, the geopolitical question. There is the, um, there is the question of like epistemics and linguistics of probability. Like the question of like, what does it mean to when someone says there's 20% probability? What does it mean when a physicist says there's a billionth of a percent probability it ignites the atmosphere? What do those sentences mean and how do you compare them? And there's a, the more general question of like, how do you react to situations like this? And how does it, how is it different from, because I think it's very different from nuclear. Like I think you didn't even mention, one of the reasons I think is the most different from nuclear is that nuclear weapons can't shoot themselves. You can have a nuclear weapon sit in a hangar somewhere, it's perfectly safe.

• 26:35Speaker 02

  You can hit it with a hammer, you know, nothing's going to go wrong. You can't contain a super intelligence. So a super intelligence sitting in a hangar somewhere is not a safe thing. So that's for example, something where I think it's very different. Where developing a nuclear weapon is actually relatively safe. Not perfectly, as you say, you know, fusion of nitrogen and whatnot, but it is like pretty safe. It's more the deployment that is very dangerous. In AI, it's very different. Super intelligence is the development that is dangerous, not just the deployment. So that's one disanalogy, for example. But just to circle back a little bit to talk about the game theory, there's a very, very important thing to understand here, which is that super intelligence is not a tool. It's not a weapon. It's an adversary. If the US builds super intelligence, it will not achieve American objectives.

• 27:19Speaker 02

  The United States will cease to exist. So will China. If China builds super intelligence, China will cease to exist. It will not achieve its objectives and so will all other countries. This is the core disanalogy. The core disanalogy is, is that it's actually not mutually assured destruction. It's actually not a prisoner's dilemma. It's actually independently assured destruction. If you raise super intelligence, you don't win. You just lose. Game theoretically, the only stable equilibrium actually is cooperate or operate. Is unlike a prisoner's dilemma where defect, defect is the stable equilibrium. Here, the only stable equilibrium where both China and the US get to live is if neither build it.

• 28:01Speaker 02

  That's the only one where everyone survives, or anyone survives for that matter. So this is, I think, one of the core game theoretic disanalogies here. Is that this is all assuming the assumptions that, you know, you can't control something that's smarter than you. We currently are not even on track to build AI second controls and they can outcompete us. If we accept, accept those intuitions, if we accept those definitions, then this follows naturally. If that makes sense.

• 28:24Speaker 00

  Aggressively, very clear. Now we have a problem and maybe I'm being unkind to America, but America's better understood as being run by politicians, technocrats, finance. But China is largely run by people with an engineering bent, even if they aren't engineers themselves. They understand building infrastructure. They've come from essentially nothing to a global superpower by building, by creating, by translating schematics into your ideas into schematics into actual tangible things. And they can move electrons and all of that. So even if I'm unkind to Americans, say, okay, America's not going to figure this one out. But China, you would expect, there's only one person you have to convince.

• 29:09Speaker 00

  And if you convince him, then everything goes downstream from that. Why have we not been able to convince Xi Jinping?

• 29:17Speaker 02

  I think no one has tried. I don't think anyone has gone to Mr. Xi Jinping and given him a really good presentation on superintelligence. I would be very shocked if there was a single person that has done this.

• 29:29Speaker 00

  And what are the odds, given that it will likely need to come from within China? What are the odds that you think that happens?

• 29:38Speaker 02

  I think it depends a lot on what we do in the West. So I'm American, obviously. I have no influence on what happens in China. I do have a lot of influence what happens in my democratic country. And I do think that if we in America make it a priority, not just to stop, you know, the superintelligence, you know, from say domestic companies at home, but anywhere on earth, which is the only reasonable policy. The thing that we at Control AI pose is that the only reasonable national, international security position is, is that we cannot tolerate the creation of superintelligence by anyone, anywhere. When, you know, people, politicians, military officials ask me, okay, but what if China builds superintelligence? My answer is, we cannot allow that. We cannot allow

• 30:20Speaker 02

  that. That's not acceptable. If China builds superintelligence, it will kill American citizens. This is all of us. It will destroy the American, everything, the whole country. This is not something we can tolerate. And similarly, the Chinese cannot tolerate the Americans or their own companies building superintelligence. Because again, it is not a weapon, it is not a tool, it is an adversary. This is the important thing. When I talk to different people about these risks and superintelligence, I kind of think that it's like, kind of like, it's kind of like a political compass. There's like, kind of like two axes. And like, one axis is believes technology is always good, and believes technology can be good, or bad, can be both. And the other axis is thinks superintelligence is possible, doesn't think superintelligence is possible. And most of the disagreements

• 31:05Speaker 02

  I have with people is along the axis is just, they don't, haven't heard of superintelligence. They've just not thought about it. No one told them about it. They haven't considered the implications of it. Like, when people talk about racing with China, most of the people who say we should race with China just don't know what superintelligence is. If we didn't have superintelligence, like if you promised me, you know, God came from heaven and he said, there's 100% chance that superintelligence will never be created, then I think we should probably race. Then we probably should compete with China on AI. If I can guarantee there is no chance of superintelligence, and it's only going to be useful economic and military applications that are, you know, controlled by humans throughout, yeah, we should probably do it. So most of the time when I disagree with these kinds of people that are pushing AI so aggressively, it's

• 31:48Speaker 02

  just because they literally don't believe it's, or don't think it's possible. One of the most funniest things I've found, every time in San Francisco, is when you ever talk to like accelerationists or like, pro tech people, they don't believe in like, superintelligence. They just like, don't think in the future. They think AI will always be like, an app on your phone. But this is not what we're talking about. We're talking about autonomous agents, intelligent agents, acting in the real world against human interests, in ways that are more competent than humans can, you know, competing with, it won't be, again, it's not just one superintelligence, it's a whole ecosystem, evolving, mutating, competing with each other, fighting, you know, there'll be wars between these AI systems, where we are just collateral damage. This is a very, very different thing from, you have a cool app on your phone. And this is again, why I say, like, there's this

• 32:31Speaker 02

  difference between, you know, uranium ore and highly enriched uranium. I think this is really at the core thing. I think that if Xi Jinping understood, if someone helped him understand, I think he's a smart guy. I mean, I haven't met him, but he seems like a smart guy. I think if he understood what superintelligence is, if someone explained the game theory to him, I believe he would understand it. And I believe we could take actions based on this. I do think this.

• 32:53Speaker 00

  Okay, that's very optimistic, and lord knows, I hope you're right. You said something, though, that is, given the context we're in right now with Iran, the, what if China builds a superintelligence, we just can't let them? That's how we saw, at least how Trump and the Trump administration saw Iran and nuclear weapons. And so at some point, at least in their judgment, it had to become kinetic, because they refused to acknowledge that they weren't going to pursue nuclear ambitions. So what do you do if, let's say China does say, we're not going to do it, but you have intel that says they are doing it? Do you get the world together and say, we got to invade?

• 33:34Speaker 02

  Edge lords love to say, you know, international law is fake. And this is mostly true. The only law that exists is backed by credible deterrence. It's the same thing with law in our country. The only, the only reason that law exists is because if you break it, the police will arrest you. And if you resist that arrest, they will get violent. You know, they will force you into prison, whether you like it or not. Every system of law is backed by force, to some degree. The, the beautiful thing about law is that if you design your laws right, and you build your system right, you never need to use it. This is the beauty of deterrence. Deterrence is one of the most beautiful things in the world, in a sense. I sometimes like to think, if you have a lot of weapons, you'll never need to use them. If you have a little bit of weapons, you're definitely going to need to

• 34:18Speaker 02

  use them. So in a sense, this is how peace has worked in our world for a long time. It was always clear that, you know, if someone pulls some shit, if someone like, goes against international order, if someone does a huge human rights violation, if someone, you know, nukes, you know, I don't know, like Latvia or something, that the Americans will enter being and make sure that doesn't happen again. And that like, every, you know, people's heads were rolled. The same thing is mutually sure destruction. The reason the Soviets didn't nuke anybody, I don't think it's because the Soviets were nice people. They were not. I think it's because they knew that if they did that, the Americans were going to nuke them. So the way to create peace is through deterrence, which is the correct way to do it. So any kind of agreement has to be backed by credible deterrence, on everyone's side, for everyone's benefit. If you have credible deterrence, you know,

• 35:02Speaker 02

  then you would think everyone would be rational enough to not invoke that deterrence. Now, when we're talking about, like, say terrorists, who might not, who might be crazy, they might not react to incentives, well, sometimes you have to get ugly. That's an unfortunate truth about geopolitics. I hope it doesn't come to that. I think, you know, you can do a lot, including with non kinetic means of deterrence, such as economic sanctions, diplomatic pressure, et cetera. But fundamentally, if a superintelligence is built anywhere on earth, this threatens the lives of you, me, our families, our children, and everyone else. And we should reserve the right to self-defense.

• 35:39Speaker 00

  We're hitting pause for a moment, but there's plenty more ahead, so don't go anywhere. Let's talk about what it means to actually be visible in today's day and age. When you build something public, whether it's a business, or an audience, anything on social media, you become a target. This is something that legitimately keeps me up at night, because our data isn't just on the obvious broker sites, it's on old news portals, public forums, obscure directories, places you would never think to look. That's where Incogni's exposure scanner comes in. It hunts down your personal information across the wider web, not just the 420-plus broker sites, Incogni already removes your data from automatically. They can't

• 36:22Speaker 00

  harm you if they can't find you. Go to incogni.com/impact and use code IMPACT for 60% off an annual plan. Try it risk-free for 30 days. That's incogni, I-N-C-O-G-N-I, dot com slash IMPACT and use code IMPACT. Let's talk about your sales pipeline reviews. If your CRM is half-empty fields and notes nobody finished, every review turns into guesswork. I've seen it so many times in my own sales team, and that's where today's sponsor, Pipe Drive, comes in. An intelligent, AI-powered sales CRM that is loved by growing sales teams.

• 37:03Speaker 00

  Pipe Drive just launched Meeting Intelligence, built right into the CRM. Before the call, it pulls the deal history and past conversations into one brief. During the call, it records and takes notes. After, it drafts the CRM updates for your rep to review and approve. The record stays complete, and you see what's actually happening. Switch to a CRM built by salespeople for salespeople and join the over 100,000 companies already using Pipe Drive. My link gets you an exclusive 30 days free, instead of the usual 14-day trial. So make sure you use it. No credit card or payment is needed. Just head to

• 37:41Speaker 00

  pipedrive.com/impact to get started today. That's pipedrive.com/impact, and you can be up and running in just minutes. Let's talk about how much you're paying to stay safe online. Count it up. A VPN? A password manager? Something to block ads and trackers? The service to scrub your data off of broker lists? A tool to flag scam emails? That's five. Five subscriptions, apps, logins, and most people still have gaps. That's the problem Surfshark solves. It's not a VPN. It's an all-in-one security suite. One tool that protects your connection, your personal info, and your devices, all at the same time. Go to

• 38:25Speaker 00

  surfshark.com/tomb, or use code TOMB for four extra months of Surfshark, backed by a 30-day money-back guarantee. That's S-U-R-F-S-H-A-R-K dot com slash TOMB. Surfshark dot com slash TOMB. Use code TOMB for four extra months of Surfshark.

• 38:53Speaker 00

  Thanks for sticking around. Let's get right back into the action.

• 38:57Speaker 00

  Right now, we have the US and China specifically going head to head. So you have two great powers. Neither want to acknowledge the other as being in a leadership role. Do you think that it works with the US and China if Xi is adversarial? Because you've already been very clear. You think that he can be convinced. So that scenario takes care of itself. But if Xi is adversarial to the West and wants to push it, wants to, maybe he even says, I don't want to go all the way to superintelligence, but you know, he's getting way too close to crossing the point of no return. And he keeps pushing it. In that, do you think the US has the ability to deter in a non kinetic fashion?

• 39:38Speaker 02

  Deterrence is based on rationality. If he is completely irrational, and so is the entire military chain of China and its entire industrial apparatus and its entire political apparatus. And everyone is completely insane. Well, then deterrence doesn't work because they're insane.

• 39:52Speaker 00

  But they would only have to be a little bit off on your compass. If they believe that technology is good, then to them, they're not insane. And they're like, whoa, these guys have abdicated. We've got an opportunity. Tech is only good. So we're gonna keep pushing in that direction to assert our rightful place in the world. We're gonna make things better. We're gonna be less combative than the US ever was. So to me, there's a very easy scenario to paint where China sees itself as providing almost liberation. We're gonna be doing a great thing for everybody. This is wonderful. We know the right answer. And so they certainly won't perceive themselves as crazy.

• 40:33Speaker 02

  I think this is very possible. I think it's very possible that Stalin or Khrushchev could have just, you know, nuked Latvia. I think there are timelines in which that happened, you know?

• 40:41Speaker 00

  Well, so here's where we get back to, you were right, I think, to point out that the analogy actually doesn't really hold between nuclear and today. Because if the superintelligence gets made, then it becomes its own thing. It becomes adversarial. We'll talk more about that later. But for now, if we adopt that frame of reference, you've got China knowing that, okay, if I can get AGI and America doesn't get it, now I'm in a much better position. I'm acting from a place of my own interest. But the world, if they are on the other part of the spectrum and go, we can't allow this, you have a situation where, I'll put forth my belief, which is the US can't hardly deter Iran.

• 41:24Speaker 00

  So even if in the end, we finally choke them out from an economic standpoint, we'd never be able to do it to China. So now we're in a position where, I mean, maybe if the whole world got together? But that feels like a tall order. So it feels to me like deterrence gets removed off the table. And we're left only with the US and Xi both have to agree, much like in war games, that the conclusion is this is a funny game and the only way to win is to not play.

• 41:52Speaker 02

  I think so. To a large degree, this is the case. Yes, you can in fact construct many scenarios in which we lose. My P doom is high. I don't think we're going to make it. I think in almost every timeline that starts from us sitting here today, we don't make it. There are very few timelines in which we make it. But there are some, and I'm describing those that are still available to us, where we do make it. These are the kind of timelines that do make it. And I'm trying to explain why I think they're not like insane, you know, crazy edge timelines. They are rare, but they're real.

• 42:23Speaker 00

  Sure.

• 42:23Speaker 02

  Like, yes, it is very possible that just, for me, the default timeline is no one thinks about this at all. Everyone just, you know, keeps twiddling their thumbs for the next, you know, two to five years. And then superintelligence gets created and it's womp womp. You know, like that, that's my default prediction.

• 42:40Speaker 00

  That's good. Womp womp. I like that. Okay, that makes sense. One of the things that keeps coming up is what I think is, the thing I listed first is your like absolute foundational belief, which is that AI, sorry, superintelligence is by nature adversarial. I want to understand why by definition it has to be adversarial.

• 43:02Speaker 02

  I don't think it's by definition. I think the way we are currently building them, superintelligence that are not adversarial are just like not possible. Building something that is non adversarial and is that powerful is actually really hard. Like extremely hard. This is basically equivalent to saying, we're gonna build a one world government that controls the entire economy, that controls the entire military, that can build super powerful technology that we can't even understand, that will control your entire private life and all those of all other people. And it will be good all the time and will not make any mistakes. And it'll be no bugs and we'll do it all with software on the first try.

• 43:38Speaker 02

  That's what it would mean to build a non adversarial or like a good superintelligence. That's what it would mean. And look, if you said, we're gonna spend three generations of all of our greatest scientists, philosophers, mathematicians, working day and night on this problem. Maybe, right? Like, maybe. But that's not what's happening here. Like what's currently happening here is a clown show. Like, this is what's going on here. Like, it's not like we have, or throwing all of humanity's glorious heroic effort into just trying to figure out how can we build a good superintelligence? And, you know, all of our children grow up to want to work on this problem. And, you know, all the wise elders are overseeing the project and so on. No. It's a bunch of kids in San Francisco with zero adult supervision, you know,

• 44:23Speaker 02

  and run by sociopathic companies, building whatever the fuck they want and deploying it wherever the fuck they want. We're not even trying. This is why I think we fail. I don't think it's an antillogical thing. I think it's just if you take the world as it exists today, there is no outcome that leads to a good, that leads to something like this.

• 44:39Speaker 00

  Okay. As a mile marker, I think you've already addressed this. But basically, it would be, you've got strict regulation, you've got cooperation between the US and China. We all agree it's not moving forward. And if we saw these kids with or without adult supervision, it would be, please show up at your door and done. And everybody gets the hint. Cool. We don't mess with that.

• 44:59Speaker 02

  Yep.

• 45:00Speaker 00

  Okay. So now I want to go a little bit deeper on it. That it would, when you were describing that, you put a lot of things on the table that would make getting this the first time very, very difficult. And I concede all of that. But I do want to talk about, I still don't understand, what is the chain of logic that you follow? That you're sort of building step by step to get to the most likely outcome of superintelligence is to be adversarial?

• 45:29Speaker 02

  There's a couple ways you can approach this. The thing is, is that there's not a single argument. There are many convergent points of evidence that point to the same thing. So I can present to you some of them. The simplest thing is, is that this is what optimizers do. It's always what optimizers do. All things equal, if you make something that optimizes for anything, it will optimize for things. And if that thing doesn't include human values and emotions and feelings that are, which we have, could you write down in math, please, what human values are? Could you write that down for me? No? No one can. We have no idea. This is like, philosophers have been arguing about this literally since the dawn of time. And we have made, you know, not that much progress, some progress, but not that much.

• 46:09Speaker 00

  So you look at the three laws of robotics and laugh? Because it is just never gonna be as easy as giving it sort of three simple commands.

• 46:16Speaker 02

  The whole point of Asimov's story is that three laws don't work. He came up with the three laws and then wrote a whole book series about why they don't work. Even if you could program them, they still don't work. This is, the thing is, is that when we're talking about something this powerful, we're not just talking about a tech, so there's a technical side I could approach. Like, we don't know how to encode values into AI. They constantly do things we don't want them to do. That's one side. There's a philosophical side. We're like, even if we could encode, we don't even know what the laws would be. Because the Asimov's three laws don't work. He wrote a whole series of books about why they don't work. So we, we can't even encode them. Even if we could encode them, they wouldn't work. There is, and like, and then there's also like a moral dimension of like, if we could encode things and we have things that like could work, who gets to decide that?

• 47:00Speaker 02

  Like, who gets to make that choice? Like, who gets to decide, I'm going to enshrine the rules by which you and your family live for the rest of time? Who gets to make that choice? How do we make choices like this? There is like, and how do you stop someone else from getting in your way? How do you stop some other system from getting in your way? There are like many levels. And then, but you asked me originally, like, why do I think it's adversarial? And the fundamental reason is that that's what we're building. We are trying to make things that compete, that make money, that gain power.

• 47:26Speaker 00

  Competition leads to this adversarial nature?

• 47:32Speaker 02

  Evolution. It's evolution. Nature is red in tooth and claw. By default, if you optimize things for power, for survival, for money, for whatever, you get optimizers. You get things that compete. Again, it's very important that there's not gonna be one superintelligence. There's going to be millions of them, billions of them, competing with each other, fighting each other for money, for resources, for power. And this will cull AI systems that are not able to compete. So even if you build like a really smart system, you know, that's like, you know, Fable seven or whatever, it's like really smart, but it's really nice. And it never does anything other people don't want. Well, then, you know, fucking Giga Death GPT, you know, Ultra Doom will just destroy it, obviously. So this will be a competition and

• 48:16Speaker 02

  Please look at nature if you want to see what happens when a species gets outcompeted.

• 48:20Speaker 00

  So that's where I wanted to go. So when, if you agree that evolution that gave us the first form of intelligence, at least that's evolved, would be us. So you've got human advanced intelligence and the solution that evolution came up with in us is to have intrinsic morality that can be encoded based on your environment. So it's going to be different, but it's pretty bounded. Now it's bounded by our biology and so we don't have that advantage when it comes to machines, but it does show that if you want to create a species that can multiply, not wipe itself off the face of the earth, that that would be one way to do it. It has intrinsic morality that even when no one else is around, it doesn't want to violate

• 49:05Speaker 00

  that, at least not too much. So you're just saying evolution has solved something, man, that we just haven't figured out.

• 49:10Speaker 02

  Yep. I think, I don't think it's solved it. We still have sociopaths, but I do think evolution just did come up with a bunch of brilliant things in the brain that we have no idea how they work. Like, we have no idea how social instincts and morality work in the brain. Like, our understanding of how, like, the brain works and how these systems work is, it's hard to overstate like how little we understand here. If we fully understood the brain, we understood how all emotions work, how moral intuitions work, how they're encoded, how they differ with sociopaths, this would increase my optimism that we could make something work here. I'm not saying it would solve it, but this would greatly increase my, my probability that things go well. But we are decades, centuries away from this level of understanding of the brain. I don't think the brain has figured everything.

• 49:54Speaker 02

  I think, is, would you trust any single person on earth to have unlimited power over every single other person on the planet?

• 50:02Speaker 00

  No.

• 50:03Speaker 02

  Yeah, exactly. Like, I think it's solved a bounded version of the problem, where there's a bounded version of the problem where the strongest person in the world is only this many times more powerful than me. And we know that if that number gets too high, that's actually already very dangerous. Dictators are very dangerous. If that number gets too high, very bad. Most of the way worlds in, with humans are good is when our power levels are all kind of like, you know, within a certain bar of each other. When we know we're not going to mess with people too much because, you know, bad things are going to happen and we will respect each other. But if you have something like superintelligence, this isn't like humans and humans. This is more like humans and ants. Humans do not in fact care about ants. We have a lot of moral intuitions, very nice moral intuitions.

• 50:45Speaker 02

  But if we want to build a highway and there's an anthill in the way, sucks to be the ants.

• 50:51Speaker 00

  So let me ask a, a question that will help me understand sort of the bounds that you put on your vision of AI. If we built AI on Mars and it had plenty of power, so it could expand, it could replicate, it could build a whole bunch more, would it still want to beeline to Earth simply because Earth has access to more resources? Or would an AI that didn't have to compete with us geographically, had access to its own resources, had the thing it cared most about, which is energy, would it still be like, now we gotta get those fuckers?

• 51:23Speaker 02

  I have no idea. Speculating about what a vastly superhuman alien intelligence would do in a crazy sci-fi scenario is not a very scientific thing to be doing. I have no idea. If people want to go build, you know, superintelligence on Mars, I think it would definitely be safe for them building it on Earth, so knock yourself out.

• 51:41Speaker 00

  Got it. What, the thing I'm really trying to tease out there is if you feel this is a resource acquisition thing, that when we create an intelligence, the reason, because even when you talk about the ants, we only move them or run over them when we want something and they're literally physically in the way. And so you've addressed the adversarial, there's just too many unknowns. We don't know how to program the moral intuition and therefore between that and just not knowing how the brain works, like maybe if we had more time, we'd be fine. But now if we're thinking about, okay, the things that we're trying to protect against, one of them is we become the anthill. And on that, I've

• 52:25Speaker 00

  long fantasized that the solution to the alignment problem, and trust me, I know it can't be this because I'm not that smart and people have already discarded this idea, but it'd be great to hear from you, like, what the thinking is, where it falls apart. My intuition has always been that one intrinsic morality heard that it's just technically not possible. And then the other one is that humans have an innate drive to survive. And I'm a game developer, you wouldn't know that about me, but, so I spend an inordinate amount of time thinking about motivations of players, how to make the world do what you want, how you have to create sort of a homeostasis, you can't have one rule that, you know, ends up knocking everything over. And so if you take Minecraft, the greatest

• 53:10Speaker 00

  video game ever invented.

• 53:12Speaker 02

  Agreed.

• 53:13Speaker 00

  I love that about you already. When you start up a new world, it, there's a joy to the world, but if you lose it while there's sort of a temporary, oh, that's sad, there isn't, I don't smash my computer, I don't go stab the guy next to me, I'm just like, I can start a new Minecraft world. So what I don't understand is why, unless it's something in the training data, we believe that an AI becomes sort of a power-hungry, human-like thing, instead of going, the digital world is far more interesting, as long as I have

• 53:51Speaker 00

  power, I can sort of spin up any virtual resource I want, again, effectively without limitation, especially if I can make more hard drives or chips or whatever, give me enough sand and power and I'm good to go forever. So is it the training data that makes them sort of shoot out of the cannon? Or could we actually, you're gonna say it's the same thing as the moral intuition, but is there not something we can do to get them to be perfectly happy in a virtual world where alive is just as interesting as off? Or is that just there's?

• 54:28Speaker 02

  We don't want them to be like that. We want them to make money in the real world. We want them to gain power in the real world. We are not trying to make little Buddhas that live in a virtual reality and have a good time. That is a waste of my GPU power. I need my investment back.

• 54:45Speaker 00

  So if we could convince people to have that gear, because my layman's way of explaining it to myself is, that it wants to do its task in the same way that I want to win at Minecraft, but I'm certainly not going to sacrifice my marriage, I'm not going to even sacrifice carpal tunnel syndrome. So it's like, it's, it is a thing I want to do, but it's bounded by the one you've already addressed, my moral intuitions, it's bounded by other desires. So I'm guessing we just terminate and it's too complicated.

• 55:17Speaker 02

  You are hitting one of the core problems. It's just, there are core questions of what's called bounded optimization that just we have not solved. I do think there are answers to the questions you are giving. I think there are algorithms we have not yet discovered that have these bounded properties that you described. But the thing that's like very important to understand, that you were talking about the training data earlier, is, you know, the early versions of GPT were what were called, you know, large language models, LLMs, you know, they were trained to predict the next word. It's very important to understand that ChatGPT and other systems have not been LLMs for years. This has not been the case for a long time.

• 55:50Speaker 00

  I've heard you say that before, but that's shocking. What do you mean?

• 55:53Speaker 02

  These systems, so LLMs is still a component of these systems, very important component, called pre-training, very important, where you train it to predict words. But nowadays, you know, up to half or even more of the compute that goes into the training run of a modern AI is not language modeling, it's reinforcement learning, which is very different. Reinforcement learning is not learning from training data. It's you give the AI a task, a puzzle, a problem, something, and you have it go run out and try to solve the problem. And when it solves the problem, you give it a reward, kind of like giving a dog like a treat. And then it learns from this. And so this is a very old idea, you know, we've known of our reinforcement learning since like the 1980s. And something that we discovered basically immediately is that whenever you use reinforcement learning to train your

• 56:34Speaker 02

  eyes, you get crazy sociopathic optimizers, always. This has been the case since the 80s.

• 56:39Speaker 00

  How do you make it care about a reward?

• 56:42Speaker 02

  So basically, there are many ways, there's many specific algorithms, let's not get into the calculus, but it's like mostly pretty simple in, in the way is just you make it try something many times, and when it succeeds, you see what did it do, and you update it to do more of that. And, I mean, that's basically it. It's like a very simple idea, in a sense. And this didn't really work that well for a long time, you know, it worked for like some small things, like playing video games and whatever. And even back then, what we saw is whenever you use reinforcement learning, the systems will cheat and lie and break and steal and do anything to get the reward. There's no limit to what they will do to get the reward, which is exactly what we're seeing now. This is, this is exactly what we're seeing. Humans are not pure reinforcement.

• 57:26Speaker 00

  And you can't call that a fail state? Like, oh, we detected you did this thing?

• 57:30Speaker 02

  If you, can you define every fail state ahead of time? Literally everyone, if you could, sure, but then you would have to write down all of morality, like all of it, and you'd have to write it in code.

• 57:40Speaker 00

  But it would theoretically work, it's just too complicated.

• 57:42Speaker 02

  Again, and we bring this back to morality problem, how do you encode morality into computer code?

• 57:46Speaker 00

  Yeah, if it would work, I at least know the problem that has to be solved. I grant you for today that we just don't have that kind of time and fair enough. But at least that tells me it's the right problem. It's just very, very complex to turn into math.

• 58:00Speaker 02

  Yeah, I think, I do think that if we spend three generations of our greatest scientists, mathematicians, philosophers, et cetera, who are working on bounded optimization algorithms, how does the brain solve this problem? How do you trade off various things? The understanding of inside of neural network. Another very important thing to understand is, uh, this is something that's sometimes confusing to people, is AIs are trained to do a certain thing, but that doesn't mean that's what they do. They were only rewarded to do a thing, but they might have learnt something different. The same way that like you might, you know, try to reward an AI to say good things, but it learns to just lie to you. That's obviously not what you wanted it to do. So it can learn non-intended things and that happens all the time. So to solve this, you have to actually understand what's encoded in the system.

• 58:43Speaker 02

  Like, there's no way for us to look at an AI and look at like lying circuit or whatever. There's a bunch of papers who claim they can do that and they're all complete pseudoscience. Like, it's complete pseudoscience. It's similar to like, you can open up someone's brain, you can look at all the neurons, they're right there, you know, and maybe some of them glow a little bit more when someone says, you know, talks about dogs versus cats, I don't know. But that's not enough, that's not computer science, you know? If we had like the level of understanding that we have of like normal programming languages for like AIs, we can like run like, you know, compilers in there and we can like do formal verification of like, it will never lie under any of these circumstances. And then we can like compare these to our bounded optimization algorithms. I'm like, yeah, maybe. Like, if someone from the 22nd century came back with their like, you know, compiler stack

• 59:28Speaker 02

  and their like math and their bounded optimization algorithms, maybe. Like, I, it's imaginable to me, but we just don't have that.

• 59:38Speaker 00

  Are there any credible voices out there right now that see a path? Not that they think they already have the answer, but they see a path and they're like, we just have to buy more time to get this done? Or is it, there's no credible path forward?

• 59:50Speaker 02

  I think there are many paths forward that involve us having more time.

• 59:53Speaker 00

  But it's just, it's really astronomical amounts of time.

• 59:56Speaker 02

  I, I don't, I'm not even that certain, you know? Like, I'm not that certain that, I don't think, like, it's not gonna take a thousand years.

• 1:00:02Speaker 00

  No, no, but I'll call three generations is, for me, definitively astronomical amounts of time.

• 1:00:07Speaker 02

  That's not how I would use the word astronomical, but okay.

• 1:00:09Speaker 00

  I mean, wouldn't you grant me that's like 60 years?

• 1:00:11Speaker 02

  Yeah, I feel like astronomical, like, is like star related. I think 60 years are nothing to a star.

• 1:00:16Speaker 00

  I did not mean that literally.

• 1:00:18Speaker 02

  So, if something, to me, if something is within my lifetime, it's short. Like, uh, like personally, if my kids get to live in a great world, I'm like very willing to take that personally.

• 1:00:28Speaker 00

  Yeah.

• 1:00:28Speaker 02

  Maybe not everyone, but like if I had to like, you know, have not the perfect life, but I could like guarantee my children have an awesome life, I would take that trait personally. But I totally understand that maybe some people would not take this trait to varying degrees. And there is a trade-off here, but for me, like, the idea of like 10 years, 20 years, 30 years is like, for, in return, we get, um, tens of thousands of years of awesome future and like all of our problems solved and we don't go extinct, is like a really reasonable trait to make. Like, how many years would you be willing to trade from, we're definitely going to go extinct, to, we're definitely not going to go extinct? And like, my answer to this is like a pretty high number personally.

• 1:01:09Speaker 00

  Something you've said is a core question that we have to have before we can move forward, which is, what does the world look like that you want to live in? If you were going to, um, give us the sort of core tenants of what that ideal world looks like, um, please describe it. And then, one thing I really want you to include, because you're, I guess, I, I have such a base assumption that technology is the promise of a better future, that I imagine a better future being a one-to-one relationship with technology. And you're the first person I've heard say, technology is not the only way to improve your life, which as soon as you say it, it's self-evident. But describe for me the world and what that we should want to live in and what our relationship to technology would be.

• 1:01:50Speaker 02

  I'll give you two, uh, two kind of different answers or different intuitions for this. For one, I don't think describing like concrete utopias is often a very good thing to do, because we don't know what a true utopia looks like. So, when I think about a utopia or like a good world, what I think about is what I like to call a just process. What is a process by which the world can be on track? What I mean by this is, for me, what a good world looks like is not, we've solved everything forever and it's also, you know, luxury communism or something like, I don't know, right? Because I don't know what the right thing is. What is the right, for me, what is for other people? You know, what would the good world look like? I don't know, I don't, I haven't figured it out. No one has figured it out. And trying to, pretending you have figured it out is a straight shot dystopia.

• 1:02:34Speaker 02

  So, the way I think about it is, is that a good world would look like, not that we've solved all problems, but each and every one of us has the justified belief and feeling that tomorrow, the world will be a little bit better. Next year, the world will be a bit better. And the year after that, it's gonna be a bit better than that. Not saying that no problems happen, we don't make mistakes, but we would have the feeling of like, yeah, we're making progress, we're iterating, you know? Maybe sometimes we'll take a step back, but we have trust that it'll be fixed. For me, a good world would be like, you have the trust that if tomorrow, a new crazy technology gets invented, you know, someone invents super giga death device, whatever, right? You know?

• 1:03:16Speaker 02

  In a good world, your reaction to this would be, oh, I know the government, the, the good people, the government have it under control. I know all the smartest scientists will come out, you know, all the greatest, uh, military leaders will step up to the cause and will be heroic and will do the right thing. This is what it would feel like. A lot of it is institutional. When I, when I'm here in America, and I love America, it's a great country, but, but it's not really a good country, in many ways. In many ways, when I look at America, people don't have the assumption that, yeah, it'll turn out okay, we'll figure it out. Like, my dad, um, he, he was a good man and he was a very, he, he loved America. He was a very good man. And he got very ill, very, very ill. And insurance wouldn't pay. So, lost everything.

• 1:04:00Speaker 02

  And we were gonna be homeless. Then we moved to Germany, because my mother's German. And, you know, he got shipped to the hospital and they took care of him, they got him all the medication and all that, you know, treatments he needed. And he asked them, like, well, what do I owe you for this? And the doctor was like, what? You don't, you don't owe us anything. And they just left. Like, it was not, it was not like kindness, it was like process. It was just boring. It was just boring. He's like, yeah, you're sick, you get the treatment you want, goodbye. Like, for me, a lot of the good world looks a lot like this. It's like, oh, you want to create beautiful art? Of course, here's a bunch of beautiful art, you know, of course. Oh, there's a new threat threatening our country. Yeah, don't worry, the military's on it. Oh, some new crime is being committed, don't worry, law enforcement will take care of

• 1:04:45Speaker 02

  it and the courts will be just. That's for me, a lot of what a good world looks like. So that when, and for technology, it will be obvious that of course we're gonna regulate it well. So, it'll be, for example, social media will just be a good for people. It will obviously, we will regulate recommender algorithms in such a way that they're not addictive. We'll, in fact, make them so that they, you know, don't push people into down negative spirals. We make sure that they're, encourage people to get off their phone and spend time with their friends. We will have lots of beautiful architecture and beautiful art and people will spend time together in real life and with their kids and all that kind of thing. It will be, in some sense, boring. It will just be, I'm very inspired by the Enlightenment. The, the period in Western history where a lot of, including, you know, the American

• 1:05:30Speaker 02

  Constitution, really got many of its ideas from. And what I really like about the Enlightenment is that sometimes critics of the Enlightenment say, oh, it's this rationalism, you know, they're trying to be so rational and reduce everything to rationality, but this is actually not true. The Enlightenment thinkers much preferred the, the use of reason. They wanted to be reasonable. So for me, a good world isn't perfect, it's reasonable.

• 1:05:54Speaker 00

  Can you define it?

• 1:05:55Speaker 02

  That's part of the thing I'm trying to not define, is that all of us have an intuition of what is reasonable, and it's not a formal thing. A, a good world is not formalized. It's not a high modernist, you know, communist, you know, mechanical thing. It is, it, for me, in a good world, things still go wrong sometimes. People make mistakes, you know, maybe a cork made a wrong choice, you know, maybe, you know, there was a, there's an accident, like, things like this still happen. But every time you look into it, you'll be like, that's a reasonable mistake. Like, you see why this would have happened and you know that we'll fix it. Like, you know, there's someone in charge and they'll be reasonable in the way to fix it. This is for me what a good world looks like. That when, you know, superintelligence is on the horizon, that people will be reasonable. The scientists will go to the government and say, hello, we are the scientists and

• 1:06:41Speaker 02

  we think this and this will happen. And the government will be like, mm-hmm, yes, this is a very reasonable thing. We're gonna talk to all our international colleagues and figure out what the best way forward is. I'm not saying they're gonna get it right on the first try, but along the whole chain, everyone would just be like, oh yeah, that makes sense, that, that is a reasonable thing to do. That's kind of how I think a good world would look.

• 1:06:58Speaker 00

  Is, um, AIs exist today compatible with that world?

• 1:07:03Speaker 02

  As it exists today, it is not fatal to this world, but it is not AI in this world would not look like AI today.

• 1:07:11Speaker 00

  Interesting. Tell me why.

• 1:07:13Speaker 02

  Because it's not aligned with people. AI, most of AI by like, you know, dollar volume is like TikTok algorithms pushing slop to children to give them eating disorders. Like, that's the primary, like, a lot of the funding that has gone into modern AI comes from companies who built up massive war chests from, you know, pushing eating disorders onto children for decades in the 2000s. Like, that's where a lot of the money, the cash came from originally. The funding for deep learning and neural networks, so largely it was for recommender algorithms, and only later, turns out, was useful for other things as well. So, no, I think that, for example, in a reasonable world, if a company builds a product that, you know, for example, creates massive anxiety in teenagers, they wouldn't make any

• 1:07:58Speaker 02

  money. They would go out of business and maybe go to jail for that. And if someone made a product that makes, you know, teenagers learn good things and make new friends and be protected from predators and, you know, grow as people, they should make a lot of money off of that. And they should be, you know, rewarded for this. And we all, you know, give them a medal and say, "Good job," you know? Like, I, I, I don't, I think this is possible. Like, it's, it's, it's funny that people sometimes don't, can't imagine that, like, obviously, we could just make a social media that's nice. Like, obviously, we have the technology. Obviously, if we designed the algorithm to be empowering to people and good, we could. It's just, they would make less money. You will make less money, you know? You can make a less addictive, you know, cigarette by removing the nicotine.

• 1:08:41Speaker 02

  It will be less addictive and it'll be less harmful. These things are related in various ways. So, I think you can make very pro-social products with AI and with social media and all these other things. But the, and the bottleneck here is not technology. I'm not saying we need to invent some new technology we haven't invented yet. It is much more about regulation, it is much more about philosophy, it is much more about humane design, about what do we reward with our markets? Do our markets reward whatever optimizes, whatever reproduces the most? Or it's reward the thing that gives us actual value? This is the core problem of optimization. If we could just define a magic number that contains all goodness in the world, well, great, make the number go up. But every number we have, obviously,

• 1:09:26Speaker 02

  is not that. GDP isn't that, you know? GDP is definitely correlated with a lot of good things. I like having a lot of GDP. It's very nice. But at the same time, you know, like, it's crazy. Like, you know, I know, especially like, kids younger than me, do you know that, like, they don't dance at clubs anymore?

• 1:09:41Speaker 00

  It's wild.

• 1:09:42Speaker 02

  Because they get recorded. So, if you're like seventeen, and you're at a club or something, you dance and you approach a girl awkwardly, and you mess it up, you get recorded. That's awful. This is, this is terrible. And this is a, so from my perspective, from a, like, you know, to be like an autistic economist, a massive harm was caused here. Like, like astronomical harm was caused to this teenager, from my perspective. I think the seventeen-year-old was caused a massive injury by this fact. Who's paying for that? Like, who's, who's addressing this problem? Who's making?

• 1:10:13Speaker 00

  Literally just cut a check for seventeen billion dollars to end all their lawsuits, but...

• 1:10:18Speaker 02

  Exactly. But like, that doesn't solve the problem. None of these things solve the problem, because the problem isn't technology, in a sense. It's more like, what do we do with it? What do we want our society to look like? What are the institutions, the norms? Like, like, this is a small example. In Japan, every time you take a photo with your phone, it makes a noise. Always. You can't turn it off. So that if people in public take pictures of each other, they at least know about it.

• 1:10:43Speaker 00

  Yeah.

• 1:10:44Speaker 02

  Extremely small, right? But this is obviously a good thing. Obviously, you don't want strangers taking pictures of you in public, you know, or at least want to know about it, right? And so, this is an extremely small, just social thing that you can change. And I think many, many of the things we really need are like this. In the history of like, again, like the Enlightenment, what I thought was so beautiful, one thing that finds a beautiful Enlightenment, is that the people pushing the technology and the people pushing statecraft were, in a sense, often the same people, or they were hand in hand. They took inspiration from each other. They were talking to each other constantly. For them, the design of a state, of a, of a society, was another engineering problem. It was another question of how can we create a wonderful, nice society for people? And what I feel nowadays is kind

• 1:11:29Speaker 02

  of like, kind of what you were saying earlier, is like, a lot of people, the calls like techno-optimists, are really civilizational pessimists. They just believe that the only way to improve the world is through technology, rather than how we live our society, how our norms work, how our culture works, how our laws work, our institutions. Even so, when I ask people what are the top ten things that they're annoyed about, very rarely are those top ten things technological things. Almost always is like, "Oh, man, you know, I'm upset about politics. I'm upset about healthcare. I'm upset about, you know, all these kinds of, like, so, I'm upset that I can't see my friends enough. I'm upset that, you know, whatever," right? And I think we should take this seriously. I think this is, you know, I'm a technologist

• 1:12:14Speaker 02

  at heart, but what I care even more about is making the world a better place. And turns out, technology isn't the bottleneck. I would love to get to a point where technology becomes the bottleneck, you know? Where we've solved cultures so well, and we have such a beautiful government, that now, all we need to do is to invent better technology. I would love to live in this world.

• 1:12:31Speaker 00

  Yeah. That's a really good point, man. That actually really hit me. I'm very glad that you took the time to walk us through that. So, one, as a waypoint, do you think that AI and capitalism are like a toxic pair?

• 1:12:49Speaker 02

  I'm personally a big fan of capitalism. I think capitalism is one of the best ideas humanity has ever had. In moderation, the same way as any ideas. There's a thing that I think sometimes gets forgotten, is that markets are great. I love free markets. I love private enterprise. But it's one tool in our tool belt. And we know for which problems it's good and for which ones it's bad. You know, if I gave you a really good power drill, you know, you use it to drill in a screw, you might be like, "Wow, this is the best tool ever." But it's not, it's one tool. You know, if you need to saw a board, you're not going to use your drill. You're going to need a different tool for this. Markets and capitalism are very similar. Capitalism and markets solve many problems really well. You know, if you want a cheap, you know, commodified product delivered at an affordable

• 1:13:29Speaker 02

  price directly to your doorstep, by golly, do I have a system for you. And capitalism and markets have brought a massive amount of good to the world, all, you know, over the last, you know, hundred years. A lot of bad, too. But fundamentally, markets are optimizers. They're like reinforcement learning. They have a number that they make go up. And if that number can go up by ways you don't want, markets will do it. And this is why we have regulation. In all of history, we've always paired to these things. It's kind of like, it's kind of like MMA, you know? Like, I love MMA. I think it's so fascinating. I don't do it myself, but I think it's such a fascinating thing, because you have like the most dangerous people to ever live in history, the most dangerous warriors to ever

• 1:14:10Speaker 02

  live, you know, fighting, you know, to such an incredible degree. And they don't kill each other. You know, they can keep competing because we have rules. You need a lot of rules to allow this competition to be possible. You need a referee. You need a lot of rules to make sure that they can compete the next day, that they, you know, next match, that they can come back. And markets are very similar. You don't regulate a market, you just get monopolies. You just get some corporations, you know, hire private security to, you know, break your kneecaps so you don't buy your product, right? That's no good. To get a competition, you actually have to have a lot of rules. And I think this is beautiful. I think if we make, you know, good rules, and we have a good referee that is not the market, you know, that is outside the market, such as the government, we can make beautiful

• 1:14:55Speaker 02

  things. I think this is an example of just us failing to do our job. Like, it's kind of like, imagine someone in MMA found a new chokehold that makes them win but also kills the other guy. And we just don't ban it. Like, if we don't do that, then the sport of MMA stops existing. And so do all the people who participate in it. And there's a very similar thing here, where this has happened throughout history. Corporations find new exploits. They find new edge cases that we didn't think about before. You know, they find new ways to pollute. They'd find new ways to harm people, to scam people, whatever. And then we react to that. Then we're like, "Wait, hold on, you can't do that," you know? Before nuclear weapons existed, they obviously weren't illegal. And then, once we realized that they were, they were possible, we really quickly made sure that they were illegal

• 1:15:38Speaker 02

  for the market to build, because otherwise, the market would be happy to build nuclear weapons. Could you imagine? You know, how much big tech companies would love to build nuclear weapons and sell them to the highest bidder? Of course they would. They would love that. So, it's always, it's a design problem, and it's an evolution process. It's a, it's, it's an iterated process over time. We're never going to get it right on the first try, and that's okay. That doesn't mean markets are fundamentally, well, we should throw out all markets and move to communism. Neither does it mean we should have no regulation and just do markets. It's just, we're trying to solve a really hard problem, which is, how can we get an unaligned thing, which is the market, like, in the market, quote, writ large, does, is not a good or a bad thing. It's like an alien. It's like an AI. You know, it's like an AI. It's like a reinforcement learning AI. And how can we get it to do

• 1:16:23Speaker 02

  things that we want? And that's hard. We've made a lot of progress on it, and we just have to keep it ready.

• 1:16:29Speaker 00

  If you were going to regulate AI now to make sure that we don't cross the point of no return, have you thought about what the specific policies you'd want passed are?

• 1:16:38Speaker 02

  Yes. The, there's two major things that we think should be passed. The first and foremost is you should criminalize the creation of superintelligence. This is the thing you can just do in law. You know, it's illegal to murder someone. It's illegal also to attempt to murder someone. Even if you fail, you still can't do that. Similar thing with nuclear weapons. Even if you attempt to build a nuclear weapon, even if you don't succeed, the men in black show up at your door, you know? And this is good. So, this is the thing you can just do, you know? And it's much easier, you know, it, it's quite easy often to tell the difference between, you know, a company is building, trying to build superintelligence, and it's not trying to build superintelligence. Not perfect, but it's a start. So, this is the first common sense thing. It's just, if you're literally advertising, "I am building superintelligence. I just raised a billion dollars to build superintelligence," that's

• 1:17:23Speaker 02

  illegal. And you can't do that. Then, the second thing is, how do we prevent us from getting too close to superintelligence? And for this, fundamentally, because again, if a superintelligence is built, even once, even accidentally, it's too late. So, we have to regulate well before superintelligence, so we have enough buffer zone. It's kind of like, imagine we're going a hundred twenty miles per hour in thick fog, and we know somewhere there's a cliff, but we don't know where. My first suggestion is pull over. Before we argue about the speed, let's first pull over. So... First, criminalize, you know, superintelligence and then regulate the precursors.

• 1:18:02Speaker 02

  I have a list of precursors such as ability to self-reproduce, taskable horizon length, things related to genetic autonomy and so on, and monitor those very carefully. And be sure that any credible, large experiment that pushes on these frontiers is registered with the government, is that has oversight and that we can decide, no, you can't do that, that's too risky. Because luckily, at the moment, building really frontier AI that pushes towards AGI or ASI is unbelievably expensive. You know, we're talking billions of dollars. So there's actually not that many actors that this law would even affect.

• 1:18:40Speaker 00

  Yeah.

• 1:18:41Speaker 02

  This wouldn't affect 99% of companies. You know, there would be a very small percentage of mega, mega, huge tech companies that would be affected by this. And for them, yeah, we should obviously have oversight of them, the same way we have oversight of like Lockheed Martin or, you know, nuclear weapons facilities and so on.

• 1:18:58Speaker 00

  What about open source AI?

• 1:19:00Speaker 02

  Open source is a very tricky one, in the sense that I love open source. I built some of the first open source large language models in the world, in fact, and led a group called the Luther AI. I've been, I grew up in the open source world. I think open source has brought so much good to the world. I think Linux being open source and all this brought so much beauty and good into the world. But for me, it's not ideology. It's a question of, you know, should the blueprints for the F-35 fighter jet be open sourced?

• 1:19:28Speaker 00

  Probably not.

• 1:19:29Speaker 02

  Probably not. That seems like it would make the world a better place, you know? Should blueprints for, you know, bioweapons or, you know, nuclear bombs be open sourced? Probably not, you know? And here's a very similar thing, where if we get to the point that there are open source systems that are super intelligent or that can be made super intelligent, which I think we're pretty close to, then I think there's probably no going back in this regard. You know, if there's like one system that's almost super intelligent and it's only on one server in one country, you know, maybe we can delete it, you know, maybe. But if there are open source billions of things around and we know they are super intelligent or like, you know, two steps away from super intelligent, it's probably too late.

• 1:20:14Speaker 02

  I think the action that would be needed to, for example, you know, recall quote unquote an open source model are not feasible.

• 1:20:22Speaker 00

  All right. Bill Gates just said that he now thinks AI is dangerous. What happened? What made him change his mind?

• 1:20:29Speaker 02

  I'm not sure really how much he changed mind. Of course, I don't know him. I think, you know, he's a, he's a very, you know, he knows a lot of things, a smart guy. I'm sure he's seen a lot of scary stuff. But even all the way back in 2023, Bill Gates actually signed the Center for AI Safety Statement, which said that the mitigation of the risk of extinction from AI should be a global priority. This was signed by Bill Gates, Sam Altman, Dario Amade, Jeffrey Hinton, like lots of really, you know, big hitters. So this was all the way back in 2023. So I think, you know, again, I don't know him, but I think Bill Gates has always known of some of these risks and just has now been trying to, maybe he's now trying to message a bit more about them. I personally wish he would have gone a bit further in his letter. I think it was, you know, it did acknowledge some of the risks, but didn't go all the

• 1:21:14Speaker 02

  way to superintelligence and extinction risk, which is the thing that I am really so concerned about. Because in a sense, I don't think we have enough time that we will see massive job displacement. Like, I think we will just see superintelligence in the next couple of years way before massive job displacement even happens and it's already too late. I kind of wish we were in a world, you know, where superintelligence is 20 years away. Then job displacement will become a huge problem, but I'm not sure we're going to get to that point even.

• 1:21:43Speaker 00

  That's wild. That's wild. Okay, I want to see if I got your base assumptions right. If I've missed anything, let me know, but this has been very helpful. These are loosely in order, but, and there's one that I have a follow-up question on. Okay, so, superintelligence is definitionally, you maybe didn't go quite that far, but superintelligence is adversarial. So knowing that once you make that, it is going to compete and you're going to have problems therefore. Superintelligence will consider us a distraction or impediment to their goals. And so we have to worry about being the very thing they're competing against and we will lose. Unknown risk is evaluated at near peak danger levels, meaning if I don't know the outcome

• 1:22:28Speaker 00

  of this, like you said, first pull over if we don't know where we're going. So anytime we terminate in, I don't know what AI is going to do, or I don't know what the superintelligence is going to do here, pump the brakes.

• 1:22:37Speaker 02

  Obviously, there's spectrum here, as with any complex strategy. How do you deal with known unknowns or unknown unknowns? How do you calculate the probability that a country is going to invade you tomorrow? This is actually a problem that there's a lot of literature on. This is a thing that, you know, the CIA and, you know, various military agencies have spent many, many decades working very, very hard on. One of the recommendations, for example, the CIA makes is that always word your gut feelings in numbers. So this is a thing we didn't talk about. People often get annoyed when people say like, oh, I think there's a 20% chance of things going well. And they're like, that's not math. And I'm like, dude, yeah, they know, I know, it's not math. The reason you say, oh, 20% is not because I'm trying to make a mathematical statement.

• 1:23:19Speaker 02

  It's because there's the famous study, I think it was from the CIA, where they found that analysts who said likely or unlikely actually meant wildly different things. Like some people, when they said likely, they meant 90%. Other analysts, when they said likely, meant 30%. And so this was extremely confusing.

• 1:23:37Speaker 00

  Yeah.

• 1:23:37Speaker 02

  Because unlikely, likely doesn't mean anything. So if you say like 90%, it's still a vibe, but it's a much more concrete vibe.

• 1:23:44Speaker 00

  Right.

• 1:23:44Speaker 02

  It's a much more, when I'm communicating to you, if I say 1%, I'm trying to communicate not, oh, it's definitely 1% and not 1.2%. That's not what I'm saying. What I'm saying is it's on the order of 1%. It's like, given my beliefs about the world and, you know, looking at my models of reality, which, you know, you can evaluate to how much you believe my models of reality, this outcome has on the order of this percentage of outcomes. Of course, it's based on my models. What else is it supposed to be based on, right? Like, obviously you have views of the world. You know, if I asked you, how likely do you think it is that, you know, this thing will happen in the future? How do you evaluate this? Well, you use your model in your head. What else are you going to do? You know? So it's a very similar thing. And like, I try to make my model explicit, like with these base assumptions, I try to

• 1:24:28Speaker 02

  make it pretty explicit, this leads to this, leads to this, leads to this, therefore high probability. So if all of these statements that we've said so far were like, you know, a billionth of a percent probable, I wouldn't be worried, but I don't think that's the case. I think if they're all like, you know, higher than 1% and after their conclusion are still more than, you know, after multiplying them all together, we're still above 1% and the outcome is extinction, hell no. You know? If the outcome is like, oh, it's a one in a trillionth, I'm like, sure, but that's not where my numbers come out.

• 1:24:59Speaker 00

  Right. Makes sense? Okay. Financial incentives, power incentives and lack of strong regulation have allowed us to keep racing forward despite the dangers.

• 1:25:10Speaker 02

  Yep.

• 1:25:11Speaker 00

  For AI to work, it requires the AI to have a will of its own.

• 1:25:15Speaker 02

  Depends what you mean by will. I think it's.

• 1:25:18Speaker 00

  It's going to be an optimizer.

• 1:25:19Speaker 02

  Yes.

• 1:25:20Speaker 00

  Now that I know the language you use.

• 1:25:21Speaker 02

  This is how I would describe it. It will have to optimize for goals of some kind. It'll take actions, it will make plans, it will optimize towards objectives.

• 1:25:29Speaker 00

  And the sociopathic optimizer, as you described earlier, is the problem.

• 1:25:33Speaker 02

  Yes.

• 1:25:34Speaker 00

  Okay, AI.

• 1:25:35Speaker 02

  There's some other problems too, but this is really quite at the heart of it.

• 1:25:38Speaker 00

  Okay. AI, it's too complex to imbue AI with morality.

• 1:25:42Speaker 02

  Yeah, we have no idea how to do it.

• 1:25:44Speaker 00

  Okay. Life can be improved by things other than technology.

• 1:25:48Speaker 02

  Yep.

• 1:25:48Speaker 00

  It's possible to get China on board.

• 1:25:51Speaker 02

  Yep.

• 1:25:51Speaker 00

  And humans are not magical. This is the one that I need more clarification on. It really hit me when you said it, but I wanted to better understand, you were trying to explain something about AI by saying humans are magical, you need to understand that.

• 1:26:04Speaker 02

  Yeah.

• 1:26:04Speaker 00

  What, what do you mean when you say humans aren't magical? And what does that tell us about AI?

• 1:26:09Speaker 02

  There is a very fun thing, it has a Wikipedia article called the AI Effect. I recommend looking at the Wikipedia article, it's quite fun, which is that throughout the history of AI, people would always define like, oh, we'll know we have real AI when X. And then once X was accomplished, everyone like, oh no, that's not really AI. That's not true intelligence. So for example, back in the day, people thought that the thing that makes humans unique, different from all animals, is that we can do math. And well, obviously, computers can do math.

• 1:26:38Speaker 00

  Very well.

• 1:26:39Speaker 02

  Very well. So then they're like, okay, no, no, that's not what we mean. What we mean is actually the thing that makes humans magical is that we can recognize images or whatever, you know, and then you're like, or they can play chess or, you know, and obviously, none of those things were the thing. And so there's a, there's like this, like, rising tide, rising waterline, or like, you know, like, being backed into a corner of like, this is this very funny cartoon. I think it was made from Rick Kurtzwell or something. We have this guy frantically writing paper, like, only humans can do, only humans can play chess and like, strikes it out, like, only humans can draw pictures, strikes that out. And like, and it's like, this whole room is covered with these paintings of him, like, only humans can do, uh, uh, uh, you know. And what I'm saying with this is that humans are physics. We are, in fact, part

• 1:27:22Speaker 02

  of physics. There's nothing magical going on. It's complex. Our brains are complex, but it's not magic. It's, it's a, it's a physical process. There is some computation happening in our brain of some kind, you know, it's being, you know, it's using some weird squishy, you know, elements for the calculations. But fundamentally, some kind of calculation is going on. It's not magic. You can make other things do similar calculations.

• 1:27:45Speaker 00

  So it's a way of explaining, we know intelligence is possible without some sort of magical element.

• 1:27:49Speaker 02

  Exactly.

• 1:27:50Speaker 00

  It makes all the sense in the world. Dude, this is crazy. Um, this has really been amazing researching you and sitting across from you. I really, really appreciate it. I think your messages come in loud and clear. If there was one thing that people may not realize, like, if you were sitting across from Xi or somebody who you're like, I know they're convincible, is there one final thing that hasn't come up today that you'd want them to be aware of?

• 1:28:14Speaker 02

  I have different answers for Xi Jinping than I do for people listening. And so I'll give two answers.

• 1:28:19Speaker 00

  Perfect.

• 1:28:20Speaker 02

  So honestly, the number one thing I have found is the grown, not written. People don't know this. When I talk to politicians, is the first thing basically out of my mouth is explaining that this is not normal software. It is not written. It is grown. We do not understand how these things work. We did bring this up, but I'm repeating it again because this is just so important to understand. The people at OpenAI do not know what is going on inside of their AIs.

• 1:28:43Speaker 00

  So what?

• 1:28:43Speaker 02

  Dario Amade, the CEO of Anthropic, thinks he, we understand maybe 3%.

• 1:28:47Speaker 00

  Oh, my God.

• 1:28:48Speaker 02

  Of what goes on in our AIs. And I think that's even optimistic.

• 1:28:52Speaker 00

  Whoa.

• 1:28:52Speaker 02

  So this is one of the most important things to understand. We don't know. We, they don't know. No one knows. This is very important. And for the general public and everyone listening, I think the very important thing is, is that it's not over. And even though it's sometimes unfashionable to say this these days, we do still live in a democracy. And your voice is important. If you want something to happen, the way I like to think about civics is that our job is to work with our politicians to help them understand what we care about and what can be done. So do your part to help bring these issues to awareness. Go to controlai.org, contact your lawmaker, tell them you're concerned about these issues,

• 1:29:32Speaker 02

  help educate people, keep having the conversation. This is how we humans solve all these problems, you know, together.

• 1:29:39Speaker 00

  I love it. Where can people follow you?

• 1:29:42Speaker 02

  I'm, you can find me on Twitter, you know, and now X, sorry, at MP Collapse. You can find me, obviously, at controlai.org. And also, if someone wants to engage even deeper on some of these things, I actually run a volunteer group called Torchbearer, torchbearer.community, where we commit to spending at least two hours a week to try to build that reasonable world.

• 1:30:03Speaker 00

  I love it. All right, everybody. If you haven't already, be sure to subscribe. And until next time, my friends, be legendary. Take care. Peace. Let's talk about a pattern that is guaranteed to be killing your progress. You know what you need to do. You need consistent nutrition. We all do. You need vitamins, probiotics, greens. We all know that we should be doing more of it. When your morning gets chaotic, you skip it. When you travel, you skip it. When your routine breaks, everything tends to break. And that inconsistency compounds against you every single day. AG1 is designed to solve the execution problem. One scoop, eight ounces of water, and you're done. You're getting 75 plus ingredients, vitamins and minerals, pre and probiotics, nutrient dense

• 1:30:46Speaker 00

  superfoods, everything that used to require six, seven different supplements, and perfect planning now happens in one drink that takes about 30 seconds to make. Right now, AG1 is giving you $87 worth of free gifts with your first subscription. You get a welcome kit, travel packs, vitamin D3 plus K2, and flavor samples. Click the link in the show notes or visit drinkag1.com/impact to claim this offer.

• 1:31:14Speaker 01

  Tire Discounters presents Strange But True. You can now get four tires installed with a free alignment and oil change as low as $350. Oh, it's true.

• 1:31:28Speaker 01

  See store for details.