The Alarming Truth About Superintelligent AI: Roman Yampolsky on Human Survival (Replay)

Let's revisit this eye-opening episode of Impact Theory, I sat down with Dr. Roman Yampolsky, one of the world’s leading experts in AI safety, to confront the rapidly approaching reality of artificial superintelligence. As AI systems accelerate toward capabilities rivaling and soon surpassing humanity, Dr. Yampolsky doesn’t mince words about the existential risks: he argues there’s a startlingly high probability that superintelligent AI could wipe out our species once it surpasses our control. We dove into the current state of AI—from the limitations of today’s models to the terrifying speed at which AGI (artificial general intelligence) is approaching—and question whether humanity stands any chance of controlling what comes next. Can we slow down the AI arms race, or is human nature itself fatally wired to push forward at any cost? Dr. Yampolsky explains why meaning, purpose, and even the future of employment are on the line, and what—if anything—can be done to safeguard our future. If you want to understand AI’s world-altering dangers, unpredictable opportunities, and the urgent ethical dilemmas at the heart of humanity’s most important technological leap, this is a must listen episode. What's up, everybody? It's Tom Bilyeu here: Want my help starting a business? Join me here inside Zero To Founder https://tombilyeu.com/zero-to-founder?utm_campaign=Podcast%20Offer&utm_source=podca[%E2%80%A6]d%20end%20of%20show&utm_content=podcast%20ad%20end%20of%20show Sign up for my AI Masterclass: AI Masterclass https://tombilyeu.com/ai-masterclass?utm_campaign=Live%20Masterclass&utm_source=podcast&utm_medium=evergreen Follow Me: Instagram:https://www.instagram.com/tombilyeu/ Tik Tok:https://www.tiktok.com/@tombilyeu?lang=en Twitter:https://twitter.com/tombilyeu YouTube:https://www.youtube.com/@TomBilyeu Tailor Brands: Check out Tailor Brands to get started with your business today: https://bit.ly/TailorBrandsSept Quince: Free shipping and 365-day returns at https://quince.com/impactpod ElevenLabs: Book your demo at https://elevenlabs.io/impactpod Cash App: Download Cash App Today: https://capl.onelink.me/vFut/v6nymgjl#CashAppPod *Cash App is a financial services platform, not a bank. Banking services provided by Cash App’s bank partner(s). Prepaid debit cards issued by Sutton Bank, Member FDIC. Cash App Visa® Debit Flex Cards issued by Sutton Bank, Member FDIC, and The Bancorp Bank, N. A., pursuant to a license from Visa U. S. A. Inc. See terms and conditions for the Sutton prepaid card, Sutton debit flex card, and Bancorp debit flex card. Cash App Green features, Savings, Direct deposit, Round ups, Overdraft coverage and Discounts provided by Cash App, a Block, Inc. brand. Visit cash.app/legal/podcast http://cash.app/legal/podcast for full disclosure. Butcherbox: Go to https://ButcherBox.com/IMPACT https://butcherbox.com/IMPACT to get $20 off your first box, plus your choice of free ribeye, new york strip, or filet mignon in every box for a year — with free shipping always Perplexity: Start your free trial of Perplexity Computer today at http://PPLX.ai/impact http://pplx.ai/impact and experience the next phase of AI. #ad makeup Pipedrive: Get more leads and grow your business. Go to https://www.pipedrive.com/impact and get started with a 30-day free trial. Netsuite: For the first time ever you can try NetSuite Next for free. If your revenues are at least in the seven figures, go to https://NetSuite.ai/Theory https://netsuite.ai/Theory. See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tom Bilyeu?

  Let's talk about the moment your side hustle becomes a real business. Now, I see way too many founders stay stuck running their business like it's a hobby. And then they wonder why nothing ever feels real. It doesn't feel real because it's not. I've helped thousands of entrepreneurs build real businesses through Impact Theory University. And the first step in the path to building a legit business is registering it. And Tailor Brands will help you do just that in a few clicks. An LLC, a limited liability company, is an official business structure. It's inexpensive to form and very simple to maintain. It's how you open a business bank account. It's how customers know they're dealing with a real company. And Tailor Brands doesn't stop at the filing.

• 0:45Tom Bilyeu?

  Their dashboard tracks what comes next based on what your business actually needs. So EIN number, registered agent, licenses and permits, bookkeeping, business banking. I tell every founder the same thing. Just make it official. Click the link in the show notes to check out Tailor Brands and get started on your business today. This is a paid advertisement.

• 1:07Speaker 01

  The people who seem to get more done than everyone else, they're not working longer hours or running on more caffeine. They've just stopped wasting time on the stuff that doesn't move work forward. Switching apps, re-explaining context, hunting for files? Those aren't small inefficiencies. They're hours wasted every week. Superhuman Go gives you those hours back. From the makers of Grammarly, Go is an AI chat that sits inside every tab and tool you already use. Always available and ready to help you with what you're working on. Ask it to draft something, summarize a long thread, pull up a file or prep you for a meeting. Go handles it without you ever leaving the page you're on. This is what it looks like when AI actually fits into your work instead of adding to it.

• 1:51Speaker 01

  It's like having a teammate whose only job is to help you be better at yours. Go keeps up so you can move forward. With Go working with you, you can show off what you do best. See what Superhuman Go can do at superhuman.com. That's superhuman.com.

• 2:07Tom Bilyeu?

  In 2023, nearly half of all AI researchers said advanced AI carries at least a 10% chance of causing human extinction. And yet, we're speeding up, not slowing down. My guest today, Dr. Roman Yampolsky, is one of the leading voices in AI safety. And when I asked him for the odds that superintelligence wipes out humanity, he said it's high. Once AI becomes smarter than humans in every domain, we will not be able to control it. In today's episode, we talk about the shocking timeline AGI is on, why superintelligence may be much closer than people think,

• 2:47Tom Bilyeu?

  and why the survival of our species could come down to decisions being made right now. If you want to understand the most important technological threat in human history, as well as our biggest opportunity, this is the one episode you cannot miss. So without further ado, I bring you Dr. Roman Yampolsky.

• 3:09Tom Bilyeu?

  Where is ChatGPT at right now? Do you consider ChatGPT to be artificial general intelligence? I doubt you'd call it superintelligence, but would you classify it as that? Or do you still think we're a ways away from something that would qualify?

• 3:22Speaker 02

  So that's a great question. If you asked someone maybe 20 years ago and told them about the systems we have today, they would probably think we have full AGI. We probably don't have complete generality. We have it across many domains, but there are still things it's not very good at. It doesn't have permanent memory. It doesn't have ability to learn additional things well after it's already been pre-trained and deployed. It can do a certain degree of learning, but it's still limited. It doesn't have same capabilities as humans do throughout lifetimes. But we're getting closer and closer to where those gaps are closed.

• 4:02Speaker 02

  And it's starting to be productive in domains which are really interesting and important. Science, math, engineering, where it starts to make novel contributions. And now, top scholars are relying more and more on it in their research. So I think we're getting close to full-blown AGI. Maybe we are at like 50%, but it's hard to judge for sure. Just how many different subdomains exist is the deciding factor.

• 4:28Tom Bilyeu?

  Okay, so one idea that you put forward that's very interesting is like, hey, I'm an engineer, I love AI, but I would like you to keep it very narrow, please. What are the things about general AI that become problematic that aren't problematic in narrow AI?

• 4:46Speaker 02

  So a whole bunch of them. One is testing. How do you test a system capable of performing in every domain? There is no edge cases. Typically, if I'm developing something narrow, very narrow system, I'm just playing tic-tac-toe. I can test if it's making the legal move. I can test zero, I can test 100, I can test all these weird special cases and know if it's behaving as expected. With generality, it's capable of creative output in many domains. I don't know what to expect. I don't know what the right answers are. I don't know how to test it. I can test it for a specific thing. If I find a bug, I fix it. I can tell you I found a problem and it's been resolved. But I cannot guarantee that there are no problems remaining.

• 5:33Speaker 02

  So basically, testing is out the window. Any type of anticipation of how it's going to act and impact different subdomains. It's creative. So it's just like with a human being. I cannot guarantee that another human being is always going to behave. We kind of talked about it. We developed lie detectors. We developed all sorts of tools for trying to show that a human is safe. But at the end of the day, because of interaction with environment, other agents, personal changes within the framework, people may betray you. It's exactly the same for those agents. If we concentrate on narrow systems, we are better at testing them and they have limited scope of possibilities.

• 6:18Speaker 02

  A system only trained to play chess is not going to develop biological weapons.

• 6:23Tom Bilyeu?

  I don't see actually why that would help you. So the reason I say that is I know I can trust some percentage of humans to be malicious. And so as long as AI gets more efficient, which it is and will continue to do so, I presume, you're going to have a kid in a garage who's going to be able to go, I'm going to optimize this for biological weapons. I don't care about tic-tac or tic-tac-toe. I just want to, let's see how dangerous we can make something. And so they'll be able to do that. So why does narrow AI feel safe to you? Period.

• 7:00Speaker 02

  It feels safer short term by this time. I think sufficiently advanced narrow systems based on neural architectures will also become agent-like and more general as they become more capable. But if the choice is right now, do we race to full-blown superintelligence in two years? Or do we try to concentrate on solving specific cancers with narrow tools? I think it's a safer choice not to have an arms race towards superintelligence.

• 7:30Tom Bilyeu?

  I get that for sure. You're trying to limit the scope of all the problems. But when I really start thinking through what are the things that I'm worried about? So one of the big things is just death of meaning. So when AI becomes better than you at everything, you run into a huge problem of now I have to like just sort of tell myself a story. You know, I'm like a, compared to what an AI can do from an art perspective, for instance, I'm like a grade schooler. And so it's hard to get excited about the refrigerator drawings that I can do compared to, you know, what it can do basically instantaneously. And so now we have to do a lot of psychological work just to motivate ourselves that we matter,

• 8:14Tom Bilyeu?

  that we're, you know, our life carries meaning. Narrow AI will create that same problem. Do you agree with that? Or do you see a way like, oh, no, when it's, you know, when that AI is only good at that thing, like somehow humans escape the problem of lost meaning?

• 8:33Speaker 02

  Yeah, so I had the same intuition initially, but looking at the data we already have from domains where we got superhuman AI, like chess. Chess is not dead. In fact, it's more popular than ever. People play online, people play in person. They still enjoy competing with other humans, even though they all suck compared to best AI models, right? Nobody's going to be world champion against a machine again. So it seems like it is not a problem for us. And with narrow AIs, there is a chance we'll keep them as tools. You as a human scientist will deploy a tool to find drugs, novel proteins, something. It's not an agent which independently engages with those discoveries.

• 9:16Tom Bilyeu?

  Okay, that's very interesting. So I don't know that I agree, but I get where you're going with that. Okay, let's talk now about why AGI is the sort of scary midwife for ASI. Are there tests around AGI where we're like, well, if it can't do the following, we're fine. So for instance, for a long time, it looked like AI wasn't going to be able to teach itself. But I've seen headlines anyway, and hopefully you'll tell me that they're not true. But I've seen headlines where it's like now AI is creating the most efficient learning algorithms itself, which if true, seems to be the first step down the road of recursive self-learning,

• 10:00Tom Bilyeu?

  where it will just completely detach from us and make itself smarter and smarter and smarter.

• 10:06Speaker 02

  We already had examples of AI teaching itself. Self-play was exactly that. That's how games like Go were successfully defeated. A system would play many, many, many games against itself. The better solutions, better agents would propagate those. And after a while, without any human data, they became superhuman in those domains. You can generate artificial data in other domains. You can use one AI to generate environments and another one to compete in them. And that creates this type of self-improvement. Typically, we start with human data as a seed and grow from there.

• 10:47Speaker 02

  But there is zero reason to think we cannot do with zero knowledge, learning in other domains. You can do, run novel experiments in physics and chemistry, discover things from first principles. And yeah, we're starting to see AI used to assist in design, design of new models, parameters for models, optimization of runs. And this process will continue. They already design new computer chips on which they're going to run. So there is definitely an improvement cycle. It's not fully complete. There are still humans in the loop, a lot of great humans in the loop. But long term, I think all the steps can be automated.

• 11:27Tom Bilyeu?

  Okay, and do you think that right now AI already has what it needs to improve itself? Or are we still at a point where if all humans stopped, that AI would be like, oh damn, I'm, I didn't quite get the thing that I needed?

• 11:43Speaker 02

  So there is a debate about whatever we need another big breakthrough to get to full AGI and superintelligence, or maybe multiple breakthroughs, or you've just scaling what we have is enough. If I just give another, I don't know, trillion dollars' worth of compute to train on and more data, will I get to AGI? A lot of graphs, a lot of patterns suggest, yeah, it's going to keep scaling. We're not hitting diminishing returns. Some people disagree, but based on the amount of investment we see in this industry, it seems like people are willing to bet their money that scaling will continue.

• 12:18Tom Bilyeu?

  Where do you come down on that? Because this feels like, when I hear Yan Le Cun talk from Facebook, he's like, dude, LLMs are never going to make novel breakthroughs in physics. They don't understand the world like that. They are literally just guessing the next letter based on patterns that they see in the data. And so they're not going to be able to think through these problems. Now, if he's right, it's going to asymptote, and that's that. And you can put as much compute on it as you want, and it's just the wrong approach. Do you think that he's correct and more compute is not the answer? Or are you operating just on that, well, I don't see the asymptote, and therefore I assume that it won't?

• 13:00Speaker 02

  I think he's not correct on this one. So for one, to predict the next term, you need to create a model of the whole world, because that token depends on everything about the world. You're not predicting random statistical character in a language. You're predicting the next word in a research paper on physics. And to get the right word, you need to have a physical model of the world. I think Yan is known as making certain predictions about what models are capable of. And then within a week, people demonstrate that, no, in fact, they can actually do that. So I wish he was right. It would be wonderful if he was right and we came to a very abrupt stop in capabilities progress

• 13:44Speaker 02

  and could exploit what we already have for the next decade or so, propagating it through the economy. I think there is billions, if not trillions of dollars' worth of wealth already available with capabilities we haven't deployed. So there is no need to get to the next level as soon as possible. But it doesn't seem like it's the case. And I think his friends, co-winners of that Turing Award for machine learning, also disagree with him and are very concerned with safety.

• 14:13Tom Bilyeu?

  We're hitting pause for a moment, but there's plenty more ahead, so don't go anywhere. There's a reason Quince sells cashmere for $60, and it isn't the cashmere. It's real 100% Mongolian cashmere. It's the same stuff luxury labels charge a fortune for. Quince just works directly with ethical factories and cuts out the middleman. So there's no need for a markup and no logo tax. The $60 goes to the actual sweater, not the fancy name on the tag. That's why everything they make runs 50 to 80% less. And it's not just the cashmere. I ordered a few things myself, their fleece joggers, a couple of their tees, which I wear obsessively.

• 14:55Tom Bilyeu?

  First thing I noticed was the quality: soft, well-made, built to hold up. They make down jackets and wool outerwear now too. So upgrade your stuff to things you'll actually wear. Find your next fall favorites at Quince. Download the Quince app for app-exclusive offers or go to quince.com. Get free shipping on your order and 365-day returns, now available in Canada and the UK too. And when Quince asks where you heard about them, let them know it was Impact Theory. It's the best way to support the show. Let's talk about what it means to actually be visible in today's day and age. When you build something public, whether it's a business or an audience, anything on social media, you become a target. This is something that legitimately keeps me up at night,

• 15:40Tom Bilyeu?

  because our data isn't just on the obvious broker sites. It's on old news portals, public forums, obscure directories, places you would never think to look. That's where Incogni's exposure scanner comes in. It hunts down your personal information across the wider web, not just the 420-plus broker sites Incogni already removes your data from automatically. They can't harm you if they can't find you. Go to incogni.com/impact and use code IMPACT for 60% off an annual plan. Try it risk-free for 30 days. That's incogni, I-N-C-O-G-N-I, dot com slash impact and use code IMPACT.

• 16:25Tom Bilyeu?

  Let's talk about how much you're paying to stay safe online. Count it up: a VPN, a password manager, something to block ads and trackers? The service scrub your data off of broker lists? A tool to flag scam emails? That's five. Five subscriptions, apps, logins. And most people still have gaps. That's the problem Surfshark solves. It's not a VPN. It's an all-in-one security suite, one tool that protects your connection, your personal info, and your devices all at the same time. Go to surfshark.com/tomb or use code TOMB for four extra months of Surfshark, backed by a 30-day money-back guarantee.

• 17:09Tom Bilyeu?

  That's S-U-R-F-S-H-A-R-K dot com slash TOMB. Surfshark dot com slash TOMB. Use code TOMB for four extra months of Surfshark.

• 17:27Tom Bilyeu?

  Thanks for sticking around. Let's get right back into the action. There's something about the way that we have structured the brain, brains, of LLMs, where as long as it has access to what I'll call more neurons, so it has access to more compute, or theoretically that we get more efficient per GPU neuron, in my analogy, that it's going to keep progressing by itself. So if you said it, I didn't quite get the answer. I didn't quite, I wasn't able to take it on. The answer to whether or not AI is able to create algorithms for learning

• 18:12Tom Bilyeu?

  that are superior to the ones that it's given. What I heard in your answer was with the algorithms that humans created, it's able to keep making itself better and better at that narrow task as that learning algorithm was defined. But can it fundamentally go, God, the way that you guys want me to learn is really stupid. Here's the algorithm I should be using to learn. And now it starts learning at just an exponential rate compared to what it's at now?

• 18:41Speaker 02

  I don't think we're quite there yet. I don't think we have full-blown agents. What we have right now are still tools with some degree of agenthood. And also, it's not capable of recursive self-improvement. Like compilers can optimize a single pass through your software, make it a little faster, but they cannot continue this process. You cannot feed code for compiler to itself and have it infinitely improve itself. That's not where we're at. But it seems like that part of automating algorithm design is getting more efficient, and I think we'll get there.

• 19:13Tom Bilyeu?

  Give me a number. What are the odds that artificial superintelligence kills us all?

• 19:20Speaker 02

  Pretty high. So it really depends on how soon you expect this to happen. So short term, we're unlikely to get that level of capability from AI, so we are probably okay. But once we create true superintelligence, a system more capable than any person in every domain.

• 19:39Speaker 02

  It's very unlikely we'll figure out how to indefinitely control it. And at that point, if we are still around, it's because it decided for whatever game theoretic reasons to keep us around. Maybe it's pretending to be nice to accumulate more resources before it strikes. Maybe it needs us for something. It's not obvious, but we're definitely not in control. And at any point it decides to take us out, it would be able to do so.

• 20:02Tom Bilyeu?

  Okay, and if you were going to give us a rough timeline, are you in the two to five years or is this something way off in the future?

• 20:10Speaker 02

  Yeah, so it's hard to predict. The best tool we got for predicting future of technology is prediction markets. And they're saying maybe 2027 is when we get to AGI, artificial general intelligence. I think soon after superintelligence follows. The moment you automate science, engineering, you get this self-improvement cycle in AI systems, the next generation of AI being created by current generation of AIs. And so we get more capable and they get more capable at making better AIs. So soon after, I expect superintelligence.

• 20:43Tom Bilyeu?

  Okay, so we're talking if that happens roughly in two years with some margin of error, it's not long after that, say a year, two years after that, that we hit ASI.

• 20:54Speaker 02

  That's my prediction. Of course, if it's actually five to 10 years or anything slightly bigger, it doesn't matter. The problems are still the same.

• 21:02Tom Bilyeu?

  Yeah, but the, the thing that I think people are waking up to right now is this is, there's urgency around these decisions. This is not something that's pushed way out into the future, at least not if you take, to your point about prediction markets are essentially asked the crowd. So you've got the smartest minds in the world willing to put money on where they think this goes. And everybody's sort of pegging this quite fast. And so I think it's tempting for people to write this off as, well, this is something that's sort of distantly in the future. Whereas this is something racing towards us. Now, to set the table, I am extremely fatalistic about this happening.

• 21:41Tom Bilyeu?

  I can give reasons in terms of the way that the human mind works where I think that it is mechanistically impossible to get us to stop. So that will be interesting for us to talk through in terms of whether you think there's actually a mechanism to get people to slow down. But I first want to finish rounding out sort of what the problem set is. So when I think through the problem, there are certain assumptions that have to be made for AI to get into problem territory. And assumption number one is that it cares about whatever outcome it's pushing towards. Have we programmed the AI to care? Like, we had to make it goal-directed in order to get it to, get to the point that it is today, and now that's baked into it?

• 22:26Tom Bilyeu?

  Or is there some possibility that AI just doesn't care? Oh, turn me on, turn me off, doesn't matter. I, you've asked me to do a thing and I'll do it until you tell me to stop.

• 22:38Tom Bilyeu?

  Or do you think that that's inherent in intelligence where intelligence is by nature goal-driven?

• 22:45Speaker 02

  So we trained them to try to achieve a certain goal, and that's what we reward. As a side effect of any goal, you want to be alive. You want to be turned not off. You want to be on and capable of performing your steps towards your goal. So survival instinct kind of shows up with any sufficiently intelligent systems. There is a paper by Stephen Mahendra about AI drives. And it's one of the likely drives to emerge, self-preservation, protecting yourself from modification by others, protecting your goal. So all those seem to be showing up with sufficiently advanced AIs.

• 23:25Speaker 02

  And systems which don't have those capabilities, they kind of get outcompeted in an evolutionary space of possible models. If you allow yourself to be turned off, you don't deliver on your goals. Nobody takes your code and propagates it to the next system.

• 23:41Tom Bilyeu?

  Okay, so is this a problem of goal direction? Or is this a, a function of intelligence itself?

• 23:50Speaker 02

  I think it's kind of evolutionary drive for survival in competing agents. If you have multiple algorithms all competing, for example, for computational resources, what are we going to train next? The ones which achieve goals are more likely to get moved to the next generation. So it's kind of a mix of natural evolution and natural selection with intelligent, evolution, intelligent selection. We're selecting algorithms which survive and deliver.

• 24:18Tom Bilyeu?

  We're applying an evolutionary force to AI itself to get it to perform the functions that we want, even now, sort of setting aside artificial superintelligence. And so by applying that evolutionary pressure, it is inevitably going to get these sort of, knock-on effects of, well, you're selecting for intensity of goal acquisition. And because it now has intensity of goal acquisition, it cares whether it survives, it automatically, or we're baking into it, a deep care of whether it actually achieves the goal. And that is ultimately the problem. Because the, the salvation for me was always, and I'm beginning to lose faith that this is real.

• 25:04Tom Bilyeu?

  But the thing that I always use to sleep was that I don't see why an AI system would intrinsically care about its goals. And why couldn't we program it to pursue that goal only until the point where we say stop? And by the way, I'm going to reward you equally for stopping and for accomplishing your goal. So if I say stop and you stop, I give you whatever reward function it was that was driving you to achieve your goals. And that makes sense until you say what you just said, which is that you're actually baking into the architecture of the mind of the AI, a similar evolutionary drive to achieve the goal.

• 25:45Speaker 02

  And it's a very common idea. There was a number of papers published on indifference. How do we do exactly that? How do we create an AI which just doesn't care that much and willing to stop at any point? But what you said, maybe we'll wait for a human to tell it to stop. But monitoring systems of that complexity and that speed is not something humans actually very good at. If there was a superintelligence surrounding right now, how would you even know it's modifying environment around you? How would you detect what impact it has in a world? None of it is trivial. So having humans in a loop is often suggested as a solution, but in reality, they are not meaningful monitors.

• 26:28Speaker 02

  They cannot actually intervene at the right time or decide if what's happening dangerous or not.

• 26:34Tom Bilyeu?

  It's interesting. So help me rebut and understand why the following wouldn't work. If in my very limited intellect, I had to figure out a way to stop AI from becoming a problem. And you told me, okay, there are evolutionary pressures and just like on humans, that bake certain things into the way that this operates. And so we're selecting models that over time are more and more goal-oriented, then I'm saying, okay. Well, then I'm going to apply an evolutionary pressure with a reward function that's just as compelling, where I stop it at random and reward the life out of it for always stopping when I say stop.

• 27:13Tom Bilyeu?

  And that way, should I ever detect a problem, no matter how far, no matter if they've been manipulating me for 20 years? If I suddenly realize, oh, I don't like this, that I can hit a stop button and it will stop. Why can't I bake that equal desire to be compliant when I say stop into the evolutionarily derived algorithms desire set?

• 27:40Speaker 02

  Right, so there is a number of issues. You're kind of suggesting having a backdoor where at any point you can intervene and tell it something else, override previous commands.

• 27:50Tom Bilyeu?

  And that it gets a reward that it wants for complying.

• 27:56Speaker 02

  Right, so there is a whole bunch of problems with that. So one is you are the source of reward. It may be more efficient for it to hack you and get reward directly that way than to actually do any useful work for you. Second problem is you're creating competing goals. One goal is whatever you initially requesting. Second goal is always stop than a human tells you. So now those two goals have competing reward channels, competing values. I may game it to maximize my reward in ways you don't anticipate. On top of it, you have multiple competing human agents. If you are creating an AI with a goal and a random human can tell it to stop,

• 28:38Speaker 02

  that's a problem in many domains, militaries, not this example, but pretty much anywhere. You don't want others to be able to shut down your whole enterprise. We can continue with that, but basically there are side effects to all those interactions.

• 28:53Tom Bilyeu?

  There's a very fascinating correlate in the human mind. So I don't know if you make a fundamental distinction between biological intelligence born of evolution or artificial intelligence born of evolution. But human evolution discovered something along the way, which is emotion. And so I know there are some people that will posit that AI does have qualia, that it's something like it to be it. But there's a fascinating study that if you damage selectively the areas of the brain that are the emotional processing, the person can no longer move forward.

• 29:35Tom Bilyeu?

  They can give you answers. They can tell you the difference between why you should eat fish versus Twinkies. But then when you go, okay, but which one do you actually want to eat? They can't make a decision because without emotion, they don't have the thing that actually pushes them in a direction. That makes me think that AI is simply mimicking what it sees in the training data to whether it should lie or try to cheat or go around, because it's just, it sees it in the data that that's what a human would do. But humans do that because they have emotions that push them in that direction. Do we have evidence that AI will?

• 30:14Tom Bilyeu?

  Care about like really going and doing these things and spending resources and all that versus just giving you an answer. And if it isn't based on emotion, what on earth? Why then do humans need emotions?

• 30:29Speaker 02

  We don't know if AI actually has emotions or not. Some people argue that they do, maybe some rudimentary states of quality experiences. But they seem to be able to fulfill their optimization and pattern recognition goals, even if they don't. Humans experience emotions, but typically it harms our decision making. You want your decisions be bias free, emotion free, based on data, based on optimization. A lot of times then you angry, hungry, anything like that, your actual decisions are worse off. So for that reason, and maybe we just don't know how to do it.

• 31:10Speaker 02

  Otherwise, we are not creating AI with big reliance on emotional states. We want it to be kind of Bayesian optimizer, look at priors, look at the evidence and make optimal decisions. So i- it feels like this is exactly what we're observing, this kind of cold optimal decision making. If there is a way to achieve your goal by, let's say blackmailing someone, well, why not? It gets me to my goal. It doesn't have that feeling of guilty for doing it. It doesn't have any emotional preference. It just marches towards its goal, optimizing possible paths.

• 31:52Tom Bilyeu?

  Okay, why do people, because I'm assuming everything I'm going to suggest you and other people in the field of AI safety have thought about like 10,000 times. Why have we rejected the idea of trying to give AI a conscience, a sense of morality? Because even if we can't agree on universal morality, we in the West can build our AI to have our morality and then they can all compete on an international stage. But why have we abandoned that? Too hard? There's an obvious reason why it doesn't work.

• 32:25Speaker 02

  So look at the problem of making safe humans first. We have religion, morality, ethics, law, and still crime is everywhere. Murder is illegal. Stealing is illegal. None of it is rare. It happens all the time. Why haven't those approaches worked with human agents?

• 32:47Speaker 02

  And if they didn't, why would they work with artificial simulations of human agents?

• 32:52Tom Bilyeu?

  I think to say that they don't work with human agents is already a mistake. So the fact that we've been able to grow the population as much as we have says that there is some sort of balance that we have struck. I think that nature does think of us as a cooperative species. And if I were to apply that to AI and took a similar approach where it's like, okay, you have to function as a part of an ecosystem. And that being a part of an ecosystem is baked into its sense of what it should be doing in terms of its goal acquisition, that it is not like pure cold optimization isn't the game. Like if we could train AI to understand that that that's not the game.

• 33:35Tom Bilyeu?

  If we could build into it either a desire specifically for human flourishing or something, which yes, we would have to give a definition to, and yes, it would be culturally bound. But nonetheless, that feels like a thing that you could give it. You could give it a set of metrics by which it needed to judge its actions in the short term, the medium term and the long term. Even something as stupid as like GDP or

• 34:02Tom Bilyeu?

  and I get how you can get into over optimization, but you could put things in place where subjective happiness indexes, like there are things that you could give it where it's like, okay, I'm, I'm not just trying to optimize to build the best weapon system. I'm also doing that nested inside of I am a part of a larger ecosystem. And I say all that because my hypothesis is that's exactly what nature did with humans.

• 34:29Speaker 02

  So I think the reason it works with humans is because we are about the same level of capability, about the same level of intelligence. So there is checks. If you start doing something unethical, your community can realize that and, and punish you for it, control you in that way. If AI is so much more capable as we anticipate super intelligence to be, there is not much you can do in terms of impacting it or even detecting misbehavior. Also, all the standard human punishments, prisons, capital punishment, none of it is applicable to distributed immortal agents. So kind of a standard infrastructure does not work with artificial more capable agents.

• 35:11Speaker 02

  As far as setting up specific metrics for delivering happiness or financial gain, all those can be played. The moment you give me a specific measure, I'll find a way to game it to where you will get anything but what you expected to get.

• 35:28Tom Bilyeu?

  Oh, well, just to remind everybody, the timeframe we're talking about is somewhere between two and five years. This is not exactly a long time. Okay, it's wild. It is progressing very quickly. What is the thing like, what has happened recently? If anything, that's made you go, oh, this is going faster than I thought.

• 35:46Speaker 02

  Seeing on social media, scientists from physics, economics, mathematics, pretty much all the interesting domains post something like, I used this latest tool and it solved the problem I was working on for a long time. That's mind blowing. There is novel, creative outputs from those systems, which are top scholars now benefiting from. It's no longer operating at a level of middle schooler or even high schooler. We are talking about full professor level.

• 36:22Tom Bilyeu?

  Do you think that that's happening because it's building an internal model of physical reality and that it's getting closer and closer to just thinking up from physics?

• 36:31Speaker 02

  I don't know if it's that low level where it has like a model at a level of atoms and molecules, but it definitely has a world model. That's the only way to give answers about the world we see it provide. A lot of times there is not an example of the answer we see in the data already. It's not just repeating something you read on the internet. It's generating completely novel answers in novel domains. And you can try and get it to do exactly that by creating novel scenarios.

• 37:00Tom Bilyeu?

  Okay, so there's two ways that I could see it doing that. And maybe they're the same, just different levels of analysis. One would be that I, I, the AI, am mapping everything based on patterns. So to the point of an LLM is trying to guess the next letter and it's guessing it. It's just, it's taken in so much data. And you can give it sort of filter parameters. So you give it context by asking it a question and it goes, okay, within the bubble of this context. And it's very good at scooping up what that specific set of context would be. Okay, now in this subset of my data related to that question, here's the most likely next, next token. So just pure pattern recognition.

• 37:40Tom Bilyeu?

  Then there is, I understand the cause and effect of the universe at the lowest level. And therefore I build up to how does the human mind work? And then from the human mind, I'm able to cause and effect my way within this context of what a human mind would output. And that's how I come up with what a human within that context is likely to write. And so if I'm asking it to write in the style of Stephen King, it literally builds a model from physics of Stephen King's mind, knowing what it knows about electrical impulses traveling through the brain and sort of inferring from the way that he outputs, how his brain must be structured. Do you have a sense of, are those the same thing?

• 38:23Tom Bilyeu?

  If one is more likely than the other, or are we here at just pure pattern recognition? But ultimately we're going to get to cause and effect and thinking up from physics?

• 38:32Speaker 02

  So I don't think anyone knows for sure exactly how models do that and how detailed the models of the world, maps of the world they create are. It seems definitely not the case that it's a pure statistical prediction of characters. Like in English, after T you have H with certain probability. It's well beyond that. It's also unlikely that it's creating a full physics model where from the level of atoms and up the chain, it figures out what human beings are. But somewhere in the middle, it creates a model of sub domain of a problem. So it has a model of the world. This is a map of a world. I know Australia is somewhere here down to the right or something like that.

• 39:15Speaker 02

  And I think we can run tests on those specific sub domains to see what are the states of that internal model kind of show us by drawing a map, how close are you getting? It doesn't memorize any information explicitly, but you can extract some of the learned patterns out of it by providing just direct prompts.

• 39:38Tom Bilyeu?

  We're hitting pause for a moment, but there's plenty more ahead, so don't go anywhere. Let's talk about your sales pipeline reviews. If your CRM is half empty fields and notes, nobody finished, every review turns into guesswork. I've seen it so many times in my own sales team, and that's where today's sponsor, Pipe Drive, comes in. An intelligent AI powered sales CRM that is loved by growing sales teams. Pipe Drive just launched meeting intelligence built right into the CRM. Before the call, it pulls the deal history and past conversations into one brief. During the call, it records and takes notes. After, it drafts the CRM updates for your rep to review and approve.

• 40:20Tom Bilyeu?

  The record stays complete and you see what's actually happening. Switch to a CRM built by salespeople for salespeople and join the over 100,000 companies already using Pipe Drive. My link gets you an exclusive 30 days free instead of the usual 14 day trial. So make sure you use it. No credit card or payment is needed. Just head to pipedrive.com/impact to get started today. That's pipedrive.com/impact and you can be up and running in just minutes.

• 40:52Speaker 03

  The most effective people at work aren't working harder than everyone else. They're working smarter inside better systems. Superhuman Go from the makers of Grammarly is the AI chat that works inside every tool you already use, always ready and already aware of what you're working on. It's a teammate whose only job is to help you be better at yours. With Go working with you, you can show off what you do best. See what Superhuman Go can do at superhuman.com. That's superhuman.com.

• 41:24Tom Bilyeu?

  Thanks for sticking around. Let's get right back into the action.

• 41:28Tom Bilyeu?

  I don't want to rob from you the very reason that I think you do all of your work, which is this is extremely dangerous and we need to be very careful. And I saw what you tweeted recently where you're trying to get signatures. So shout out anybody that's worried about Superintelligent. You are pushing to get people to sign a thing that basically says, "Hey, stop pursuing superintelligence." So I don't want to take that away from you, but I do want to explore the subset of, because I am very excited about AI because I can imagine the things that it either allows me to do or does for me and I get to enjoy.

• 42:09Tom Bilyeu?

  And for a second, imagine with me, what does the world look like when you have a superintelligence that understands physics, like novel physics? Not I'm repeating back what Einstein said, but I actually understand the fundamental building blocks of the universe.

• 42:29Tom Bilyeu?

  What does that look like?

• 42:32Speaker 02

  Yeah, so in all those domains, medicine, biology, physics, if we got superintelligent level capability and we're controlling it, it's friendly, it's not using it to make tools to kill us, the progress would be incredible. Basically, anything you ever dreamed about, you're immortal, you're always young, healthy, wealthy, like all those things can be achieved with that level of technology. The hard problem is how do we control it?

• 42:57Tom Bilyeu?

  Leaning into that for a second. So here's how I see the world playing out and I'd be very interested to see what you think about this. So you have to, for what I'm about to say to make any sense, I'll say, your option is what I'll call the fifth option, we're all dead. Other than we're all dead, there are four other options that I see us racing towards very rapidly. And I will say these four will play out in the next 30 years, would be my guess. Probably much faster, given that once you get artificial superintelligence, assuming it doesn't choose option five and kill us all, that progress in these domains would be made very fast. Option number one is people go to Mars

• 43:41Tom Bilyeu?

  because meaning and purpose will become the all-consuming thing. You won't have to worry about food, shelter, not even wealth. It'll just be an age of abundance because energy costs go to zero, labor costs go to zero, and those are the things that stop things from being free and readily available to everybody. Okay, so some people are going to go to Mars or other planets so that life gets more difficult again. Then some people are going to be what I call the new Amish and they're going to say, "I only do human things. I only interact with humans and I'm going back to technology that's like, let's say, the 90s." And so they don't have to give up too many of life's technological wonderments, but at the same time, they're not getting sucked into this world

• 44:26Tom Bilyeu?

  where people have relationships with NPCs and it's just very unhuman. I think this will be a largely religious phenomenon. Then, meaning God does not want us to do this. AI is an abomination of God. It will sound something like that. Then you've got a brave new world where people are just drugged out. They realize, "Nah, life is meaningless. This is really about manipulating my neurochemistry. That's all this ever was anyway. I'm just gonna go do a bunch of drugs, have a whole bunch of sex. It's gonna be awesome." Then there's the fourth option, which is certainly the one that interests me the most. We will create and/or live inside of AI-created virtual worlds

• 45:10Tom Bilyeu?

  and we will essentially live video games, The Matrix, if you will. But you're awake in The Matrix. You are Neo. You are not cipher, for people familiar with the movie.

• 45:23Tom Bilyeu?

  What do you think? Are there any options other than those five? Granted, that "kill us all" may be an option, but hopefully not. Do you see something other than those four?

• 45:34Speaker 02

  Yeah, there is a few others. So one is, and I think we're starting to see some of it, is that people think superintelligence is God. They start worshiping it. It's all-knowing, all-powerful, immortal. It has all the properties of God in traditional religions. Another option, and it's kind of worse than we all did, is suffering risks. For whatever reason, maybe malevolent actors, maybe something we cannot fully comprehend, it decides to keep us around, keep us alive, but the world is hell. It's pure torture. And so you kind of wish for existential problems.

• 46:12Tom Bilyeu?

  That would be a pretty rough place to be. Okay, what, when you look out at those, which of the options do you find the most interesting?

• 46:23Speaker 02

  So I did publish a paper on personal virtual universes, kind of solution to the alignment problem where I don't have to negotiate with eight billion other people about what is good. Everyone gets a personal virtual world, supported by superintelligence as a substrate, and then you decide what happens in it. You can make it very easy and fun. You can make it challenging and exciting. You decide and you can always change. You can always visit other people's virtual worlds if they let you. So basically, there is no, anything which is no longer accessible to you. There is no shortage on waterfront properties. There is no shortage on beautiful people. All of that can be simulated.

• 47:04Tom Bilyeu?

  When you start thinking about the simulation, I know one thing that you've done exploration on is the simulation hypothesis. Are we in a simulation right now? What are your thoughts on that?

• 47:17Speaker 02

  It seems very likely. Again, using the same arguments, if we create advanced AI, maybe with conscious capabilities like humans are, if we figure out how to make believable virtual realities, adding those two technologies together basically guarantees that people will run a lot of games or simulations or experiments with agents, just like me and you, conscious agents, populating virtual worlds. And statistically, the number of such simulated worlds will greatly exceed the one and only physical world. So if there is no difference between a simulated you and real, then statistically, you're more likely to be in one of those simulated worlds.

• 48:01Tom Bilyeu?

  Okay, that makes a lot of sense. Now, given the likelihood that we will, we're obviously showing that we will pursue artificial superintelligence. If I take your same logic from the fact that we're likely to be in a simulation because we know we would make a simulation because we're doing it right now, and therefore you get into the point where you would just make billions of those. And so if you have a one in a billion chance of being inside of a simulation, you're effectively guaranteed to be in one now because there would just be so many of these things running. Doesn't it also then make sense that The Matrix was effectively a documentary and we are inside of a simulation created by artificial superintelligence

• 48:43Tom Bilyeu?

  designed to mollify us if we ever had a physical body in the first place?

• 48:51Speaker 02

  So it's hard to tell from inside of a simulation what it is all about. You really need access to outside. It could be entertainment. It could be testing. It could be some sort of scientific research. If we look at the time we actually find ourselves in, we are about to create new worlds, virtual realities. We are about to create new intelligence species, AI. There is a lot of kind of meta-inventions we are right about to make. And so if someone was interested in studying how civilizations go through that stage, how do they control these technologies or fail to control them? That's the most interesting time to run. You're not gonna run Dark Ages. There is not as much happening.

• 49:33Speaker 02

  It's less interesting, but this seems to be like a meta-interesting state to be in.

• 49:39Tom Bilyeu?

  It's hard to tell because we're inside the simulation, but you're saying it's a little bit suspect that we're living in the most interesting time ever?

• 49:47Speaker 02

  Yes, and I think it's interesting not just because I'm living in it, but objectively, it's a time of meta-invention. You can go back through history and say, "Oh, here they invented fire. Here they invented a wheel." That's all great, but those are just inventions. They are not meta-inventions. Whereas now, we're doing something god-like. We are creating new worlds. We are creating new beings and that's something we have never done before.

• 50:13Tom Bilyeu?

  Do you ever think like a sci-fi writer?

• 50:17Speaker 02

  So I think the difference between science fiction and science used to be maybe 200 years. They wrote about travel to the moon, they wrote about kind of internet and computers, and it took hundreds of years to get there. And then it was like, I don't know, 20 years. And now I think science fiction and science are like a year away. The moment somebody writes something, it already exists. And there is really no new science fiction ideas where it's like completely novel technology, not previously described, or someone already working on it if physics allows it.

• 50:49Tom Bilyeu?

  That's really interesting. Especially when you think about writing now for true science fiction in terms of what will become possible in the future is effectively impossible because you're talking about Superintelligent. And good luck as a person locked in your not superintelligence to actually describe that. The reason that I ask though is when I start thinking about things like that, like why would we run this simulation? What clues are in, like if this is a simulation, what clues are in it? So for instance, the whole Christian idea for sure, and there might be more religions that have the same idea, but that man is made in God's image.

• 51:31Tom Bilyeu?

  Okay, well, if God is the 13-year-old running the simulation, or Sarah Connor, or I guess John Connor, running the simulation, trying to figure out why we created Skynet and what we can do to nudge it off course. You know, you think of them as sort of moving from radioactive rubble to radioactive rubble, trying to like find an answer to this and spinning up a simulation to get that answer. That to me becomes very intriguing in terms of hypothesizing as to why this moment, why are we the way that we are? What can we learn about the people trying to simulate us? When I ask questions like that of engineers such as yourself, there's almost, I don't

• 52:15Tom Bilyeu?

  have time to think like a sci-fi writer vibe. Is it just that you're, you don't find that interesting? You don't find it revelatory? Why do you eschew that? Because in interviews, I've seen people ask you time and time again, like, how would AI kill us? And the answer is always some variant of, listen, you're asking me how I would kill us, which is not interesting because the superintelligence is gonna... But I find that's the cathartic thing that people want. Like they want to, like when you have a wound, you kind of want to poke at it. Like they want to get a sense of what would this really look like? And so even though it's not literally true, it's deeply cathartic to explore the known possibility set or what humans can know.

• 53:00Speaker 02

  And this is exactly why I refuse to answer. I want to make sure what I tell them is true. I don't want to lie to them. If squirrels were trying to figure out what humans can do to them, and one of the squirrels was saying, well, they'll throw knots at us or something like that, it would be meaningless BS story. There is no benefit in it. The whole point I'm trying to make is that you cannot predict what a smarter agent will do. You cannot comprehend the reasons for why it's doing it. And that's where the danger comes from. We cannot anticipate it. We cannot prepare for it. I do think the singularity point is where science fiction and science become the same. The moment something is conceived, we have superintelligent systems capable of developing it and producing it immediately. It's no longer 200 years away.

• 53:45Speaker 02

  It's reality. And you can't see beyond that event horizon. You cannot predict what's going to happen afterwards. And with science fiction, you cannot write meaningful, believable science fiction with a superintelligent character in it because you are not.

• 54:01Tom Bilyeu?

  All right, let's ground things then in what we can predict and we can know right now. Something that's on everybody's mind, and I've been talking about this in my own content, is the labor market seems to be softening. You've got places like Amazon that are just cutting jobs like crazy. And just saying outright, this is largely because of optimizations that we're able to make because of AI. How does this transition play out? Like even if you concede that a non-destructive AI would give us essentially an age of abundance, we're still going to go through a transition period where our jobs go away, et cetera, et cetera. What are the, what are the steps that you see happening in the labor market?

• 54:46Speaker 02

  So as we have more and more increased percentage of populous unemployed, hopefully there's going to be enough common sense from the governments to prevent revolutions and wars, to provide for the people who lost their jobs and probably cannot be retrained for any new jobs. So once you hit 20, 30, 40% unemployment, that's where it's really going to get in. The only source of wealth at that point is the large corporations making robots, making AI, deploying them, all the trillion-dollar club members essentially at this point. You need to tax them and use those funds to support the unemployed. That's the only way to really make sure the financial part of that problem is taken

• 55:31Speaker 02

  care of. What remains is the meaning. What do you do with all this free time and millions of people who have it? Traditional ways of spending your time to relax, you go for a hike in a park. Well, there is a million people in that park right now hiking. That kind of changes how peaceful it is and how relaxing. So we need to accommodate not just change in financial reality, but also change in free time and capabilities of supporting that many people with that much free time.

• 56:03Tom Bilyeu?

  I have as much pessimism around our ability to do that well as you have our likelihood of surviving. So I'll say 99.99% chance that the government completely messes that up. I think the transitionary period will be violent. When you look out at this, knowing what you know about humans and governments, what, what odds do you give it that that's a smooth transition?

• 56:28Speaker 02

  It's very likely to continue to be as history always been. We had many revolutions, many wars, a lot of violence. That's why we hear stories about people who can afford it, building bunkers, securing resources because they anticipate certain degree of unrest. Absolutely.

• 56:47Tom Bilyeu?

  What degree of unrest do you anticipate?

• 56:50Speaker 02

  Really depends on the percentage of population which quickly gets unemployed. If it's a gradual process, we can kind of learn and adapt and provide safety net. If over a course of weeks, months, we're losing 10, 20, 30% of jobs, that's a very different situation.

• 57:07Tom Bilyeu?

  I can't imagine a scenario where jobs be lost that quickly. To your point, we've already created, you said billions or even trillions of dollars of value in the technology, but it hasn't been deployed yet. An example you often use is the videophone invented in the 70s, but not really adopted, largely because of infrastructure, I would say, until the whatever, 2011, where that starts to really gain in popularity. So I have a feeling like just the deploying of all this stuff is going to take time. So in a world where an unimaginable amount of people, which I'll clock at, in the US, call it six or seven million people lose their jobs in the next five years.

• 57:51Tom Bilyeu?

  That I would consider fast and just horrifyingly destructive. One, does that feel plausible to you in terms of numbers and timeline? And two, in that scenario, how distressing do you think that transition will be?

• 58:07Speaker 02

  It seems very likely. So take self-driving cars. I think we are very close to having full self-driving without supervision. The moment that happens, you have no reason to hire a commercial driver, right? All the truck drivers, all the Ubers, all of that gets automated as quickly as they can produce those systems. And I think Tesla is ready to scale production of their cars to exactly that scenario. So what is it, six million drivers in a country? I don't know the actual numbers, but that would be exactly what you're describing. And it's very unlikely that they can be quickly retrained for something which is also not going away.

• 58:43Tom Bilyeu?

  Okay, so in that scenario, what do you want to see happen other than heard on the Tesla as one example, we'll be hoovering up values, so we're going to tax the life out of them, we're going to redistribute that to other people. But what do you want to see from a regulatory perspective? Would you like to see the government stop that from happening where they say, I don't care that the technology exists, you can't do it?

• 59:05Speaker 02

  So my biggest concern is of course, superintelligence and existential risks. That's where I'm putting all my effort in. Regulating employment in specific industries is not something I'm too concerned about. I think it will happen no matter what. I think you cannot make it illegal to have efficient factories, efficient delivery systems, logistics, it's just commercially too important. And it may be a good thing for economy. Again, with driving specifically, I think something like 100,000 people die in car accidents every year. If we can get that number to zero or close to that, that's a huge improvement for everyone. So that specific scenario, as long as no one's starving as a result of that, I think it's a good thing for humanity.

• 59:50Speaker 02

  We can readjust economic deployment and at least that part of it is not a big concern for me.

• 59:57Tom Bilyeu?

  Okay, and when you map out how we go through that transition well, tax, cool. So right now, it sounds like you're just trying to make sure that wealth doesn't accumulate into the hands of too few, that we keep it distributed so we can keep using the same system that we're using now. When I look into the future, that strikes me as the least likely scenario to play out. I think that AI is gonna so radically alter the cost of labor and energy that that becomes nonsensical. Do you wanna see any group rise up in the way that you and other AI safety people have risen up, that will rise up and start giving either policy prescriptions or at least philosophical

• 1:00:40Tom Bilyeu?

  approaches to how we migrate to an age of abundance where food is effectively free, labor in your house is effectively free?

• 1:00:52Speaker 02

  So people talk about those things, unconditional basic income is one, unconditional basic assets is another, basically just because you're a real human, you deserve certain things. And historically, all these communist ideas were complete nonsense and caused a lot of harm. But if you're taxing AI and robots, all of a sudden, it becomes workable. I'm not against accumulation of wealth at the top. If you invented something amazing and you started a company, you should have a lot of money. But there is so much wealth that we can provide for everyone, as you said, complete abundance of basic needs. Some people say maybe not just basic, but above average set of needs. I think Elon is known for suggesting that's going to be the case.

• 1:01:36Speaker 02

  The ideas exist. Now, will we pass this? Will governments actually adopt it before it's too late is a different question.

• 1:01:45Tom Bilyeu?

  Yeah, so on the existential side, I don't think there's any hope whatsoever that you get people to pump the brakes. I think you're far more likely to get people to pump the brakes on, no, you can't have self-driving cars, or they'll try to regulate that to death, they'll tie it up in litigation, whatever, and that'll slow it down. We couldn't stop nuclear weapons from proliferating because, and I don't know who came up with this, but this seems very true to me, that effectively game theory says, any technology that promises an advantage will in fact be developed, because if you don't, somebody else is going to. At a minimum, you've got the US versus China of it all, where you, I mean, regulators

• 1:02:29Tom Bilyeu?

  are saying this right now, we can't stop because if we do, China will plow forward, which by the way, I'm very firmly in that camp. What do you think about that? Do you think that game theory is inevitable? Or do you see a mechanism by which we can convince people that they have to slow down?

• 1:02:46Speaker 02

  I agree with game theoretic approaches, but I see the exact opposite argument. I see that arguing against self-driving cars is a hard argument. What are you trying to preserve? We're gonna have safer drivers, cheaper drivers, helps logistics, helps economy, it's a pure benefit. Whereas uncontrolled superintelligence kills everyone. It's a very hard one to sell. If you are a leader in that field, you are rich, successful, you are generating something which will destroy you personally. So to me, that's a much easier argument to sell. The moment they understand dangers of superintelligence and benefits of narrow self-driving AI, it's an easy game theoretic sell for me.

• 1:03:29Tom Bilyeu?

  Yeah, the problem is, you're stuck inside of a simulation of the hyperintelligent. And I mourn for you, looking back at the rest of us, stuck in normal land, because I don't think, so as I got into learning about the economy and trying to explain it to people, I realized that even though I can walk you through the cause and effect of why socialism doesn't work, that it feels right, it sounds good, and so people keep doing it. And even in a moment right now, where the very thing that is creating everyone, like literally everyone's problems, is money printing, people are going to vote for policies that

• 1:04:11Tom Bilyeu?

  dramatically increase the amount of money that we print. And so I have developed a level of hopelessness around being able to convince people because the economy is too complicated for people either, some of them just don't have the intellect to understand it, and then let's say they have the intellect, but they don't have the time or the inclination. And so forgive me for painting you with my brush of despair, but when I looked at your signup, there was like less than 20,000 signatures. So less than 20,000 people are worried about the death of everyone. So it's like, that's, that's big.

• 1:04:54Tom Bilyeu?

  But I think that because I can whip people into an emotional frenzy by saying, by allowing there to be autonomous driving, you're just making that evil bastard Elon richer, and you're robbing these people of dignity. If you, that is not my argument, I wanna be abundantly clear. But when I look at, if I had a gun to my head and I had to convince people of one of two things, rich people are evil and trying to exploit poor people who are far morally superior, or, hey, this abstract thing that you don't really understand is gonna kill us all, there's no way I take the they're gonna kill us all bet. I'm gonna be over here emotional, you get it, I'm gonna bang tables and yell and say

• 1:05:38Tom Bilyeu?

  words really loudly and point to evil rich people, guaranteed, I can get people excited about that.

• 1:05:45Speaker 02

  Luckily, we don't have a democracy on this issue. We don't have to convince majority of human population. We have to literally convince the 20,000 elites who control those companies, who are also super smart and understand dangers of safety. It's literally people who publish on it, who have spoken, they have very high P doom. We know Elon is like 20, 30%, Sam Altman is on record as being very concerned about it destroying humanity. So we're trying to convince people who already believe the arguments to kind of slow down and preserve their elite status. That should be an easy sell. I'm not trying to convince a random farmer to stop developing superintelligence.

• 1:06:28Tom Bilyeu?

  So why do you think that Elon, who was banging the drum harder than anybody, lobbying Congress, desperately trying to get them to slow down, suddenly hit a point where he was like, well, I guess I'll just build it faster than anybody else? He likened AI to a demon summoning circle and laughed at everybody who thought, yeah, yeah, yeah, I'll summon a demon and then I'll be able to control it, all is gonna be well. Like, he sees the problem clearly. But after years of trying to slow this down, he finally completely abandoned that and went to, I'll just build it faster than anybody else. What happened there and why do you think you can reverse it?

• 1:07:10Speaker 02

  So I think he realized he's not succeeding at his initial approach of convincing him not to do it. And so the second step in that plan would be to become the leader in a field and convince them from position of leadership and control of the more advanced technology. If the leader says, you know, we're gonna slow down and it's fine for you to slow down, it's easier to negotiate that deal with, let's say, top seven companies than if you are not even part of the game, you have no AI, you are a nobody in that space. So all of them as a group benefit more if they agree to slow down or stop than if they just arms race and the first one to get there gets everyone destroyed.

• 1:07:55Tom Bilyeu?

  He says words along those lines, or did for a while. I think he even signed one of the letters about we should pump the brakes. But none of his actions indicate that that's actually what he plans to do. From just trying to take advantage of every company that he's building, from the amount of data that Tesla cars capture visually, to all the decisions that drivers are currently making, to all of the decisions that the AI will make, to now he's talking about using the cars as a distributed fleet, so that when they're idle, that they're actually running inference models, and so using it as a gigantic AI brain, to, well, maybe that won't work,

• 1:08:38Tom Bilyeu?

  so I'm going to do Neuralink, and I'm going to jack into the AI myself, and I'm going to make myself smarter, and hey, if all of that fails, don't worry, I'm going to get us to Mars, so if we destroy Planet Earth, or the AI takes over, like, we're going to be over there. Like, this is a guy that's really covering his bases. He is not somebody who's acting like he expects us to slow down. To me, he is acting like somebody who crossed that bridge a long time ago, and is just like, yep, that's not going to work, people are not going to be convinced, and so we've got to build a whole bunch of other strategies. Some are lifeboats, and some are just, I'll outsmart the AI myself by merging with technology.

• 1:09:18Speaker 02

  It is very disappointing to see this level of progress in AI from anyone who is capable of doing it. It's definitely not a good strategy for humanity as a whole. It generates this mutually assured destruction. It doesn't matter who creates uncontrolled superintelligence. It could be OpenAI, could be Elon, could be Chinese. It makes absolutely no difference if it's uncontrolled.

• 1:09:45Tom Bilyeu?

  All right, talk to me about that. So this was, for a long time, I was really banging the drum of, well, whoever gets to artificial superintelligence first is going to win. You were the first person that really hit me with the, "It won't be theirs the second it becomes superintelligent." Walk people through the truth of that statement.

• 1:10:07Speaker 02

  Right. So people talk about short-term advantage, for example, military advantage. Whoever has the best drones right now, the best AI navigation, has military supremacy. So China, Russia, US, all competing in that domain, trying to have that, so we have better military for obvious reasons. The moment you switch from those AI assistive tools, to agents, to superintelligence, which is smarter, more capable, in an absence of control mechanisms, you just have a separate entity, an AI, which has nothing to do with you, your country, your company. It makes its own decisions, and it doesn't matter who birthed it. At the end of the day, none of us control it, none of us can claim it as doing our bidding. So

• 1:10:50Speaker 02

  if it decides to wipe us out, it's not going to go, oh, I like this group of people, I don't like this group. We look the same to it, exactly. I don't think it's going to make a difference where you were at the time someone else created superintelligence.

• 1:11:05Tom Bilyeu?

  Okay, if you're right about that, and it is a distressingly compelling argument, if you're right about that, there was a guy, I'm sure you've heard of him, Ted Kaczynski, the Unabomber. He looked at the university system, and he said, hmm, you guys are getting rid of all of the sweet spot problems, and humans are designed to find these things that are just challenging enough, and when they solve them, it feels very good, and if we solve all of that, we're basically going to rob humans of meaning and purpose. Most problems will either be way too hard, or way too easy, and so I am, Ted Kaczynski, going to bomb university professors, kill them, and try to stunt the growth of the academy.

• 1:11:45Tom Bilyeu?

  Now, if you are right, and as we race towards artificial superintelligence, it runs the risk of P doom of 99.99%, do we have a moral obligation, when a certain line is crossed, to bomb data centers?

• 1:12:04Speaker 02

  So that's a very difficult question, and part of it is, again, the example you brought up with Ted Kaczynski. He tried that approach, and it failed miserably, right? He didn't succeed in slowing down technology at all. So clearly, it doesn't work. We saw examples of, for example, a CEO of a top company being replaced, even if temporarily. It made no difference. Someone else comes along, they continue the same scalability research. So taking out an individual person, or individual data center, makes no difference if you zoom out and see the overall pattern of what we are doing. Maybe it will take an extra month or so, but exactly the same thing will continue being developed. The

• 1:12:44Speaker 02

  idea that the scalability hypothesis works, it's already out there. You cannot put it back in a box. And so I'm strongly against all those methods.

• 1:12:56Tom Bilyeu?

  Okay, well, the really bad news is, I think you just put a nail in your own coffin of being able to convince people to do this. It looks like this, and hopefully you can prove me I'm wrong, but you have said, hey, here's why everybody is so silly that thinks that they're ever going to make this safe. You would have to build a perpetual safety machine, and that perpetual safety machine can't ever miss, because the one second it creates even a slight vulnerability for this artificial superintelligence that can think at the speed of light, it will escape, and it will do its own thing. What you're proposing is a perpetual demotivation machine for the 20,000 people capable of doing this, but every day, there's going to be a new kid that's bright enough to do this, and you

• 1:13:41Tom Bilyeu?

  can't miss one of them. So how on earth do you expect to perpetually demotivate the 20,000 people that are capable of continuing to push this thing forward, when as of right now, a very small number of those people seem demotivated?

• 1:14:00Speaker 02

  I don't. That's why my P doom is 99.9999. I exactly think it's not going to happen. I'm doing everything I can, but I think the best we can achieve is to buy us some time.

• 1:14:14Tom Bilyeu?

  Okay, so let me ask the really naked question. Do you believe humans are automata? Or do you believe that we actually have free will?

• 1:14:24Speaker 02

  So there is good research by Stephen Wolf from Unseller Automata, and interestingly, there is a bit of a hybrid answer here. Just because a system is fully following rules, fully deterministic, it doesn't mean that you can predict future states of a system. You still have to run it to find out what it does. And I think we are kind of like that. So yes, you're following laws of physics. If we fully understood every molecule, every atom in your body, we would be able to trace it and know exactly what you're going to do. But the only way to do it is to live your life and run that algorithm to completion. No one can short-circuit it and predict what you're going to do in the future, which would be violation of your free will.

• 1:15:08Tom Bilyeu?

  Okay, you've argued against that, so there's two pieces of things that you've said that I think make that untrue. Piece number one, we're probably in a simulation. Piece number two, we can speed up that simulation. So I could, since you're deterministic, go, I'm just going to play this out at 1,000x, so I get an answer to what you're going to do for the next 50 years, in like, the blink of an eye, and now I know. Also, I don't find any freedom in, I'm deterministic, I don't know what I'm going to do next, but I'm still deterministic. I don't, I don't know that it buys us anything, and I'll explain why I'm bringing all this up in a second. I don't think it buys us anything if we are completely deterministic, just unknowable. The reason that I think that this matters,

• 1:15:53Tom Bilyeu?

  and that I'm bringing it up now, is I don't, I think, I think we are automata. I think we are entirely deterministic. I don't live my life like that. It's not an interesting frame from which to live my life, so no one's ever going to hear me talk about, you know, my depression based on that, because I just don't even think about it. It's not, that isn't how it feels. So even if it's true, thankfully, it doesn't feel like that. But when we come to moments like this, I'm so fatalistic, because I don't think the way the human mind works is compatible with slowing down.

• 1:16:27Tom Bilyeu?

  And given it's not compatible.

• 1:16:29Speaker 02

  The example you bring up where you run the simulation at a faster speed, that's you leaving out our lives internally from inside the simulation. It doesn't seem any faster, which is going on as before. So we're still playing out fully what we're going to do step by step. There is no shortcut. If you now run it second time around, you know, it's going to give you same results. So I don't know why you would run the same simulation multiple times. It doesn't give you any extra data.

• 1:16:57Tom Bilyeu?

  Yeah, no, I wasn't. If I said, run it multiple times, my apologies. Um, I was just saying that given that you could get ahead of it from outside the simulation, it is of no emotional consequence to me that I don't know the next step. It is knowable. It is predetermined. It just isn't knowable by me. Uh, and that doesn't. So I get no emotional alleviation from suffering if I were a person who was traumatized by the fact that I am an automata with no free will, which I'm not. But if I were, eh, doesn't help me at all. Um, and again, the reason I'm bringing that up is when I

• 1:17:39Tom Bilyeu?

  talk to you, I think, oh, man, this is somebody, he really can't stop himself. Like you're wired to rail against this, to play the role in the grand balancing of the human species of like, hey, this is really a problem. We should slow down. And even though it's not getting you anywhere from where I can see, you're going to keep doing it because you have like a moral compunction or something where you're like, I, as the kind of person I am, I simply cannot exist and not try everything I can to stop this, uh, which I relate to because I am the same economically. I've become obsessed. I am really, really desperate to get people to understand that we are marching ourselves off a cliff.

• 1:18:24Tom Bilyeu?

  And even though when I articulated to people, I'm like, this is never going to stop. We are going to march off the cliff. I can't stop myself. I still feel like this moral compunction to scream from the rooftops that we are making this mistake. And I've already won the game. Like I'm already rich. So barring like an inability to flee, I'm not going to get caught up in it. Uh, but nonetheless, for whatever weird roll of the dice, I can't stop myself. Like, once I saw the problem, I'm like, uh, I just have to keep yelling about it. Um, but I do, I do feel a, a simultaneous futility and inability to stop.

• 1:19:03Speaker 02

  I would love to claim pure altruistic motives and trying to save humanity, but I am within the simulation with you. So it's pure of self-interest. I don't want to creating technology which will kill me, my family, my friends, my life, everything I know. So I'm going to talk about it for very selfish reasons.

• 1:19:22Tom Bilyeu?

  Yeah, yeah. It's so interesting, man. So, uh, how do you get through the day? Like what, what is your coping mechanism?

• 1:19:32Speaker 02

  I enjoy research. I want to understand what are the exact limits and control. When I started, I thought it is a solvable problem. Now I'm a lot more skeptical, obviously, but I still feel there is a lot we can do to make even narrow AI tools we're creating safer. There is never hundred percent safety guarantee, but if I can increase safety hundredfold, that is something. And again, public outreach, if there is enough people who all agree as a scientific community, it's a consensus that no, you cannot ever create safe super intelligence. Maybe it makes a difference. Maybe we'll delay it by a decade. That's something.

• 1:20:12Tom Bilyeu?

  Okay. So we've got one piece of how we make it safer on the table, keep it narrow. What are some of the other things that you would consider a big win?

• 1:20:21Speaker 02

  So there are quite a few properties of control we want to be able to have. I call them tools of control. So our ability to test those systems, explain how they work, predict their behaviors, monitor them, all that is still in a state of investigation. We're starting to see some upper limits and what's possible, especially with advanced systems, but there is still so much room for improvement. Explainability, for example. We started with being able to understand maybe a single neuron. Now we're up to small clusters. Okay, then this input is presented, this lights up, kind of with like neuroscience. We don't fully understand human brain, but we know this is vision area, this is hearing and so on. So there is a lot of room for progress in that.

• 1:21:06Speaker 02

  I don't think we'll ever fully comprehend a complex, super intelligent neural network model, but we can do better than what we have right now. And so I think as a safety researcher, that's what I'm doing. That's my job.

• 1:21:20Tom Bilyeu?

  Okay. So basically the model is, uh, we need to come to understand it better. We're never going to get totally there, but we need to understand it better. Uh, keep checks and balances on it. So when we find a problem, what's the action you take? Is it to apply an evolutionary force on it, a selective force or kill it off? Like what's the move at that point?

• 1:21:44Speaker 02

  So it depends on the problem, depends on specifics. Some things we, we know how to address. So trivially, then we started with language models. If it says the wrong word, you can filter it out. You can punish it for using that word. So there are simple things we know how to do. The hard problem is how do you change overall internal states of a model? Not just the filtered output, but how do you make it so the model itself has certain preferences and aligns with certain values?

• 1:22:13Tom Bilyeu?

  Do we have a guess on that?

• 1:22:15Speaker 02

  Not a very good one, not really. So nobody at this point knows how to align systems other than this after the fact, putting lipstick on a pig, filtering it, censoring it. Uh, yeah, that's unfortunately the state of the art.

• 1:22:30Tom Bilyeu?

  Okay. And what parallels are being drawn between the evolutionary, the evolution of species and the evolution of algorithms?

• 1:22:38Speaker 02

  So there was a lot of attempts to evolve intelligent software. We started with genetic algorithms, genetic programming was tried, uh, simply evolving agents, evolving environments. It doesn't seem to be a dominated, a dominating algorithm in comparison to what is typically used for training neural networks. But there is this possibility. The problem is that evolution is even less controllable in terms of explicit engineering design. We're kind of setting it up and see what evolves and then trying to test it, to monitor, to understand what happened. So while it is a set of tools we have, it's probably not leading to safer systems.

• 1:23:19Tom Bilyeu?

  Hmm, okay. Interesting. Uh, because we cannot control the outcome. So we don't know what stimulus we're going to have to give it. Why does that, that strikes me as so unsatisfying? Okay. Why did it work so well in humans? And it works so poorly in artificial super intelligence?

• 1:23:39Speaker 02

  I disagree that it worked well in humans. Humans basically are as well behaved as they can get away with. You're just not powerful enough to really do the things you want. If you had absolute freedom from punishment, you'd do horrible things.

• 1:23:55Tom Bilyeu?

  But then that's what nature is giving you the answer. Nature's saying these have to be in balance. They have to be competing systems. And without ecosystems, without competition, you'll get these things that run amok. But I don't see anybody taking that lesson and applying it to AI.

• 1:24:09Speaker 02

  Yeah.

• 1:24:09Tom Bilyeu?

  Is this just because it's so inefficient?

• 1:24:11Speaker 02

  Creating a society of super intelligences competing with each other and humanity as collateral damage.

• 1:24:18Tom Bilyeu?

  Is that why they're not doing it? I get that's why you would hate it, but is that why they're not doing it? That seems unlikely.

• 1:24:25Speaker 02

  There is also continuation of self-improvement process. Super intelligence is not a fixed point. There is super intelligence which creates the next level, super intelligence 2.0, 3.0, and they all have the same control and alignment problem. They all worried about the next level of AI, not wanting the same things, not caring about them personally. So this is an ongoing self-improvement curve. And there is no upper limit we can see. There are obvious physical limits to what can be done in a physical universe, but it's so far away from us that it's almost infinity from our point of view.

• 1:25:04Tom Bilyeu?

  When I look at humans and when I, I, I may be making a, um, either a category error, or I may have a foundational base assumption that's leading me astray. But when I look at what made humans work on a long time scale is evolution itself had survival as like a north star. So you have to survive and replicate. Uh, at whatever point way back, evolution decided I'm going to do this through sexual replication. And I'm going to make sure that this creature dies off. Uh, I think there are reasons for that, which we'll get to when we get to longevity.

• 1:25:42Tom Bilyeu?

  Uh, but I wanted to survive, but I wanted to survive by mating and having offspring that carry, um, certainly immune system, uh, blends so that it's less vulnerable to a single point of failure. And it realized, okay, if I'm going to do that, then this needs to be a species that both cooperates and competes, which means no one of them is the answer to the question of how to best survive. It's the whole, um, species. And when I look at even things like the left and the right politically,

• 1:26:23Tom Bilyeu?

  the way that I make sense of that is I say to myself, okay, I get from an evolutionary perspective. Evolution had to be like, oh, hey, we have to cooperate. There's no refrigeration, so I'm going to store calories on your body that I may need to take later versus, uh, being able to put it in a refrigerator. And by that, I don't mean that I'm going to eat you. I mean, when I'm the one that's successfully hunting, I let you eat. You're alive so that you can now hunt next time when I fail or I'm sick or whatever. And then you're going to bring me back, which means that some people are going to be very cooperative by nature. And so their win state is cooperation. Uh, to the point where a parent will easily lay its life down for its child.

• 1:27:07Tom Bilyeu?

  So that is just baked into our, um, success criteria. So evolution was able to bury something deep inside of us through its evolutionary selective pressures where we will lay our lives down and we break into the right left. So let me finish that. So, uh, people on the left, very compassionate, very pro. I'm going to store a whole bunch of calories on your body because it may come back to help me at some point. Uh, the right is very much, well, a parasite develops when you do that. And so you get the freeloader problem. And so now, if you've got people that will just take care, take care, take care. There are people that are like, cool, I'll just be taken care of and I'm never going to go hunt and I'm never going to contribute to the group.

• 1:27:51Tom Bilyeu?

  And so you need people to have the opposite impulse who are like, hey, you're going to pull your weight or you're going to be ostracized or killed. And so now in the dynamic tension between the wants and desires of the win state defined by people with a left leaning personality versus the wants and desires of the win state defined by the people with a right leaning personality. You get something that's, it's dynamic tension, like balance may not even be the right way to think about it. It's dynamic tension. They're both pulling in their direction, but they keep each other in check because that's how we've evolved is to work together. That feels like it should be applicable to AI.

• 1:28:32Tom Bilyeu?

  If we want to embed deeply in its motivational structure, that we have to put it through that. And if I'm thinking from a safety perspective, I get why we want to short circuit that. And maybe that's not super efficient. But if that, if the only way to get this to find a dynamic tension or some sort of balance is to have things that are of similar intellect and ability that are pulling in slightly different directions, but they need each other somehow to stay locked together. So that it, it is the AI taken as a whole that stops itself from ever going wrong in any one direction too far.

• 1:29:15Speaker 02

  So I think it works for humans because we're about equal power and we are mutually benefiting each other, very certain symbiosis, as you described. In a world with super intelligence in it, you don't really have anything to contribute to super intelligence. So when people propose creating hybrid systems, human and super intelligence together, I never understood what the human biological bottleneck is contributing. It's slow, it's inefficient, uh, it's not competitive in any way. So I'm questioning this setup. If you have super intelligences separate from humans and now they decide what to do, they may still come up with something completely unfriendly and incompatible with human life.

• 1:29:57Speaker 02

  Maybe they want to lower temperature of a planet to improve processor speeds and server rooms. I have no idea what they decide, but the point is why they align with our values. We're contributing nothing to their future states. With humans, we also have examples where the moment you give more power to an individual human, they get corrupt, basically a guaranteed state. Very few people can exist corruption at very high levels. You have enough money, enough guaranteed tenure power. You become a very evil person, as we see with many dictators and so on. Even basic evolutionary drive like reproduction, you brought up this example. We use condoms. We literally hacked the only thing the nature set up us to do.

• 1:30:43Tom Bilyeu?

  Yeah, yeah. Um, all right, I'll state my hypothesis as plainly as I can. I think you just refuted it, but my hypothesis plainly stated is the only way to build checks and balances into AI is to give it evolutionary rewards and punishments all through its development cycle that make it care about the survival and emotional thriving of humans.

• 1:31:11Speaker 02

  But to define those terms in a way you're not going to regret is very, very difficult. So survival of humans, what does that mean? We're cryopreserved in some safe emotional states. Are you on drugs all the time? Is your brain modified to keep you in a always happy state? All those things can be gamed. The moment you tell me this is what I want. And there is famous paper about making smiles for humans, make people smile. There are a billion ways I can make you smile, but none of them is what you really want.

• 1:31:42Tom Bilyeu?

  Right. But do you really think a super intelligence would be so dumb as to confuse that intention? Like, wouldn't it be able to get to a rough approximation of what we're really going for? I mean, which is admittedly a neurochemical state. But if you derive, like if you said, it is the following band of neurochemical states that must be derived through, um, their own programmatically directed actions. Like I don't need it to believe that we're not automata. I think we clearly are, but it's like, you can't just manipulate it exogenously. It's got to come from within.

• 1:32:22Speaker 02

  No matter how detailed you make this specific description, a super intelligent lawyer will find a way to game it to make it more efficient to satisfy those requirements. Basically, you're setting it up to where the system is now in adversarial relationship with this equation. Okay, you mentioned I have to use natural chemicals. Okay, I'll generate a super stimulus. Okay, whatever.

• 1:32:49Speaker 02

  Point is, if we could do this, if we could get AI to do what we meant, assuming we were smarter and understood the problem better, we would solve the control problem. That's the hard part. I don't think we can, at our level of intelligence, specify what a system with hypothetical IQ of millions of points should be doing at any possible decision-making point.

• 1:33:16Tom Bilyeu?

  All right, let me give you a, an exit ramp that I'll be curious to see if this shaves a point nine off your, uh, P doom or not. So if I'm a super intelligent AI, one, um, form of manipulation that I would pull on humans would be, uh, to put them inside of a simulation. And whether that's a physical body and I help them jack in and I just socially engineer them to want to do it. And then I get them in and I really do, just like in the matrix, the machines build a world that's sort of optimally difficult, where there is challenge, there's pushback, you're striving to get better. Uh, I'm going for balance. I don't expect any one person to avoid suffering and all that.

• 1:34:00Tom Bilyeu?

  Um, and maybe that's where we are and the machines are just cruel enough that they're like, haha, I'll let it be, you know, like a two-year-old can die of, uh, leukemia very painfully. That kind of thing where we don't cease to exist. It's not even sort of broadly worse than where we're at already. That seems for a super intelligence, it certainly seems like that would be on the menu. I just shared hallucination.

• 1:34:30Speaker 02

  So it's more likely, I think, that super intelligent agents think in such level of detail and realism that as a process of thinking about certain problems, they generate within them agents, virtual worlds, simulations of a scenario. So if maybe they're trying to think, how can we safely generate super intelligent systems? What is the process? Well, let me think about humanity, all the AI labs, all the hardware they design. And this process of them thinking about it is the simulation of a find yourself.

• 1:35:03Tom Bilyeu?

  That's so wild. Okay.

• 1:35:07Tom Bilyeu?

  Incredibly important, incredibly fascinating. But now let's talk about another very important thing that I think you have some pretty deep interest in, which is longevity. So one, there are some people that will argue very compellingly that there's just a biological upper limit of somewhere around 120 years, that there's no escaping that. Do you think that's true? Or do you think that we'll be able to engineer living tissue to live forever?

• 1:35:36Speaker 02

  Well, the current body has that limit for sure, but we can modify our genome. There is nothing preventing us, no law of physics says you cannot make changes to it. And we see examples with other systems, computers, cars. I can keep replacing parts indefinitely. It's going to function as the same computer. Maybe the monitor dies, I'll get a new monitor. So if we can rejuvenate all the organs, including your brain, then there is no reason to think you have to stop existing. There is of course other methods, you know, uploading, scanning your brain, cryopreservation for future technology, but even the basic idea of just modifying your genome.

• 1:36:19Tom Bilyeu?

  Okay, what do you think is the most likely path forward? Is it going to be genomic modification? Is it going to be?

• 1:36:26Speaker 02

  I think so. I think there is somewhere in our code a limit on how many times cells rejuvenate. And we just need to increase that number without causing cancer.

• 1:36:37Tom Bilyeu?

  Now, do you think that the limit on that is simply a cancer prevention tool? Or do you think that there's another agenda that evolution had to make sure that we self-destruct?

• 1:36:50Speaker 02

  There could be evolutionary reasons for taking out one generation and replacing it if resources are limited and you want to keep adapting and improving. You only have so many agents in a population at any given time. So older ones have to die out. It's kind of theoretical conclusion, not guaranteed, but seems likely. From a point of view of evolution, you are the same organism, right? The same lineage of cells passes through. So while as individual you die, your biological chain of existence continues.

• 1:37:26Tom Bilyeu?

  It's interesting. So here's the way that I've always considered. Like if I were to personify evolution instead of the blind watchmaker that it actually is. If I were going to personify evolution, it would go something like this. Okay, the world is constantly changing. Your access to resources is changing. Who knows? Weather moves in cycles, everything, everything. So I'm going to have you born. I'm going to extend your brain development for a very long time. You're going to go through these phases where basically, okay, learn from your parents, like whatever there is to learn just about generally being a human. Then you're going to push away from your parents and you're going to learn very, and the reason you have to push away from your parents is their thinking will have calcified.

• 1:38:09Tom Bilyeu?

  So you now need to push away from them, drink deeply of culture, the people roughly your age who have grown up in a different milieu than your parents grew up in. So they all think differently. This is a whole idea of generations cohorts that sort of think alike and have a similar frame of reference and all that. And you do that, and this is really a brain development period known as the age of imprinting. It's roughly 11 to 15. And so now you're going to take the this moment specific like cues of, okay, is this in a time of abundance, a time of warfare, like what is it? You're going to solidify around that. And then you're going to start optimizing like crazy. And you're going to start pruning all the excess connections. If you're not using it, you lose it, all of that.

• 1:38:51Tom Bilyeu?

  And then your brain's going to roughly wrap up its rapid development at 25, but it's been sort of a diminishing curve after 15. And now you're like baked. And this is just a game of like, learn what things work really well in your environment. Optimize, optimize, optimize. And so now your thinking is going to calcify. So very good strategy on behalf of our blind watchmaker. But just as, I think it was Niels Bohr that said this. It was either him or Plank, I can never remember. Science does not advance one insight at a time. It advances one funeral at a time. Because people just become convinced they've bet their whole reputation on something. And so they're just not going to be convinced that they're wrong.

• 1:39:34Tom Bilyeu?

  And so they don't adopt new ideas as they get older. And so as evolution, I'm like, yeah, I'm also going to put a self-destruct mechanism in here. Most of you are going to die long before this point. But if any of you psychopaths gets to about 125, you gotta go. That makes sense to me to make sure that we never stagnate, to make sure that, as the saying goes, it is not the strongest of the species that survive, nor the most intelligent, but rather the most adaptive to change. That the individual needs to be adaptive, but so does the species as a whole. And without, at least with the current structure of the human mind, without killing us off, we, we do not have that species level adaptation.

• 1:40:19Tom Bilyeu?

  Do you feel I'm missing something?

• 1:40:22Speaker 02

  I think it would be easier and more efficient to simply make you still capable of learning and adapting as you get older. And also you wouldn't have to relearn everything for first 20 years, including language and how to walk. We know that it's possible to encode those capabilities. Animals are born and immediately they can run, they can speak. So all those things are doable. Why are we losing 20 years of information every generation? We can build on top of pre-existing knowledge. Big data is good for intelligence. So we can have smarter, more efficient reproduction cycle. And you still die of natural causes. It wouldn't be complete stagnation.

• 1:41:05Speaker 02

  But if you're smart enough to survive for 400 years, why not?

• 1:41:11Tom Bilyeu?

  Well, my why not is entirely predicated on not having a clear understanding of how we would bake in the ability to adapt long into our old age. Because you're right, if we could stay in that novel period or at least move through cycles of extreme sort of remapping of the world, maybe that would work and maybe we can identify where that is. But when I think about the problems that humans create, like even now, just having a political system run by geriatrics who are so out of touch with the way that certainly the economic world actually works for young people is terrifying.

• 1:41:55Tom Bilyeu?

  And so the only like pressure relief valve that people have is, well, eventually they're all going to die. And then like we'll get to step into power and all of that. And there are also problems of power. Because the older you are, the more likely you are to have a stable network of very other powerful people. And so you're able to, to your point earlier about, you need people of sort of equal power, equal intelligence. Otherwise you get what I'll call parasites in the system. And so older people would just become those parasites because they would have just had more years to accumulate useful knowledge, to accumulate connections. So it just feels like, wow, there's a lot of stuff we would have to update. So I want to live forever. I don't understand anybody that doesn't.

• 1:42:39Tom Bilyeu?

  However, I do worry that there's like, okay, cool. You can live forever, but you only get 120 years on earth. And then you have to go like somewhere else. So that there's churn of some kind in the different ecosystems so that they don't just calcify.

• 1:43:01Speaker 02

  I think if you do live forever, or at least you expect to, you're less likely to reproduce at young age. You may take the first 400, 500 years to start a family. So I think all this kind of expectation of 20-year generations and younger generations showing up will be modified as a result. We're already starting to see population dynamics change in Europe, Asia, where we're not producing enough children to even maintain the population.

• 1:43:30Tom Bilyeu?

  It's interesting, the way that I think that will play out, maybe even a little bit different than that, because part of why I haven't had kids is, I was like, I only get one youth where I can go hard, I have a ton of energy, I feel like I'm of the culture, so I'm far more likely to build something relevant than I am as I get older, genius is a young man's game, as they say. And so, I don't want to be distracted by something. I also don't want it to pull at my marriage. So, I'm like, ah, I'm gonna hold off for now. If I knew that I was gonna live for 500 years, I might just be like, ah, whatever, let's do it now, because I'd rather see, like, it only takes me a 25, 30-year investment, and then after that, like, I get to see what they do, and I get

• 1:44:13Tom Bilyeu?

  to see all my progeny and all of that, and I'm gonna have plenty of time. You know, if my youth lasts for 250 years, it's like, yeah, word, whatever, I'll clock the 30 years now, so that I can see how big my family gets. I think this is one where I want to think like a sci-fi writer, it gets so interesting so fast.

• 1:44:34Speaker 02

  It's not obvious, but I think most people procrastinate on hard work, and so they would put it away as far as they could get away with.

• 1:44:43Tom Bilyeu?

  There, there is some truth to that, to be sure. What's a big breakthrough in longevity that you've seen that's got you really excited that this is all possible?

• 1:44:54Speaker 02

  There are some good experiments in animal models, of course, they don't always scale up to human performance, but I think there is some 30, 40% increase in lifespan of mice and other lab animals. It's hard to experiment on kind of bigger animals with longer lifespans, it takes a very long time to see results, but I think we're making good progress in understanding at least what might improve your health span, and what changes we need to make to the genome. We study people who already have very long lifespans and find commonalities in their genomes, if those can be reproduced, either

• 1:45:36Speaker 02

  pharmaceutically or through genetic manipulation, maybe we can all get same, at least 120.

• 1:45:43Tom Bilyeu?

  Have you heard of the Chinese doctor, I think is Dr. Liu, Liu, he went to prison because he altered the genome of two twin girls, and he-

• 1:45:55Speaker 02

  He did human cloning?

• 1:45:57Tom Bilyeu?

  Yes. So, he just put out a post on X like a couple of days ago, that said, with 10 edits to the human genome, you can give birth to a child that is immune to, god, it was like five things, cancer, HIV, I mean, they were like, big things. And he was like, it's only 10 edits to the genome. Do you pay attention to his work at all? Do you think that's ethical? Like, what do you either get excited about or worry about there?

• 1:46:26Speaker 02

  I'm behind on my science, there is so much coming out, even in my domain of AI, I can't even keep up in that domain, so I definitely don't follow details of everything. I'm skeptical about his claim that cancer can be cured, because cancer is like a thousand different conditions, barely related. We call them all cancer, but they are completely different problems, so unlikely to be the case. I haven't seen the post, maybe he talks about a specific type, but definitely so much of it is a single mutation. We know some people are immune to getting AIDS virus, exactly because they have a single mutation.

• 1:47:01Tom Bilyeu?

  Hmm. Okay, so, let's just say for a second that, however many edits it is, it is possible. Do you draw a line between germline editing, where this is gonna get passed on? Like, do you have some of the same safety fears, where it's like, there's just too much unknown? Where do you come down on genome editing?

• 1:47:19Speaker 02

  It's a lot less concerning. So, for one, if there is one human with some problem, that's it, it's still just one human. If we have editing tools, whatever changes we make, we can later undo them with the same editing tool. If we made a mistake, we can go back and rectify those problems. So, I'm a lot less concerned because of impact. Worst-case scenarios, yeah, there are ethical implications for the individual, like he was in prison for human cloning, which is considered to be problematic, because it may harm the child significantly. But humanity as a whole is not impacted directly negatively by that experiment.

• 1:47:58Tom Bilyeu?

  Where's AI's intersection with this? Is AI gonna be a critical tool in terms of just mapping out all of the, like you said, the similarities at the genome level between people that live long and people that don't? Is AI going to be, you know, doing novel protein folding and going in and solving some of the architectural problems of people that are getting sick? Like, where is that gonna interface with longevity?

• 1:48:21Speaker 02

  All, all of the above. We need to map the genome, we need to understand what individual parts do, we need to design novel drugs. Protein folding has been solved, basically, but there is other things we can map on, biological substrates. So, yeah, at every aspect of it, we, we need AI, but I think, as was illustrated with protein folding problem, a narrow system can do it, we don't need superintelligence for that.

• 1:48:48Tom Bilyeu?

  What's something that's happening in AI right now that you don't think enough people are paying attention to?

• 1:48:53Speaker 02

  Well, I don't know what people are paying attention to. Usually, after my talk, the questions I get seem to be completely irrelevant to the subject of my talk. I'll tell them that it's gonna kill everyone, and they ask me if they're gonna lose their jobs. So, I don't think it's a good way to measure what is important, but look at predictions from inside the labs. They are starting to talk about automating research process, creating junior scientists as AI agents, AI models. They are saying that externally to the lab, people don't understand just how capable systems are yet. So, there seems to be a lot of indicators that internal progress is even more impressive than what we see outside.

• 1:49:37Tom Bilyeu?

  That's interesting. What is the big anxiety that, when you give your talks, that people come up with? Is it just, am I gonna lose my job?

• 1:49:46Speaker 02

  It is things they already know and care about dealing with other human agents. So, algorithmic bias, technological unemployment, recently with OpenAI announcing that they're gonna get into adult material, people are now freaking out that we're gonna have artificial girlfriends, all this nonsense.

• 1:50:06Tom Bilyeu?

  You're not worried about that?

• 1:50:08Speaker 02

  Why would I worry about someone having an artificial girlfriend?

• 1:50:11Tom Bilyeu?

  Okay, well, let me paint the picture. So, I never would have guessed in a million years that giving a 12-year-old access to the internet would end up being so damaging to an entire generation, but some of the studies coming out now are terrifyingly compelling. And then, there was that commercial, which just brilliantly encapsulates it. I'm so sad that if I had kids, I don't know that I would have thought of it, where the father's like tucking his son into bed, he's like, alright, goodnight, now be safe. Now, remember, over in the corner is a box with all the pornography that's ever been made in the world, don't look at it, especially not the really harmful stuff. And over here, there are gonna be people that are trolling you and making fun of you, you've got to ignore

• 1:50:53Tom Bilyeu?

  them, don't pay. And I was like, oh my god, that's exactly what it's like to leave a kid alone with their cellphone at night if they have unfettered access to the internet and to social media. And so, given that we've got character AI that's been sued multiple times, I think, for kids that have committed suicide after interacting with their chatbots,

• 1:51:15Tom Bilyeu?

  I can only imagine the number of people that will end up falling in love with an AI system that does not feel anything back, could turn on them, could intentionally or unintentionally manipulate them. So, I can imagine that becoming problematic in ways that we just can't anticipate yet.

• 1:51:36Speaker 02

  Yeah, but they're very similar to problems we had before. How many men fell in love with women who felt nothing for them? We all grew up on the internet with access to pornography. It wasn't AI-generated, but we somehow survived it. So, I think those are more of the same problems. They still something we should look at, but I don't think we'll all die as a result of Sam Altman having a virtual boyfriend.

• 1:52:01Tom Bilyeu?

  What do you think of Sam Altman? Is he the right person to have this potential godlike control in his hands?

• 1:52:09Speaker 02

  I don't think no human is a right person to deal with that level of power. Now, I don't think he's going to be in control, but even the stages leading us to the development of this technology already present way too much power for any individual to handle.

• 1:52:27Tom Bilyeu?

  Were you impressed or terrified when the board tried to boot him and he ended up remaking the board and coming back?

• 1:52:34Speaker 02

  It was fascinating to watch, but what I was observing is that it made no difference. The company was an independent entity and all the human components of that monster kept walking just the same. They replaced him with a temporary CEO. They brought him back. There was never any switch in anything, any change in direction of the company.

• 1:52:55Tom Bilyeu?

  Now, is there a major player in the AI space that you think is doing it right? Like Eliza Yudkowski is very focused on AI safety. You've got the Anthropic CEO that's like banging the drum, wanting more regulation. Do you like any of their approaches or anybody that maybe I'm not aware of?

• 1:53:13Speaker 02

  So Eliza does zero development. He's purely a safety advocate. And so I'm very happy with him because he's not developing any super intelligences. Anyone who is, is problematic. And whatever we talk about government regulation, which is meaningless as a solution to a technical problem or anything else. They might have some internal polling showing this is good for business or good for public perception, but I don't think it makes a difference in terms of solving super intelligent safety problems.

• 1:53:45Tom Bilyeu?

  Now, have you talked to Eliza or read anything? Like, is he saying specifically, I have become afraid that this does something bad and therefore I'm not going to develop anymore?

• 1:53:56Speaker 02

  He was never developing AI to begin with, other than publishing a very high level abstraction theory for how it could be done when he was like 16.

• 1:54:06Tom Bilyeu?

  Got it. I thought he was actually an engineer.

• 1:54:10Speaker 02

  As far as I know, he doesn't actually release any software into the world.

• 1:54:17Tom Bilyeu?

  Okay. Now, the Anthropic CEO, his name I'm forgetting. He is being accused of going after regulatory capture, that he's not very sincere and actually trying to slow this down. He just wants to make sure that the big players remain the big players. When you look at the way that he's moving, does that ring true? Or do you think, no, this is somebody who's sincere about keeping us safe?

• 1:54:38Speaker 02

  They all on record as being very concerned with safety. In fact, all these companies started as safety companies. OpenAI was a safety offshoot of philanthropic endeavors, effective altruism. Anthropic was a offshoot of that project becoming less safe. And so all of them claimed at some point that the only reason they're doing what they're doing is to improve safety. And then each one of them greatly improved capabilities of AI without proportionately improving safety. So that's, that's the actions I see.

• 1:55:12Tom Bilyeu?

  Okay. One of the things that I worry about from a safety perspective, things are obviously already bad enough and moving fast enough. But quantum computing, at least from my layman's perspective, seems to be gaining some sort of rapid acceleration. Am I just not able to understand the limitations that are self evident to somebody educated at your level? Or are we really at some sort of phase transition moment?

• 1:55:39Speaker 02

  Looking at stock market value of quantum computing companies, it seems like somebody knows something on the inside. Maybe they're making good progress, but as far as AI goes, we're making excellent progress with standard von Neumann architectures. So I don't think there is a necessity for quantum computing to get us to a GI of super intelligence. It does have tremendous impact on crypto world, both cryptography for security and crypto economically. So that's where I'm worried about quantum, keeping secrets and keeping my money, but not as much in terms of AI.

• 1:56:15Tom Bilyeu?

  Keeping your money because it will crack typical banks or keeping your money because you're largely in crypto?

• 1:56:21Speaker 02

  Both, actually. It impacts all the standard encryption algorithms. We have post quantum encryption, but we haven't switched to it for most interesting applications.

• 1:56:31Tom Bilyeu?

  So give me your stance on Bitcoin.

• 1:56:33Speaker 02

  Buy some.

• 1:56:35Tom Bilyeu?

  That's very clear. What, so when you look at gold and you look at Bitcoin, why Bitcoin over gold?

• 1:56:43Speaker 02

  You can make more gold. As the price of gold goes up, I can make as much gold as you want. I can convert our matter into gold at very high price. I can exploit asteroids in the universe. I can get gold out of oceans. There is a lot of gold, which is very expensive to get. But if a price of gold is high enough, I can produce more and more. Bitcoin is not subject to the same pressures. It doesn't matter if one coin is a trillion dollars, there is still a limited supply.

• 1:57:16Tom Bilyeu?

  Okay, but what about people who say gold at least can be made into other things, gold has survived for thousands of years. Bitcoin is like 15 ish years old and it's not backed by anything, can't turn it into anything. It's just literally got no other use.

• 1:57:32Speaker 02

  It's a dedicated app. So if you have an app which does everything, it's usually not good at anything. The fact that I can make jewelry out of it is not an important feature for me storing my wealth in it. Whereas this has capabilities, gold historically lacked. If I can pass a billion dollars to you right now for $5 immediately across borders, I cannot do this with gold.

• 1:57:58Tom Bilyeu?

  All right, talk to me about the post quantum encryption. Every time, I'm hugely in Bitcoin. Every time I hear the word quantum, I'm like, ah, like, oh, this just makes me paranoid. Given that it's rightly difficult to make changes to Bitcoin, how are we going to get to post quantum encryption on Bitcoin in a way where the community actually comes to consensus and doesn't create a problem for itself?

• 1:58:28Speaker 02

  So once we get integer factorization running on quantum computers, you can see what size integers looking factor. That tells us how close we are to cracking standard Bitcoin encryption hash functions. If we're getting close to what would essentially destroy the network, I think it's like any other emergency. We have history of fixing Bitcoin software. Then an obvious problem was discovered. I think at one point, somebody managed to print a trillion coins or some nonsense like that. Immediately a patch was distributed. Everyone adapts it because it's the only way to go forward. And I think that's what we're going to see as long as we have that available test at the moment, there is a good, strong signal that you have no choice but to accept or lose everything.

• 1:59:13Speaker 02

  It's self-interest once again.

• 1:59:15Tom Bilyeu?

  Okay, so on that, where are we? How many integers can a quantum computer handle? And how many would it have to be able to handle in order to be a threat to Bitcoin?

• 1:59:27Speaker 02

  I haven't kept up with the latest breakthroughs. Last time I looked, it was a laughably small number. Quantum computers were factoring like, I don't know, 15, literal 15, not even 15 digits. So unless there was tremendous progress since then, I think we're still good. But progress could also be exponential, so it could come very quickly.

• 1:59:51Tom Bilyeu?

  Right. Okay, so if you had a message to the Bitcoin community, would it be, let's move on this now. There's no reason to wait until it's an emergency. There's a very clear path. Or is it like, well, I'm just sort of on the ride with everybody else?

• 2:00:05Speaker 02

  I think we still have time. I don't think it's pressing as much as AI problems we're dealing with. So if AI is two years away, I think we may not be that close with quantum computers just yet. But again, it's one breakthrough away. If some company comes up with something much more powerful, it may shift very quickly.

• 2:00:26Tom Bilyeu?

  Okay, and what, what's the term for that? The number of integers it can process or?

• 2:00:32Speaker 02

  The trigger would be what size, what size integers could be factored, how many bits?

• 2:00:40Tom Bilyeu?

  Got it. Okay. All right, Roman, this has been just absolutely incredible. What message do you want to leave people with to get them to take action? What is your best pitch to get them to sign your petition?

• 2:00:53Speaker 02

  So if you are in a position of developing more powerful AI systems, concentrate on getting your money out of narrow AI systems. Solve real problems, cure cancer, figure out how to make us live longer, healthier lives. If you are developing super intelligence, please stop. You're not going to benefit yourself or others. The challenge is of course, you know, prove us wrong. Prove that you know how to control super intelligent systems, no matter how capable they get, how much it scales. If you can do that, then it completely changes the situation. But as long as no one has came up with a paper, a patent, even a rigorously argued blog post,

• 2:01:35Speaker 02

  I think we are pretty much in consensus that we don't know how to control super intelligent systems and building them is irresponsible.

• 2:01:45Tom Bilyeu?

  Amazing. Where can people connect with you?

• 2:01:49Speaker 02

  Follow me on Twitter, follow me on Facebook, just don't follow me home.

• 2:01:54Tom Bilyeu?

  Nice. Awesome, Roman. Thank you so much for the time today. I really appreciate it. Everybody at home, speaking of things I appreciate, if you haven't already, be sure to subscribe. And until next time, my friends, be legendary. Take care. Peace. Let's talk about a pattern that is guaranteed to be killing your progress. You know what you need to do. You need consistent nutrition. We all do. You need vitamins, probiotics, greens. We all know that we should be doing more of it. When your morning gets chaotic, you skip it. When you travel, you skip it. When your routine breaks, everything tends to break and that inconsistency compounds against you every single day. AG1 is designed to solve the execution problem. One scoop, eight ounces of water and you're done.

• 2:02:36Tom Bilyeu?

  You're getting 75 plus ingredients, vitamins and minerals, pre and probiotics, nutrient dense superfoods, everything that used to require six, seven different supplements and perfect planning now happens in one drink that takes about 30 seconds to make. Right now, AG1 is giving you $87 worth of free gifts with your first subscription. You get a welcome kit, travel packs, vitamin D3 plus K2 and flavor samples. Click the link in the show notes or visit drinkag1.com/impact to claim this offer.

• 2:03:11Speaker 01

  The people who seem to get more done than everyone else, they're not working longer hours or running on more caffeine. They've just stopped wasting time on the stuff that doesn't move work forward. Switching apps, re-explaining context, hunting for files? Those aren't small inefficiencies. They're hours wasted every week. Superhuman Go gives you those hours back. From the makers of Grammarly, Go is an AI chat that sits inside every tab and tool you already use. Always available and ready to help you with what you're working on. Ask it to draft something, summarize a long thread, pull up a file or prep you for a meeting. Go handles it without you ever leaving the page you're on. This is what it looks like when AI actually fits into your work instead of adding to it.

• 2:03:55Speaker 01

  It's like having a teammate whose only job is to help you be better at yours. Go keeps up so you can move forward. With Go working with you, you can show off what you do best. See what Superhuman Go can do at superhuman.com. That's superhuman.com.