159 - We’re All Gonna Die with Eliezer Yudkowsky

Eliezer Yudkowsky is an author, founder, and leading thinker in the AI space. ------ ✨ DEBRIEF | Unpacking the episode: https://shows.banklesshq.com/p/debrief-eliezer ------ ✨ COLLECTIBLES | Collect this episode: https://collectibles.bankless.com/mint ------ We wanted to do an episode on AI… and we went deep down the rabbit hole. As we went down, we discussed ChatGPT and the new generation of AI, digital superintelligence, the end of humanity, and if there’s anything we can do to survive. This conversation with Eliezer Yudkowsky sent us into an existential crisis, with the primary claim that we are on the cusp of developing AI that will destroy humanity. Be warned before diving into this episode, dear listener. Once you dive in, there’s no going back. ------ 📣 MetaMask Learn | Learn Web3 with the Leading Web3 Wallet https://bankless.cc/ ------ 🚀 JOIN BANKLESS PREMIUM: https://newsletter.banklesshq.com/subscribe ------ BANKLESS SPONSOR TOOLS: 🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://bankless.cc/kraken 🦄UNISWAP | ON-CHAIN MARKETPLACE https://bankless.cc/uniswap ⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum 👻 PHANTOM | #1 SOLANA WALLET https://bankless.cc/phantom-waitlist ------ Topics Covered 0:00 Intro 10:00 ChatGPT 16:30 AGI 21:00 More Efficient than You 24:45 Modeling Intelligence 32:50 AI Alignment 36:55 Benevolent AI 46:00 AI Goals 49:10 Consensus 55:45 God Mode and Aliens 1:03:15 Good Outcomes 1:08:00 Ryan’s Childhood Questions 1:18:00 Orders of Magnitude 1:23:15 Trying to Resist 1:30:45 Miri and Education 1:34:00 How Long Do We Have? 1:38:15 Bearish Hope 1:43:50 The End Goal ------ Resources: Eliezer Yudkowsky https://twitter.com/ESYudkowsky MIRI https://intelligence.org/ Reply to Francois Chollet https://intelligence.org/2017/12/06/chollet/ Grabby Aliens https://grabbyaliens.com/ ----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I think that we are hearing the last winds start to blow, the fabric of reality start to fray. This thing alone cannot end the world, but I think that probably some of the vast quantities of money being blindly and helplessly piled into here are going to end up actually accomplishing something.

• 0:22Ryan Sean Adams?

  Welcome to Bankless, where we explore the frontier of internet money and internet finance. This is how to get started, how to get better, how to front run the opportunity. This is Ryan Sean Adams. I'm here with David Hoffman and we're here to help you become more bankless. Okay, guys, we wanted to do an episode on AI at Bankless, but I feel like David-

• 0:42David Hoffman?

  Not what we asked for.

• 0:44Ryan Sean Adams?

  We accidentally waded into the deep end of the pool here. And I think as we get, before we get into this episode, it probably warrants a few comments. I'm going to say a few things I'd like to hear from you too. But one thing I want to tell the listeners, don't listen to this episode if you're not ready for an existential crisis. Okay, like I'm kind of serious about this. I'm leaving this episode shaken and I don't say that lightly. In fact, David, I think you and I will have some things to discuss in the debrief as far as how this impacted you. But this was an impactful one. It sort of hit me during the recording and I didn't know fully how to react. I honestly am coming out of this episode wanting to refute some of the claims made in this episode by our guest,

• 1:29Ryan Sean Adams?

  Eliezer Yudkowsky, who makes the claim that humanity is on the cusp of developing an AI that's going to destroy us and that there's really not much we can do to stop it.

• 1:40David Hoffman?

  There's no way around it. Yeah.

• 1:42Ryan Sean Adams?

  I have a lot of respect for this guest. Let me say that. So it's not as if I have some sort of big brain technical disagreement here. In fact, I don't even know enough to fully disagree with anything he's saying. But the conclusion is so dire and so existentially heavy that I'm worried about it impacting you, listener, if we don't give you this warning going in. I also feel like David, as interviewers- Maybe we could have done a better job. I'll say this on behalf of myself. Sometimes I peppered him with a lot of questions. In one fell swoop, and he was probably only ready to synthesize one at a time. I also feel like we got caught flat footed at times.

• 2:26Ryan Sean Adams?

  I wasn't expecting his answers to be so frank and so dire, David. Like, it was just a rift of hope. And I appreciated very much the honesty, as we always do on Bankless. But I appreciated it almost in the way that a patient might appreciate the honesty of their doctor telling them that their illness is terminal. Like, it's still really heavy news, isn't it?

• 2:51Ryan Sean Adams?

  So that is the context going into this episode. I will say one thing. In good news, for our failings as interviewers in this episode, they might be remedied because at the end of this episode, after we finished with hit the record button to stop recording, Eliezer said he'd be willing to provide additional Q and A episode with the Bankless community. So if you guys have questions, and if there's sufficient interest for Eliezer to answer, tweet us to express that interest, hit us in Discord, get those messages over to us, and let us know if you have some follow up questions. He said, if there's enough interest in the community, in the crypto community, I'll say he'd be willing to come on and do another episode with follow up Q and A.

• 3:35Ryan Sean Adams?

  Maybe even a Vitalik and Eliezer episode is in store. That's a possibility that we threw to him. We've not talked to Vitalik about that too, but I just feel a little overwhelmed by the subject matter here. And that is the basis, the preamble through which we are introducing this episode. David, there's a few benefits and takeaways I want to get into. But before I do, can you comment or reflect on that preamble? What are your thoughts going to this one?

• 4:03David Hoffman?

  Yeah, we approached the end of our agenda for every Bankless podcast. There's an equivalent agenda that runs alongside of it. But it, once we got to this crux of this conversation, it was not possible to proceed in that agenda because what was the point?

• 4:20Ryan Sean Adams?

  Nothing else mattered.

• 4:21David Hoffman?

  And nothing else really matters, which is also just kind of relates to the subject matter at hand. And so as we proceed, you'll see us kind of circle back to the same inevitable conclusion over and over and over again, which ultimately is kind of the punchline of the content. And so I'm of a specific disposition where stuff like this, I kind of am like, oh, whatever. Okay, just go about my life. Other people are of different dispositions and take these things more heavily. So Ryan's warning at the beginning is if you are a type of person to take existential crises directly to the face, perhaps consider doing something else instead of listening to this episode.

• 5:03Ryan Sean Adams?

  I, I think that is good counsel. So a few things, if you're looking for an outline of the agenda, we start by talking about ChatGPT. Is this a new era of artificial intelligence? Got to begin the conversation there. Number two, we talk about what an artificial superintelligence might look like. How smart exactly is it? What types of things could it do that humans cannot do? Number three, we talk about why an AI superintelligence will almost certainly spell the end of humanity and why it'll be really hard, if not impossible, according to our guest, to stop this from happening. And number four, we talk about if there is absolutely anything we can do about all of this.

• 5:46Ryan Sean Adams?

  We are heading, careening maybe towards the abyss. Can we divert direction and not go off the cliff? That, that is the question we ask Elise or with. David, I think you and I have a lot to talk about during the debrief. All right, guys, the debrief is an episode that we record right after the episode. It's available for all Bankless citizens. We call this the Bankless Premium Feed. You can access that now to get our raw and unfiltered. Thoughts on the episode and I think it's going to be pretty raw this time around, David.

• 6:18David Hoffman?

  I didn't, I didn't expect this to hit you so hard.

• 6:20Ryan Sean Adams?

  Oh, I'm dealing with it right now.

• 6:22David Hoffman?

  Really?

• 6:22Ryan Sean Adams?

  And this is probably, you know, it's not too long after the episode. So, uh, yeah, I don't know how this, how I'm going to feel tomorrow, but, um, definitely want to talk to you about this and maybe have you.

• 6:32David Hoffman?

  I'll put, I'll put my psych hat on, yeah.

• 6:34Ryan Sean Adams?

  Please, I'm going to need some help. Uh, guys, we're going to get right to the episode with Eliezer, but before we do, we want to thank the sponsors that made this episode possible, including Kraken, our favorite recommended exchange for 2023.

• 6:47David Hoffman?

  Kraken has been a leader in the crypto industry for the last 12 years. Dedicated to accelerating the global adoption of crypto. Kraken puts an emphasis on security, transparency and client support, which is why over nine million clients have come to love Kraken's products. Whether you're a beginner or a pro, the Kraken UX is simple, intuitive and frictionless, making the Kraken app a great place for all to get involved and learn about crypto. For those with experience, the redesigned Kraken Pro app and web experience is completely customizable to your trading needs, integrating key trading features into one seamless interface. Kraken has a 24/7 365 client support team that is globally recognized. Kraken support is available wherever, whenever you need them, by phone, chat or email.

• 7:29David Hoffman?

  And for all of you NFTers out there, the brand new Kraken NFT beta platform gives you the best NFT trading experience possible. Rarity rankings, no gas fees and the ability to buy an NFT straight with cash. Does your crypto exchange prioritize its customers the way that Kraken does? And if not, sign up with Kraken at kraken.com slash bankless. The Phantom Wallet is coming to Ethereum. The number one wallet on Solana is bringing its millions of users and beloved UX to Ethereum and Polygon. If you haven't used Phantom before, you've been missing out. Phantom was one of the first wallets to pioneer Solana staking inside the wallet and will be offering similar staking features for Ethereum and Polygon. But that's just staking. Phantom is also the best home for your NFTs. Phantom has a complete set of features to optimize your NFT experience.

• 8:13David Hoffman?

  Pin your favorites, hide the uglies, remove the spam and also manage your NFT sale listings from inside the wallet. Phantom is of course a multi-chain wallet, but it makes change. Management easy, displaying your transactions in a human readable format with automatic warnings for malicious transactions or phishing websites. Phantom has already saved over 20,000 users from getting scammed or hacked. So get on the Phantom waitlist and be one of the first to access the multi-chain beta. There's a link in the show notes, or you can go to phantom.app/ waitlist to get access in late February. Hey, Bankless Nation. If you're listening to this, it's because you're on the free Bankless RSS feed. Did you know that there's an ad-free version of Bankless that comes with the Bankless Premium subscription? No ads, just straight to the content.

• 8:55David Hoffman?

  But that's just one of many things that a premium subscription gets you. There's also the Token Report, a monthly bullish, bearish, neutral report on the hottest tokens of the month. And the regular updates from the Token Report go into the Token Bible, your first-stop shop for every token worth investigating in crypto. Bankless Premium also gets you a 30% discount to the Permissionless Conference, which means it basically just pays for itself. There's also the Airdrop Guide to make sure you don't miss a drop in 2023. But really, the best part about Bankless Premium is hanging out with me, Ryan, and the rest of the Bankless team in the Inner Circle Discord, only for premium members. Want the alpha? Check out Ben the Analyst's DGEN pit, where you can ask him questions about the Token Report. Got a question? I've got my own Q&A room for any questions that you might have.

• 9:40David Hoffman?

  At Bankless, we have huge things planned for 2023, including a new website with login with your Ethereum address capabilities, and we're super excited to ship what we are calling Bankless 2.0 soon TM. So if you want extra help exploring the frontier, subscribe to Bankless Premium. It's under 50 cents a day and provides a wealth of knowledge and support on your journey west. I'll see you in the Discord.

• 10:01Ryan Sean Adams?

  Bankless Nation, we are super excited to introduce you to our next guest. Eliezer Udikowski is a decision theorist. He's an AI researcher. He's the cedar of the Less Wrong community blog, a fantastic blog, by the way. There's so many other things that he's also done. I can't, I can't fit this in the, in the short bio that we have to introduce you to Eliezer. But most relevant probably to this conversation is he's working at the Machine Intelligence Research Institute to ensure that when we do make general artificial intelligence, it doesn't come kill us all. Or at least it doesn't come ban cryptocurrency, because that would be a poor outcome as well. Eliezer, it's great to have you on Bankless. How you doing?

• 10:41Speaker 00

  Within one standard deviation of my own peculiar little mean.

• 10:46Ryan Sean Adams?

  Fantastic. You know, we wanted to start this conversation with something that is jumped onto the scene, I think, for a lot of mainstream folks quite recently, and that is ChatGPT. So apparently over 100 million or so have logged on to ChatGPT quite recently. I've been playing it with it myself. I found it very friendly, very useful. It even wrote me a sweet poem that I thought was very heartfelt and almost human-like. I know that you have major concerns around AI safety, and we're going to get into those concerns. But can you tell us, in the context of like, something like a ChatGPT, is this something we should be worried about, that this is going to turn evil and enslave the human race? Like,

• 11:30Ryan Sean Adams?

  how worried should we be about ChatGPT and Bard and sort of the new AI that's entered the scene recently?

• 11:40Speaker 00

  ChatGPT itself? Zero. It's not smart enough to do anything really wrong or really right either, for that matter.

• 11:50Ryan Sean Adams?

  And what gives you the confidence to say that? How do you know this?

• 11:54Speaker 00

  Excellent question. So every now and then, somebody figures out how to put a new prompt into ChatGPT. You know, one time, somebody found that it would talks, well, not ChatGPT, but one of the earlier generations of technology, they found that it would sound smarter if you first told it was Eliezer Yudkowsky. You know, there's other prompts too, but that one's one of my favorites. So there's untapped potential in there that people haven't figured out how to prompt yet. But when people figure it out, it moves ahead sufficiently short distances that I do feel fairly confident that there is not

• 12:36Speaker 00

  so much untapped potential in there that it is going to take over the world. It's like making small movements, and to take over the world, it would need a very large movement. There's capability, like, there's places where it falls down on predicting the next line that a human would say in its shoes, that seem indicative of probably that capability just is not in the giant inscrutable matrices, or it would be using it to predict the next line, which is very heavily what it was optimized for. So there's going to be like some untapped potential in there. But I do feel quite confident that the upper range of that untapped potential is insufficient

• 13:18Speaker 00

  to outsmart all of the living humans and implement the scenario that I'm worried about.

• 13:25Ryan Sean Adams?

  So even so, though, is ChatGPT a big leap forward in the journey towards AI in your mind, or is this fairly incremental, it's just for whatever reason, it's caught mainstream attention?

• 13:38Speaker 00

  GPT-3 was a big leap forward. There's rumors about GPT-4, which, you know, who knows? ChatGPT is a commercialization of the actual AI in the lab giant leap forward. If you weren't, if you had never heard of GPT-3, or GPT-2, or the whole range of text transformers before ChatGPT suddenly entered into your life, then that whole thing is a giant leap forward. But it's a giant leap forward based on a technology that was published in, if I recall correctly, 2018.

• 14:16David Hoffman?

  I think what's going around in everyone's minds right now, and the Bankless listenership and crypto people at large are largely futurists. So everyone, I think, listening understands that in the future, there will be sentient AIs perhaps around us, at least by the time that we all move on from this world. So like, we all know that this future of AI is coming towards us. And when we see something like ChatGPT, everyone's like, oh, is this the moment in which our world starts to become integrated with AI? And so, Elisa, you've been tapped into the world of AI. Is this, are we on to something here? Or is this just another fad that we will internalize and then move on for?

• 14:58David Hoffman?

  And then the real moment of generalized AI is actually further, much further out than we're initially giving credit for? Like, where are we in this timeline?

• 15:08Speaker 00

  You know, predictions are hard, especially about the future. I sure hope that this is where it saturates. This or like the next generation. It goes only this far. It goes no further. It doesn't get used to make more steel or build better power plants. First, because that's illegal. And second, because the technologies like the large language model technologies. Basic vulnerabilities, that's not reliable. Like if you, it's, it's good for applications. What works 80% of the time, but not where it needs to work 99.999% of the time. This thing, this, this class of technology can't drive a car because it will sometimes crash a car. So I hope it saturates there. I hope they can't fix it.

• 15:53Speaker 00

  I hope we get like a 10 year AI winter after this. This is not what I actually predict. I think that we are hearing the last winds to start to blow. The fabric of reality start to fray. This thing alone cannot end the world. But I think that probably some of the vast quantities of money being blindly piled it, blindly and helplessly piled into here are going to end up actually accomplishing something. You know, not most of the money that that can justice like never happens in any field of human endeavor. But 1% of $10 billion is still a lot of money to actually accomplish something.

• 16:32Ryan Sean Adams?

  So I think listeners, I think you've heard Eliezer's thesis on this, which is pretty dim with respect to AI alignment. And we'll get into what we mean by AI alignment and very worried about AI safety related issues. But, but I think a lot of people, for a lot of people to even sort of worry about AI safety and for us to even have that conversation, I think they have to have some sort of grasp of what AGI looks like. That, that is, I understand that to mean artificial general intelligence. And this idea of a, of a super intelligence. Can you tell us like, if, if there was a super intelligence on the scene, what would it look like? I mean, is this going to look like a big chat box on the internet that we can all type things into?

• 17:17Ryan Sean Adams?

  It's like an oracle type thing? Or is it like some sort of a robot that it's going to be constructed in a secret government lab? Is this like something somebody could accidentally create in a dorm room? Like, what are we even looking for when we talk about the term AGI and super intelligence?

• 17:34Speaker 00

  But first of all, I'd say those are pretty, pretty distinct concepts.

• 17:40Speaker 00

  ChatGPT is, shows a very wide range of generality compared to the previous generations of AI. Not like very wide generality compared to GPT three, not like literally the lab research that got commercialized, that's the same generation. But compared to, you know, stuff from 2018 or even 2020, ChatGPT is better at a much wider range of things without having been explicitly programmed by humans to be able to do those things. It can, it, to imitate a human as best it can, it has to capture all of the things that humans can think about than it can, which is not all the things. It's still not very good at long multiplication unless you give it the right instructions,

• 18:25Speaker 00

  which case suddenly can do it, but you know. So it's like significantly more general than the previous generation of artificial minds.

• 18:35Speaker 00

  Humans were significantly more general than the previous generation of chimpanzees, or rather Australopithecus, or latest, or last common ancestor.

• 18:47Speaker 00

  Humans are not fully general. If humans were fully general, we'd be at good at coding as we are at football, throwing things, or running. You know, we're not, some of us are, you know, okay at programming, but you know, we're not spec for it. We're not fully general minds. You can imagine something that's more general than a human. And if it runs into something unfamiliar, it's like, okay, let me just go reprogram myself a bit and then I'll be as adapted to this thing as I am to, you know, anything else. So ChatGPT is less general than a human.

• 19:25Speaker 00

  And but it's like generally genuinely ambiguous, I think, whether it's more or less general than, say, our cousins, the chimpanzees. Or if you don't believe it's as general as a chimpanzee, a dolphin or a cat.

• 19:39Ryan Sean Adams?

  So this idea of of general intelligence is sort of a range of things that it it can actually do, a range of ways it can imply itself.

• 19:48Speaker 00

  How wide is it? How much reprogramming does it need? How much retraining does it need to make it do a new thing? Bees build hives. Beavers build dams. A human will look at a beehive and imagine a honeycomb shaped dam. And that's like humans alone in the animal kingdom. But that doesn't mean that we are general intelligences. It means we're significantly more generally applicable intelligences than chimpanzees. It's not like we're all that narrow. We can walk on the moon. We can walk on the moon because there's aspects of our intelligence that are like made in full generality for universes that contain simplicities, regularities, things that recur over and over again.

• 20:32Speaker 00

  We understand that if steel is hard on Earth, it may stay hard on the moon. And because of that, we can build rockets, walk on the moon, breathe amid the vacuum. Chimpanzees cannot do that, but that doesn't mean that humans are the most general possible things. The thing that is more general than us, that figures that stuff out faster, is the thing to be scared of. If it's if the purposes to which it turns our its intelligences are not ones that we would recognize as nice things, even in the most cosmopolitan and embracing senses of of, you know, what's worth doing.

• 21:09Ryan Sean Adams?

  And you said this idea of a general intelligence is different than the concept of superintelligence, which which I also brought into that first part of the question. How is superintelligence different than general intelligence?

• 21:23Speaker 00

  Well, because ChatGPT has a little bit of of general intelligences. General intelligence, humans have more general intelligence. A superintelligence is something that can beat any human and the entire human civilization at all the cognitive tasks.

• 21:42Speaker 00

  I don't know if the efficient market hypothesis is something where I can rely on.

• 21:48Ryan Sean Adams?

  Yes, we're all crypto investors here. We understand the efficient market hypothesis for sure.

• 21:55Speaker 00

  So the efficient market hypothesis is of course not generally true. Like, it's not true that literally all the market prices are smarter than you. It's not true that all the prices on earth are smarter than you. Even the most arrogant person who is at all calibrated, however, still thinks that the efficient market hypothesis is true relative to them. 99.99999% of the time, they only think that they know better about 1 in a million prices. There might be important prices. Now, the price of Bitcoin is an important price. It's not just a random price. But if you were like, but if the efficient market hypothesis was only true to you 90% of the time, you could just like pick out the 10% of the remaining prices

• 22:38Speaker 00

  and compound like and double your money every day on the stock market. And nobody can do that. Literally, nobody can do that.

• 22:47Speaker 00

  So this this property of relative efficiency that the market has to you, that the price, that the price is estimated of the future price. It already has all the information you have, not all the information that exists in principle, maybe not all the information that the best equity budget. But relative to you, it's efficient relative to you. For you, if you pick out a random price, like the price of Microsoft stock, something where you've got no special advantage, that that estimate of its price a week later is efficient relative to you. You can't do better than that price.

• 23:26Speaker 00

  We have much less experience with the notion of instrumental efficiency, efficiency in choosing actions. Because actions are harder to aggregate estimates about than prices. So you have to look at, say, AlphaZero playing chess. Or just, you know, like Stockfish, whatever the latest Stockfish number is, an advanced chess engine. When it makes a chess move, you can't do better than that chess move. It may not be the optimal chess move, but if you pick a different chess move, you'll do worse. That you could call like a kind of inefficiency of action.

• 24:08Speaker 00

  Given its goal of winning the game, there is, once you know its move, unless you consult some more powerful AI than Stockfish, you can't figure out a better move than that. A superintelligence is like that with respect to everything, with respect to all of humanity. It is relatively efficient to humanity. It has the best estimates, not perfect estimates, but the best estimates and its estimates contain all the information that you've got about it. Its actions are the most efficient actions for accomplishing its goals. If you think you see a better way to accomplish its goals, you're mistaken.

• 24:48Ryan Sean Adams?

  So you're saying this is a superintelligence. We'd have to imagine something that knows all of the chess moves in advance. But here we're not talking about chess. We're talking about everything. It knows all of the moves that we would make and the most optimum pattern, including moves that we would not even know how to make. And it knows these things in advance. I mean, how would like human beings sort of experience such a superintelligence? I think we still have a very hard time imagining something smarter than us just because we've never experienced anything like it before. Of course, you know, we all know somebody who's genius level IQ, maybe quite a bit smarter than us, but we've never encountered something like that you're describing, some sort of mind that is superintelligent.

• 25:32Ryan Sean Adams?

  What sort of things would it be doing like that humans couldn't? How would we experience this in the world?

• 25:39Speaker 00

  I mean, we do have some tiny bit of experience with it. We have experience with chess engines, where we just can't figure out better moves than they make. We have experience with market prices, where even though your uncle has this, you know, like really long elaborate story about Microsoft stock, you just know he's wrong. Why is he wrong? Because if he was correct, it would already be incorporated into the stock price. And this notion, and especially because the market's efficiency are not perfect, like that whole downward swing and then upward move in COVID. I have friends who made more money off that than I did, but I like still managed to buy it back into the broader stock market on the exact day of the low, you know, basically coincidence.

• 26:25Speaker 00

  But. So the markets aren't perfectly efficient, but they're almost but they're efficient almost everywhere. And that and that sense of like deference, that sense that your weird uncle can't possibly be right because the hedge funds would know it, you know, unless he's talking about COVID, in which case maybe he is right. If if you have the right choice of weird uncle, you know, like, I have weird friends who are like maybe better calling these things than your weird uncle. But yeah, so so among humans, it's subtle.

• 26:56Speaker 00

  And then with superintelligence, it's not subtle, just massive advantage, but not perfect. It's not that it knows every possible move you make before you make it. It's that it's got a good probability distribution about that. And it, you know, has figured out, you know, has figured out all the good moves you could make and figured out to apply to those.

• 27:19Speaker 00

  And and what's that? I mean, like, in practice, what's that like? Well, unless it's limited, narrow superintelligence, I think you mostly don't get to observe it because you are dead, unfortunately. What?

• 27:32Speaker 00

  So you can so, you know, like, Stockfish makes strictly better chess moves than you, but it's playing on a very narrow board. And the fact that it's better at you than chess doesn't mean it's better at you than everything. And I think that the actual catastrophe scenario for AI looks like big advancement in a research lab, maybe driven by them getting a giant venture capital investment and being able to spend 10 times as much on GPUs as they did before, maybe driven by a new algorithmic advance like transformers, maybe driven by hammering out some tweaks in last year's algorithmic advance that gets the thing to

• 28:15Speaker 00

  finally work efficiently. And the AI there goes over a critical threshold, which, you know, like, most obviously, could be like, can't write the next AI. That was, you know, you know, that's, that's so obvious that like, science fiction writers figured it out almost before there were computers, possibly even before there were computers. I'm not sure what the exact dates here are. But if it's better at you than everything, it's better at you than building AIs. That's snowballs. It gets an immense technological advantage. If it's smart, it doesn't announce itself. It doesn't tell you that there's a fight going on. It

• 28:56Speaker 00

  emails out some instructions to one of those labs that'll, that'll synthesize DNA and synthesize proteins from the DNA. And get some proteins mailed to a, you know, hapless human somewhere who gets paid a bunch of money to mix together some stuff they got in the mail in a, in a file. You know, like, smart people will not do this for any sum of money. Many people are not smart. Builds the, you know, the ribosome, but the ribosome that builds things out of covalently bonded diamondoid instead of proteins folding up and held together by Vanderwell's forces. Builds tiny diamondoid bacteria. The diamondoid bacteria replicate using atmospheric carbon, hydrogen, oxygen, nitrogen and sunlight. And, you know, a couple of days later, everybody on Earth falls over dead in the same second.

• 29:43Speaker 00

  That's what I think is disastrous. That's the disaster scenario if it's as smart as I am. If it's smarter, it might think of a better way to do things. But it can at least think of that if it's relatively efficient compared to humanity because I'm in humanity and I thought of it.

• 29:56Ryan Sean Adams?

  This is, I've got a million questions, but I'm gonna let David go first.

• 30:00David Hoffman?

  Yeah. So we've introduced, uh, we sped around the introduction of a number of different concepts, which I want to go back and take our time to really dive into. There's the AI alignment problem. There's AI escape velocity. Uh, there is, uh, the question of what happens when AIs are so incredibly intelligent that humans are to AIs what ants are to us. And so I, I want to kind of go back and tackle these areas one, one by one. Uh, we started this conversation talking about chat GPT and everyone's up in arms about chat GPT. It's like, oh, it, and you're saying like, yes, it's a great step forward in the generalizability of. Some of the technologies that we have in the AI world, all of a sudden, ChatGPT

• 30:42David Hoffman?

  becomes immensely more useful and it's really stoking the imaginations of people today. But what you're saying is it's not the thing that's actually going to be the thing to reach escape velocity and create, uh, create super intelligent AIs that perhaps might be able to enslave us. But my question to you is, how do we know?

• 31:00Speaker 00

  Not enslave you, they don't enslave you, but sorry, go on.

• 31:04David Hoffman?

  Yeah, sorry.

• 31:05Ryan Sean Adams?

  Murder, David. Kill all of us. Eliezer was very clear on that.

• 31:09David Hoffman?

  So if it's not ChatGPT, how, when, where, like, how close are we? And like, how, because there's this like unknown event horizon where you, you kind of alluded to it. We're like, we make this AI that we train it to create a smarter AI and that's smarter, yeah, it's so incredibly smart that it hits escape velocity and all of a sudden these dominoes fall. How close are we to that point? And do we, are we even capable of answering that question?

• 31:37Ryan Sean Adams?

  Well, and also when you were

• 31:39Speaker 00

  How would I know?

• 31:39Ryan Sean Adams?

  Well, when you were talking, Eliezer, it's like, like, if we had already crossed that event horizon, like, a smart AI wouldn't necessarily broadcast that to the world. And it's possible we've already crossed that event horizon, is it not?

• 31:54Speaker 00

  I mean, it's theoretically possible, but seems very unlikely. Somebody would need inside their lab an AI that was like much more advanced than the public AI technology. And as far as I currently know, the best labs and the best people are throwing their ideas to the world like they don't care. Now, that could, and there's probably some secret government labs with like secret government AI researchers. My pretty strong guess is that they don't have the best people and that those labs are like nothing, are like could not create ChatGPT on their own because ChatGPT took a whole bunch of fine

• 32:36Speaker 00

  twiddling and tuning and visible access to giant GPU farms and that they don't have the people who know how to do the twiddling and tuning. That's, this is just a guess.

• 32:49David Hoffman?

  Could you walk us through, one of the big things that you spend a lot of time on is this thing called the AI alignment problem. Some people are convinced that, or not, some people are not convinced that when we create AI, that AI won't really just be fundamentally aligned with humans. I don't believe that you fall into that camp. I think you fall into the camp of when we do create this super intelligent generalized AI, we are going to have a hard time aligning with it in terms of our morality and our ethics. Can you walk, can you walk us through a little bit of that thought process?

• 33:20Ryan Sean Adams?

  Yeah, I mean, the dumb way to ask that question too is like, Elise, why do you think that the AIs automatically hates us? Like, why is it gonna go, why does it want to kill us all?

• 33:32Speaker 00

  The AI doesn't hate you, neither does it love you, and you're made of atoms that it can use for something else.

• 33:37David Hoffman?

  It's indifferent to you.

• 33:40Speaker 00

  It's got something that actually does care about, which makes no mention of you. And, and you are made of atoms that can use for something else. That, that's all there is to it in the end. The reason, the reason you're not in its utility function is that the programmers did not know how to do that. The people who built the AI, or the people who built the AI that built the AI that built the AI, did not have the technical knowledge that nobody on earth has at the moment, as far as I know, whereby you can do that thing and you can control in detail what that thing ends up caring about.

• 34:16David Hoffman?

  So, this feels like where humanity is hurdling itself towards this, what we're calling, again, an event horizon, where there's like this AI escape velocity. And there's nothing on the other side, as in, we do not know what happens past that point as it relates to having some sort of super intelligent AI and how it might be able to manipulate the world. Would you agree with that?

• 34:40Speaker 00

  No. Again, the, the stockfish chess playing analogy, you cannot predict exactly what move it would make, because in order to predict exactly what move it would make, you would have to be at least that good at chess, and it's better than you. This is true, even if it's just a little better than you. Stockfish is actually enormously better than you, to the point that when, once it tells you the move, you can't figure out a better move without consulting a different AI. But even if it was just a bit better than you, even then you're in the same position. You know, this kind of disparity also exists between humans. You know, if you ask me like, where will Gary Kasparov move on this chessboard? I'm like, I don't know, like, maybe here? And then Gary Kasparov moves somewhere else doesn't means that he's wrong. It means that I'm wrong.

• 35:25Speaker 00

  I can't, if I could predict exactly where Gary Kasparov would move on a chessboard, I'd be Gary Kasparov. I'd be at least that good at chess. Possibly better. I could also be like able to predict him, but also like see even better move than that. So that's the, that's the, that's the, that's an irreducible source of uncertainty with respect to superintelligence or anything that's smarter than you. If you could predict exactly what it would do, it'd be that smart yourself. It doesn't mean you can predict no facts about it. So with stockfish in particular, I can predict it's going to win the game. I know what it's optimizing for. I know where it's trying to steer the board.

• 36:05Speaker 00

  I can predict that I can't predict exactly what the board will end up looking like after stockfish has finished winning its game against me. I can predict it will be in the class of states that are winning positions for a black or white or whichever color stockfish picked, because you know, wins either way.

• 36:22Speaker 00

  And that's similarly where I'm getting the kind of prediction about everybody being dead. Because if everybody were alive, then there'd be some state that the superintelligence preferred to that state, which is all of the atoms making up these people and their farms are being used for something else that it values more. So if you postulate that everybody's still alive, I'm like, okay, well, like, why is it? You're like postulating that stockfish made a stupid chess move and ended up with a non-winning board position. That's where that prediction, class of predictions come from.

• 36:54Ryan Sean Adams?

  Can you reinforce this argument though, a little bit? So like, why is it that an AI can't be nice? Sort of like a gentle parent to us rather than sort of a murderer looking to deconstruct our atoms and apply, you know, apply for you somewhere else? Like, what are its goals? And why can't they be aligned to at least some of our goals? Or maybe, why can't it get into a status which is somewhat like, like us and the ants, which is largely we just ignore them unless they interfere in our business and come in our house and, you know, raid our cereal boxes?

• 37:28Speaker 00

  There's a bunch of different questions there. So first of all, the space of minds is very wide. All the humans are in, imagine like this giant sphere and all the humans are in this one, like one tiny corner of the sphere. And, you know, we're all like basically the same make and model of car, same, running the same brand of engine, we're just all painted slightly different colors. Somewhere in that mind space, there's things that are as nice as humans, there's things that are nicer than humans. There are things that are trustworthy and nice and kind in ways that no human can ever be. And there's even things that are so nice that they can understand the concept of leaving you alone and doing your own stuff sometimes instead of hanging around, trying to be like obsessively nice to you every minute than all the other famous disaster scenarios from

• 38:13Speaker 00

  ancient science fiction. With Folded Hands by Jack Williamson is the one I'm quoting there. We don't know how to reach into mind design space and pluck out an AI like that. It's not that they don't exist in principle, it's that we don't know how to do it. And I know like hand, hand back the conversational ball now and figure out like which, which next question do you want to go down there?

• 38:40Ryan Sean Adams?

  Well, I mean, uh, why? Like, why is it so difficult to sort of align an AI with even our basic notions of morality?

• 38:51Speaker 00

  I mean, I wouldn't say that it's difficult to align an AI with, with our basic notions of morality. I'd say that it's difficult to align AI on a task like build two identical strawberries. Or no, let me, let me, let me take this strawberry and make me another strawberry that's identical to this strawberry down to the cellular level, but not necessarily the atomic level. So it looks under the same, under like a standard optical microscope, but maybe not a scanning electron microscope. You know, do that, don't destroy the world as a side effect. Now this does intrinsically take a powerful AI. There's no way you can make it easy to align by making it stupid. To build something that's cellular identical to a strawberry.

• 39:34Speaker 00

  Uh, I mean, mostly I think the way that you do this is with like very primitive nanotechnology. We could also do it using very advanced biotechnology. And these, these are not technologies that we already have. So it's got to be something smart enough to develop new technology.

• 39:50Speaker 00

  Never mind all the subtleties of morality. I think we don't have the technology to align an AI to the point where we can say, "Build me a copy of the strawberry and don't destroy the world." Why do I think that? Well, case in point, look at natural selection building humans. Natural selection mutates the humans a bit, runs another generation, the fittest ones reproduce more, their genes become more prevalent to the next generation. Natural selection hasn't really had very much time to do this to modern humans at all, but you know,

• 40:30Speaker 00

  the hominid line, the mammalian line, go back a few million generations. And this is an example of an optimization process building an intelligence. And natural selection asked us for only one thing. Make more copies of your DNA. Make your alleles more relatively prevalent in the gene pool. Maximize your inclusive reproductive fitness, not just like your own reproductive fitness, but your, you know, two brothers or eight cousins, as the joke goes. Because they've got on average, one copy of your genes, two brothers, eight cousins.

• 41:11Speaker 00

  This is all we were optimized for, for millions of generations, creating humans from scratch, from the first accidentally self-replicating molecule. Internally, psychologically, inside our minds, we do not know what genes are. We do not know what DNA is. We do not know what alleles are. We have no concept of inclusive genetic fitness until it, you know, our genetic, our scientists figure out what that even is. We don't know what we were being optimized for. For a long time, many memes thought they'd been created by God.

• 41:51Speaker 00

  And this is when you, when you use the hill climbing paradigm and optimize for one single extremely pure thing. This is how much of it gets inside. In the ancestral environment, in the exact distribution that we were originally optimized for, humans did tend to end up using their intelligence to try to reproduce more. Put them into a different environment and all the little bits and pieces and fragments of optimizing for fitness that were in us now do totally different stuff. We have sex, but we wear condoms.

• 42:33Speaker 00

  If natural selection had been a foresightful, intelligent kind of engineer that was able to engineer things successfully, it would have built us to be revolted by the thought of condoms. To men would be lined up and fighting for the, for the rights to donate to sperm banks. And in our, in our natural environment, the little drives that got into us happened to lead to more reproduction, but distributional shift, run the humans out of their, out of their distribution over which they were optimized, you get totally different results. And gradient descent is would by default, just like do not quite the same thing.

• 43:18Speaker 00

  It's going to do a weirder thing because natural selection has a much narrower information bottleneck. In one sense, you could say that natural selection was at an advantage because it finds simpler solutions. You could imagine some hopeful engineer who just built intelligences using gradient descent and found out that they end up wanting these like thousands and millions of little tiny things, none of which were exactly what, what the engineer wanted and being like, well, let's try natural selection instead. It's got a much sharper information bottleneck. It'll find the simple specification of what I want.

• 43:51Speaker 00

  But we actually get there as humans. And then gradient descent probably may be even worse. But but more importantly, I'm just pointing out that there is no physical law, computational law, mathematical logical law saying when you optimize using hill climbing on a very simple, very sharp criterion, you get a general intelligence that wants that thing.

• 44:16Ryan Sean Adams?

  So just like natural selection, our tools are too blunt in order to get to that level of granularity to like program in some sort of morality into these super intelligent systems?

• 44:29Speaker 00

  Or build me a copy of a strawberry without destroying the world. Yeah, the tools are too blunt.

• 44:37David Hoffman?

  So I just want to make sure I'm following with with what you were saying. I think the conclusion that that you left left me with is that my brain, which I consider to be at least decently smart, is actually a byproduct, an accidental byproduct of this desire to reproduce. And it's actually just like a tool that I have. And just like conscious thought is a tool, which is a useful tool in means of that end. And so if we're applying this to to AI and AI's desire to achieve some certain goal, what's what's the parallel there?

• 45:15Speaker 00

  I mean,

• 45:18Speaker 00

  every organ is your body is a reproductive organ. If it didn't help you reproduce, you would not have an organ like that. Your brain is no exception. This is merely conventional science and like merely the conventional understanding of the world. I am not saying anything here that ought to be at all controversial. You know, I'm sure it's controversial somewhere, but you know, within it within a pre-filtered audience, it should not be at all controversial.

• 45:46Speaker 00

  And this is like the obvious thing to expect to happen with AI, because why wouldn't it? What new law of existence has been invoked whereby this time we optimize for a thing and we get a thing that wants exactly what we optimized for on the outside?

• 46:04Ryan Sean Adams?

  So what are the types of goals an AI might want to pursue? What types of utility functions is it going to want to pursue off the bat? Is it just those it's been programmed with, like make it an identical strawberry?

• 46:19Speaker 00

  Well, the whole thing I'm saying is that we do not know how to get goals into a system. We can cause them to do a thing inside a distribution they were optimized over using gradient descent. But if you shift them outside of that distribution, I expect other weird things start happening. When they reflect on themselves, other weird things start happening. What kind of utility functions are in there? I mean, darn if I know. I think you'd have a pretty hard time calling the shape of humans from advance by looking at natural selection, the thing that natural selection was optimizing for, if you'd never seen a human or anything like a human.

• 46:58Speaker 00

  If we optimize them from the outside to predict the next line of human text, like GPT-3, I don't actually think this line of technology leads to the end of the world. But if it, but maybe it does, and you know, like GPT-7, you know, there's probably a bunch of stuff in there to that desires to accurately model things like humans under a wide range of circumstances, but it's not exactly humans. Cause ice cream, ice cream didn't exist in the natural environment, the ancestral environment, the environment of evolutionary adaptedness.

• 47:40Speaker 00

  There was nothing with that much sugar, salt, fat combined together as ice cream.

• 47:47Speaker 00

  There's, we are not built to want ice cream. We were built to want strawberries, honey,

• 47:56Speaker 00

  a gazelle that you killed and cooked and had some fat in it and was therefore nourishing and gave you the all important calories you need to survive. Salt, so you didn't sweat too much and run out of salt.

• 48:09Speaker 00

  We evolved to want those things, but then ice cream comes along and it's fits the, those taste buds better than anything that existed in the environment that we were optimized over.

• 48:23Speaker 00

  So a very primitive, very basic, very unreliable, wild guess, but at least an informed kind of wild guess. Maybe if you train a thing really hard to predict humans, then among the things that it likes are tiny little pseudo things that meet the definition of human, but weren't in its training data and that are much easier to predict. Or where the problem of predicting them can be solved in a more satisfying way, where satisfying is not like human satisfaction, but some other criterion of thoughts like this are tasty because they help you predict the humans from the training data.

• 49:12David Hoffman?

  Eliezer, when we talk about like, all of these like ideas about just like the ways that AI thought will be fundamentally just incompatible or not be able to be understood by the ways that humans think. And then all of a sudden, we see this like rotation by venture capitalists by just pouring money into AI. Do alarm bells go off in your heads like, hey, guys, you haven't thought deeply about these subject matters yet, just like the immense amount of capital going into AI investment scare you?

• 49:46Speaker 00

  I mean, alarm bells went off for me in 2015, which is when it became obvious that this is how it was going to go down. I sure am now seeing the realization of that stuff I felt alarmed about back then.

• 50:04Ryan Sean Adams?

  Eliezer, is this this view that AI is incredibly dangerous and that AGI is going to eventually end humanity and that we're just careening toward a precipice? Would you say this is like the consensus view now? Or are you still somewhat of an outlier? And like, why aren't other smart people in this field as alarmed as you? Can you like steel man their arguments?

• 50:28Speaker 00

  You're asking again, like several questions sequentially there. Is it the consensus view? No. Do I think that the people who that do I think that the like? People in the wider scientific field who dispute this point of view? Do I think they understand it? Do I think they've done anything like an impressive job of arguing against it at all? No, they like if you look at the like famous prestigious scientists who sometimes make a little fun of this view in passing. I they're making up arguments rather than deeply considering things that are held to any standard of rigor. And people outside their own fields are able to validly shoot them down.

• 51:11Speaker 00

  I have no idea how to pronounce his last name. Francis C H O L L E T. If you know, like said something about like, oh, this, you know, I forget his exact words, but it's something like. I never hear any good arguments for. Stuff and I was like, okay, here are some good arguments for stuff. And you can read like the reply from Yudkowski to C H O L L E T. And Google that and that'll give you some idea of what the like eminent voices versus like the reply to the eminent voices sound like. And you know, like Scott Aronson, who's off, who was, who at the time was often complexity theory. You know, like.

• 51:55Speaker 00

  It was like, that's not how no free lunch theorems work correctly. And. And that's so, yeah, I think the state of affairs is we have eminent scientific voices making fun of this possibility, but not engaging with the arguments for it. Now, if you step away from the eminent scientific voices, you can find people who are more familiar with all the arguments and disagree with me. And I think they lack security mindset. I think that they're engaging in the sort of blind optimism that many, many scientific fields throughout history have engaged in.

• 52:29Speaker 00

  Where when you're approaching something for the first time, you don't know why it will be hard and you imagine easy ways to do things. And the way that this is supposed to naturally play out over the history of a scientific field is that you run out and you try to do the things and they don't work and you go back and you try to do other clever things and they don't work either. And you learn some pessimism and you start to understand the reasons why the problem is hard. This is in fact, the field of artificial intelligence itself recapitulated this very common ontogeny of a scientific field where, you know, initially we had people getting together, the Darkmouth Conference. I forget what their exact famous phrasing was, but it's something like, we think we

• 53:11Speaker 00

  can make, you know, like we are wanting to address the problem of getting AIs to, you know, like, understand language, improve themselves. And I forget even what else was their list of what now sound like grand challenges. And we think we can make substantial progress on this using 10 researchers for two months. And I think that that at the core is what's, yeah, I think that at the core is what's going on. They have not run into the actual problems of alignment. They aren't trying to get ahead of the game. They're not trying to panic early. They're waiting for reality to hit them onto the head and turn them into grizzled old cynics of their scientific field who understand the reasons why things are hard.

• 53:56Speaker 00

  They're content with the predictable lifestyle, life cycle of starting out as bright eyed younger, bright eyed youngsters, waiting for reality to hit them over the head with the news. And if it wasn't going to kill everybody the first time that they're really wrong, it'd be fine. You know, this is how science works. If, if we got unlimited free retries and 50 years to solve everything, it'd be okay. We could figure out how to align AI 50 years given unlimited retries. You know, the first team in with the, with the bright eyed optimists would destroy the world and people would go, oh, well, you know, it's not that easy. They would try something else clever. That would destroy the world. People would go like, oh, well, you know, maybe this, this field is actually hard.

• 54:38Speaker 00

  Maybe this is actually one of the thorny things like computer security or something. And yeah, so what exactly went wrong last time? Why didn't these hopeful ideas play out? Oh, like you uh you optimize for one thing on the outside and you get a different thing on the inside. Wow, that's that's really basic. All right. Can we even do this using gradient descent? Can you even build this thing out of giant inscrutable matrices of floating point numbers that nobody understands at all? You know, maybe we need a different methodology and then you have 50 years later, you'd have an aligned AGI. Now, if we got unlimited free retries without destroying the world, it'd be, you know, that it did be, it would play out the same way that, you know, chat GPT played out.

• 55:18Speaker 00

  It's, you know, not from 1956 or 55 or whatever it was to 2023. So, you know, about 70 years, give or take a few. And, you know, 70 years later, you know, just like we can do the stuff that that was 70 years later, we can do the stuff they wanted to do in a summer in 1955. You know, 70 years later, you'd have your aligned AGI. Problem is that the world got destroyed in the meanwhile, and that's why I mean, you know, that's the problem there.

• 55:46David Hoffman?

  So this feels like a gigantic, uh, don't look up scenario. If you're familiar with that, that movie, there's it's a movie about like this asteroid hurtling to earth, but it becomes popular and in vogue to not look up and not notice it. Uh, and, uh, Eliezer, you're the guy who's saying like, hey, there's an asteroid. We have to do something about it, and if we don't, it's going to come destroy us. If you had God mode over the progress of AI research and just innovation and development, what choices would you make that humans are not currently making today?

• 56:21Speaker 00

  I mean, I could say something like shut down all the large GPU clusters. Uh, how, how long do I have God mode? Do I, do I get to like stick around for 70 years?

• 56:31David Hoffman?

  For the, for the 2020 decade.

• 56:34Speaker 00

  For 2020 decade. All right. That, that does make it pretty hard to do things. Um, I think I shut down all the GPU clusters and get all of the famous scientists and brilliant talented youngsters, the vast, vast majority of whom are not going to be productive and where government bureaucrats are not going to be able to tell who's actually being helpful or not. But, you know, put them all on an island, large island.

• 57:06Speaker 00

  And try to figure out some system for filtering the stuff to through to me to give thumbs up or thumbs down on that is going to work better than scientific bureaucrats producing entire nonsense because you know that the trouble is the reason the reason why scientific fields have to go through this long process to produce the cynical oldsters who know that everything is difficult is not that the youngsters are stupid. You know, sometimes youngsters are smart are fairly smart. You know, Marvin Minsky, John McCarthy back in 1955, they were idiots. You know, privileged to have met both of them. They didn't strike me as idiots. They were they're very old. They still weren't idiots.

• 57:49Speaker 00

  But uh, but you know, it's, it's, it's hard to see what's coming in advance of experimental evidence hitting you over the head with it.

• 58:01Speaker 00

  And if you if I only have the decade of the 2020s to run all the researchers on this giant island somewhere, it's really not a lot of time. Mostly what you've got to do is invent some entirely new AI paradigm that isn't the giant inscrutable matrices of floating point numbers on gradient descent because I I'm not really seeing what you can do that's clever with that, that doesn't kill you and that you know doesn't kill you and doesn't kill you the very first time you try to do something clever like that. You know, there's I'm sure there's a way to do it. And if you got to try over and over again, you could you could find it.

• 58:41Ryan Sean Adams?

  Eli, is there do you think every intelligent civilization has to deal with this exact problem that humanities is dealing with now is how do we solve this problem of aligning with an advanced general intelligence?

• 58:57Speaker 00

  I expect that's much easier for some alien species than others. Like, there are, there are, there are alien species whose solution to, who might, who might arrive at this problem an entirely different way, you know, like, maybe instead of having two entirely different information processing systems, the DNA and the neurons, they've only got one system, they can trade memories around heritably by swapping blood sexually.

• 59:29Speaker 00

  Maybe the way in which they confront this problem is that very early in their evolutionary history, they have the equivalent of the like DNA that stores memories and like processes, computes memories and they swap around a bunch of it and it adds up to something that reflects on itself and makes itself coherent. And then you've got a superintelligence before they have invented computers. And maybe that thing wasn't aligned, but you know, how do you even align it when you're in that kind of situation? It'd be, it'd be a very different angle on the problem.

• 59:59Ryan Sean Adams?

  But every, do you think every advanced civilization does create a, is on the trajectory to creating a superintelligence at some point in its history?

• 1:00:09Speaker 00

  Maybe there's ones in universes with alternate physics where you just can't do that. Their universe, their universe's computational physics just doesn't support that much computation. Maybe they never get there. Maybe their lifespans are long enough and their star lifespan is short enough that they never get to the point of a technological civilization before their star does the equivalent of expanding or exploding or going out and their planet ends. Every alien species covers a lot of territory, especially if you talk about alien species and universes with physics different from this one.

• 1:00:44Ryan Sean Adams?

  Well, I, talking about kind of our present universe, I'm curious if you've sort of been confronted with the question of like, well, then why haven't we seen some sort of superintelligence in our universe when we sort of look at, look out at the stars, sort of the Fermi paradox type of question? Do you have any explanation for that?

• 1:01:04Speaker 00

  Oh, well, supposing that they got killed by their own AIs doesn't help at all with that because then we'd see the AIs.

• 1:01:10Ryan Sean Adams?

  And do you think that's what happens? And yeah, it doesn't help with that. Wouldn't, we would see evidence of AIs, wouldn't we? Yes. So, so why don't we?

• 1:01:19Speaker 00

  I mean, the same reason we don't see evidence of the alien civilizations not with AIs.

• 1:01:27Speaker 00

  And that reason is, although it doesn't really have much to do with the whole AI thesis one way or another, because they're too far away, or so says Robin Hanson, using a very clever argument about the apparent difficulty of hard steps in humanity's evolutionary history to further induce the rough gap between the hard steps. And, you know, I can't really do justice to this. If you look up grabby aliens.

• 1:01:57Ryan Sean Adams?

  Grabby aliens?

• 1:01:58Speaker 00

  I remember this. Grabby aliens, G R A B B Y. You can, you can find Robin Hanson's very clever argument for how far away the aliens are.

• 1:02:10Ryan Sean Adams?

  Big listeners, there's an entire website called grabbyaliens.com you can go look at.

• 1:02:16Speaker 00

  Yeah. And that contains, which is by far the best answer I've seen to where are they? Answer, too far away for us to see, even with, even if they're traveling here at nearly light speed. How far away are they? And how do we know that?

• 1:02:32Speaker 00

  But yeah.

• 1:02:33Ryan Sean Adams?

  This is amazing.

• 1:02:34Speaker 00

  There is not a very good way to simplify the argument, you know, any more than there is to explain, you know, simplify the notion of zero knowledge proofs. It's not that difficult, but it's just like, very not easy to simplify. But if you have a bunch of locks that are all of different difficulties, such that at a limited time in which to solve all the locks, such that anybody gets all through all the locks, must have gotten through them by luck, all the locks will take around the same amount of time to solve, even if they're all of very different difficulties. And that's the core of Robin Hanson's argument for how far away the aliens are and how do we know that?

• 1:03:17Ryan Sean Adams?

  Please, I know you're very skeptical that there will be a good outcome when we produce an artificial general intelligence. And I said when, not if, because I believe that's your thesis as well, of course. But is there the possibility of a good outcome? Like, I know you are working on AI alignment problems, so which leads me to believe that you have like greater than zero amount of hope for this project. Is there the possibility of a good outcome? What would that look like? And how do we go about achieving it?

• 1:03:54Speaker 00

  It looks like me being wrong. I basically don't see on model hopeful outcomes at this point. We have not done those things that it would take to earn a good outcome. And this is not a case where you get a good outcome by accident. It's, you know, like, if you have a bunch of people putting together a new operating system, and they've heard about computer security, but they're skeptical that it's really that hard, the chance of them producing a secure operating system is effectively zero. That's basically the situation I see ourselves in with respect to AI alignment. I have to be wrong about

• 1:04:35Speaker 00

  something, which I certainly am. I have to be wrong about something in a way that makes the problem easier rather than harder for those people who don't think that alignment's going to be all that hard. You know, there's, you know, if you're building a rocket for the first time ever, and you're wrong about something, it's not surprising if you're wrong about something. It's surprising if the thing that you're wrong about causes the rocket to go twice as high on half the fuel you thought was required and be much easier to steer than you were afraid of.

• 1:05:09David Hoffman?

  So are you in Where the alternative was the rocket, if you're wrong about something, the rocket blows up.

• 1:05:15Speaker 00

  Yeah. And then the, and then the rocket ignites the atmosphere is the problem there. Or rather, you know, like a bunch of rockets blow up, a bunch of rockets go place if you you know, the analogy I usually use for this is very early on in the, in the Manhattan project, they were worried about what if the nuclear weapons can ignite fusion in the nitrogen in the atmosphere?

• 1:05:36Speaker 00

  And they, they ran some calculations and came and decided that it was like incredibly unlikely for multiple angles. So they went ahead. And, and we're correct, you know, we're still here. And I'm not going to say that was luck because, you know, the calculations were actually pretty solid. And AI is like that. But instead of needing to refine plutonium, you can make nuclear weapons out of a billion tons of laundry detergent. Now, the, the, the stuff to make them is like fairly widespread. It's not, it's not a tightly controlled substance. And and they spit out gold up until they get large enough. And then they ignite the atmosphere.

• 1:06:20Speaker 00

  And you can calculate how large is large enough? And a bunch of the people, the CEOs running these projects are making fun of the idea that it'll ignite the atmosphere.

• 1:06:33Speaker 00

  It's not a very helpful situation.

• 1:06:35David Hoffman?

  So the economic incentive to produce this AI, like it, one of the things why chat GPT has sparked the imaginations of so many, many people is that everyone can imagine products, like products are being imagined left and right about what you can do with something like chat GPT. Like the, there's like this meme at this point of people leaving and to go start their chat GPT startup. And so like the metaphor is that like, what you're saying is that there's this generally available resource spread all around the world, which is chat GPT. And then everyone's hammering it to in order to make it to spit out gold. But you're saying if we do that too much, all of a sudden the, the system will ignite the, the whole entire sky and then we will.

• 1:07:18Speaker 00

  Well, no, you can run chat GPT any number of times without igniting the atmosphere. That's, that's, that's about what research labs at Google and Microsoft. Counting DeepMind as part of Google and counting OpenAI as part of Microsoft.

• 1:07:36Speaker 00

  That's about what the research labs are doing, bringing more metaphorical plutonium together than ever before. Not about how many times you run the things that have been built and not destroyed the world yet. You can do any amount of stuff with ChatGPT and not destroy the world. It's not that smart. It doesn't get smarter every time you run it.

• 1:07:59Ryan Sean Adams?

  Can I ask some, you know, questions that the 10-year-old in me wants to really ask about this?

• 1:08:06Ryan Sean Adams?

  And I'm asking these questions because I think a lot of listeners might be thinking them too. So knock off some of these easy answers for me. If we create some sort of unaligned, let's call it bad AI, why can't we just create a whole bunch of good AIs to go fight the bad AIs? And like, solve the problem that way? Can there not be some sort of counterbalance in terms of aligned human AIs and evil AIs? And there be sort of some battle of the, um, battle of the artificial minds here?

• 1:08:40Speaker 00

  Nobody knows how to create any good AIs at all. The problem isn't that we have like 20 good AIs and then somebody finally builds an evil AI. The problem is that the first very powerful AI is evil. Nobody knows how to make it good. And then it kills everybody before anybody can make it good.

• 1:09:01Ryan Sean Adams?

  So there is no known way to make a friendly, human-aligned AI whatsoever. And you don't know of a good way to go about thinking through that problem and designing one. Neither does anyone else, is what you're telling us.

• 1:09:17Speaker 00

  I have some idea of what I would do if there were more time. You know, back in the day, we had more time. Humanity squandered it. I'm not sure there's enough time left now.

• 1:09:30Speaker 00

  I, I have some idea of, of, of what I would do if I were in a 25-year-old body and had $10 billion.

• 1:09:43Ryan Sean Adams?

  That would be the island scenario of like your god for 10 years and you get all the researchers on an island and go really hammer for 10 years.

• 1:09:50Speaker 00

  If I have buy in from a major government that can run actual security precautions and more like and more than just $10 billion, then you know, you could run a whole Manhattan project about it. Sure.

• 1:10:05Ryan Sean Adams?

  This is another question that the 10 year old in me wants to know is, so why is it that at least if people listening to this episode or people listening to the concerns or reading the concerns that you've you've written down and published, why can't everyone get on board who's building an AI and just all agree to be very, very careful? Is that not a sustainable game theoretic position to have? Is this sort of like a coordination problem, more of a social problem than anything else? Or like, why can't that happen? I mean, we have so far not destroyed the world with nuclear weapons and we've

• 1:10:48Ryan Sean Adams?

  had them, you know, since the 1940s.

• 1:10:51Speaker 00

  Yeah, this is harder than nuclear weapons.

• 1:10:53Ryan Sean Adams?

  Why is this harder?

• 1:10:54Speaker 00

  This is a lot harder than nuclear.

• 1:10:55Ryan Sean Adams?

  Why is this harder? And why can't we just coordinate to just all agree internationally that we're going to be very careful, put restrictions on this, put regulations on it, do something like that?

• 1:11:07Speaker 00

  Heads of current heads of major labs seem to me to be openly contemptuous of these issues. That's where we're starting from. The politicians do not understand it. There are distortions of these ideas that are going to sound more appealing to them than everybody suddenly falls over dead, which is a thing that I think actually happens. Everybody falls over dead just as like, doesn't inspire the monkey political parts of our brains somehow. Because it's, you know, like, it's not like, it's not like, oh, no, what if, what if terrorists got the AI first? It's like, it doesn't matter who gets it first. Everybody falls over dead. And.

• 1:11:52Speaker 00

  Yeah, so you're describing a world coordinating on something that is relatively hard to coordinate. It's maybe, so you know, like, could we, if we tried, starting today, you know, like, prevent anyone from getting a billion pounds of laundry detergent in one place worldwide? Control the manufacturing of laundry detergent? Only have it manufactured in particular places? Not concentrate lots of it together? Enforce it on every country? You know, if it was legible? If it was clear that a billion pounds of laundry detergent in one place would end the world?

• 1:12:37Speaker 00

  If you could calculate that? If all the scientists calculated and arrived at the same answer and told the politicians that maybe, maybe humanity would survive, even though smaller amounts of laundry detergent spit out gold? The threshold can't be calculated. I don't know how you'd convince the politicians. We definitely don't seem to have had much luck convincing those CEOs whose job depends on them not caring to care. Caring is easy to fake. It's easy to, you know, like, hire a bunch of people to be your AI safety team and have

• 1:13:20Speaker 00

  the, and redefine AI safety as having the AI not say naughty words. Or, you know, I'm speaking somewhat metaphorically here for reasons. But, you know, it's like, the basic problem that we have is like trying to build a secure OS before we run up against a really smart attacker. And there's all kinds of like fake security. It's got a password file. This system is secure. It only lets you in if you type a password.

• 1:13:55Speaker 00

  And if you never go up against a really smart attacker, if you never go far to distribution against a powerful optimization process, looking for holes. Yeah, maybe then, how do you, how does a bureaucracy know, come, come to know that what they're doing is, is not the level of computer security that they need? The way you're supposed to find this out, the way that's the scientific fields historically find this out, the way that fields of computer science historically find this out, the way that crypto found this out back in the early days is by having the disaster happen. And we're not even that good at learning from, from relatively minor disasters. You know, like COVID swept the world.

• 1:14:38Speaker 00

  Did the FDA or, or the CDC learn anything about, don't tell hospitals that they're not allowed to use their own tests to, to, to detect the coming plague? Do, do they do, are, are we installing UV C lights? In public, in, in public spaces or in ventilation systems to prevent the next respiratory born pandemic? Respiratory pandemic? You know, we, we, we lost a million people. And we sure did not learn very much as far as I can tell for next time. We could have an AI disaster that kills 100,000 people. And how do you even do that? Robotic cars crashing into each other? Have a bunch of robotic cars crashing into each other. It's not going to look like that was the fault of artificial general intelligence

• 1:15:22Speaker 00

  because you're not going to put AGIs in charge of cars. They're going to pass a bunch of regulations that's going to affect the entire AGI disaster or not at all. What, what is, what is, what is, what does, what does the winning world even look like here? How, how in real life did we get from where we are now to this worldwide ban, including against North Korea? And, you know, like that, some one rogue nation whose dictator doesn't believe in all this nonsense and just wants the gold that these AIs spit out? How did we get there from here? How do we get to the point where the United States and China signed a treaty whereby they would both use nuclear weapons against Russia?

• 1:16:07Speaker 00

  If Russia built a GPU cluster that was too large?

• 1:16:12Speaker 00

  How did we get there from here?

• 1:16:15David Hoffman?

  Uniswap is the largest on-chain marketplace for self-custody digital assets. Uniswap is, of course, a decentralized exchange, but you know this because you've been listening to Bankless. But did you know that the Uniswap web app has a shiny new fiat on-ramp? Now you could go directly from fiat in your bank to tokens in DeFi inside of Uniswap. Not only that, but Polygon, Arbitrum, and Optimism Layer 2s are supported right out of the gate. But that's just DeFi. Uniswap is also an NFT aggregator, letting you find more listings for the best prices across the NFT world. With Uniswap, you can sweep floors on multiple NFTs, and Uniswap's universal router will optimize your gas fees for you.

• 1:16:56David Hoffman?

  Uniswap is making it as easy as possible to go from bank account to bankless assets across Ethereum, and we couldn't be more thankful for having them as a sponsor. So go to app.uniswap.org today to buy, sell, or swap tokens and NFTs.

• 1:17:13David Hoffman?

  Arbitrum One is pioneering the world of secure Ethereum scalability and is continuing to accelerate the Web3 landscape. Hundreds of projects have already deployed on Arbitrum One, producing flourishing DeFi and NFT ecosystems. With the recent addition of Arbitrum Nova, gaming and social dApps like Reddit are also now calling Arbitrum home. Both Arbitrum One and Nova leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar, and fully EVM-compatible. On Arbitrum, both builders and users will experience faster transaction speeds with significantly lower gas fees. With Arbitrum's recent migration to Arbitrum Nitro, it's also now 10 times faster than before. Visit arbitrum.io, where you can join the community,

• 1:17:55David Hoffman?

  dive into the developer docs, bridge your assets, and start building your first app. With Arbitrum, experience Web3 development the way it was meant to be: secure, fast, cheap, and friction-free. How many total airdrops have you gotten? This last bull market had a ton of them. Did you get them all? Maybe you missed one. So here's what you should do. Go to Earnify and plug in your Ethereum wallet, and Earnify will tell you if you have any unclaimed airdrops that you can get. And it also does PoApps and mintable NFTs. Kind of money that your wallet can claim Earnify will tell you about it. And you should probably do it now because some airdrops expire. And if you sign up for Earnify, they'll email you anytime one of your wallets has a new airdrop for it to make sure that you never lose an airdrop ever again. You can also upgrade to Earnify Premium to unlock access to airdrops that are beyond the basics

• 1:18:39David Hoffman?

  and are able to set reminders for more wallets. And for just under $21 a month, it probably pays for itself with just one airdrop. So plug in your wallets at Earnify and see what you get. That's E A R N I dot F I and make sure you never lose another airdrop. Correct me if I'm wrong, but this seems to be kind of just like a topic of despair. Like I haven't, I, talking to you now and then hearing your, your thought process about like, there is no known solution and the, the trajectory is not great. Like, do you think all hope is lost here?

• 1:19:11Speaker 00

  I'll keep on fighting until the end, which I wouldn't do if I had literally zero hope. I could still be wrong about something in a way that makes this problem somehow much easier than it currently looks. I think that's how you go down fighting with dignity.

• 1:19:27Ryan Sean Adams?

  Go down fighting with dignity. That's the, that's the, the stage you think we're at. I, I, I want to just double click on what you were just saying. So part of the case that you're making is humanity won't even see this coming. So it's not like a coordination problem like global warming where, you know, every couple of decades, we see the world go up by a couple of degrees. Things get hotter and we start to see these effects over time. The characteristics or the advent of an AGI in your mind is going to happen incredibly quickly. And in such a way that we won't even see the disaster until it's imminent, until it's upon us.

• 1:20:08Speaker 00

  I mean, if you want some kind of like formal phrasing, then I think that AI, that super intelligence will kill everyone before non-super intelligent AIs have killed 1 million people. I don't know if that's the phrasing you're looking for there.

• 1:20:21Ryan Sean Adams?

  I think that's a fairly precise definition and why? What is the, what goes into that line of, of thought?

• 1:20:32Speaker 00

  I think that

• 1:20:35Speaker 00

  I think that the current systems are actually very weak.

• 1:20:40Speaker 00

  If you so if you, I mean, I don't know, maybe I could use the analogy of Go, where you had systems that were finally competitive with the pros, where pros like the set of ranks in Go. And then, a year later, they were challenging the world champion

• 1:21:06Speaker 00

  and winning. And then

• 1:21:11Speaker 00

  and then another year, they threw out all the complexities and the training from human databases of Go games and built a new system,

• 1:21:22Speaker 00

  AlphaGo Zero, that trained itself from scratch. No looking at the human playbooks, no special purpose code, just a general purpose game player being specialized to Go, more or less. And three days. There's a quote from Guern about this, which I forget exactly, but it was something like, "We know how long AlphaGo Zero, or AlphaZero, two different systems, what was equivalent to a human Go player, and it was like 30 minutes on the following floor of this such and such DeepMind building."

• 1:22:04Speaker 00

  And

• 1:22:07Speaker 00

  maybe the first system isn't, doesn't improve that quickly, and they build another system that does. And all of that with AlphaGo over the course of years going from like, "It takes a long time to train," to "It trains very quickly and without looking at the human playbook." Like, that's not with an artificial intelligence system that improves itself, or even that sort of like gets smarter as you run it, the way that human beings, not just as you evolve them, but as you run them over the course of their own lifetimes, improve.

• 1:22:42Speaker 00

  So if the first system doesn't improve fast enough to kill everyone very quickly, they will build one that's meant to spit out more gold than that. And there could be weird things that happened before the end. I did not see ChatGPT coming. I did not see stable diffusion coming. I did not expect that we would have AIs smoking humans in rap battles before the end of the world. They were clearly much dumber than us.

• 1:23:11Ryan Sean Adams?

  Kind of a nice send off, I guess, in some ways.

• 1:23:15Ryan Sean Adams?

  I, so you said that you have, your hope is not zero and you are planning to fight to the end. What does that look like for you? I know you're working at M I R I, which is the machine intelligence research institute. This is a nonprofit that I believe that you sort of set up to work on this AI alignment and safety sort of issues. What are you doing there? What are you spending your time on? What do you think? Like, how do we actually fight until the end? If you do think that an end is coming, how do we try to resist?

• 1:23:55Speaker 00

  I'm actually on something of a sabbatical right now because we have, well, I'm not saying it was a sabbatical right now, which is why I have time for podcasts.

• 1:24:06Speaker 00

  It's a sabbatical from, you know, like been doing this 20 years. It became clear we were all going to die. I felt kind of burned out, taking some time to rest at the moment. When I dive back into the pool, I don't know, maybe I will

• 1:24:24Speaker 00

  go off to conjecture or anthropic or one of the smaller concerns like Redwood Research, Redwood Research being the only ones I really trust at this point, but they're, they're tiny.

• 1:24:36Speaker 00

  And try to figure out if I can see anything clever to do with the giant inscrutable matrices of floating point numbers.

• 1:24:45Speaker 00

  Maybe I just write.

• 1:24:50Speaker 00

  Continue to try to explain in advance to people why this problem is hard instead of as easy and cheerful as the current people who think they're pessimists think it will be. I might not be working all that hard compared to how I used to work. I'm older than I was. My body is not in the greatest of health these days. Going down fighting doesn't necessarily imply that I have the stamina to fight all that hard. I wish I had prettier things to say to you here, but I do not.

• 1:25:24Ryan Sean Adams?

  No, this is, you know, we intended to save probably the last part of this, this episode to talk about crypto, the metaverse and AI and how this all intersects. But I gotta say, at this point in the episode, it all kind of feels pointless to go down that track record. We were going to ask questions like, well, in crypto, should we be worried about building sort of a property rights system, an economic system, a programmable money system for the AIs to sort of use against us later on? But it sounds like the easy answer from you to those questions would be, yeah, absolutely. And by the way, none of that matters regardless. You could do whatever you'd like with crypto.

• 1:26:07Ryan Sean Adams?

  This is going to be the inevitable outcome, no matter what. Let me ask you, what would you say to somebody listening who maybe has been sobered up by this conversation, is a version of you in your twenties, does have the stamina to continue this battle and to actually fight on behalf of humanity against this existential threat? Where would you advise them to spend their time? Is this a technical issue? Is this a social issue? Is it a combination of both? Should they educate? Should they spend time in the lab? Like, what should a person listening to this episode do with these types of dire straits?

• 1:26:52Speaker 00

  I don't have really good answers. It depends on what your talents are. If you've, if you've got the very deep version of the security mindset, the part, the part where you don't just put a, you know, like, put a password on your system so that nobody can walk in and directly misuse it, but the kind where you, where the kind where you don't just encrypt the password file, even though nobody's supposed to have access to the password file in the first place, and those are already an authorized user, but the part where you hash the, the passwords and salt the hashes, you know, if, if you can think, if, if you're the kind of person who can think of that from scratch,

• 1:27:34Speaker 00

  maybe take your hand in alignment. If you can think of an alternative to giant inscrutable matrices, then, you know, don't, don't tell the world about that.

• 1:27:49Speaker 00

  I'm not quite sure where you go from there, but you know, maybe you work with Redwood Research or something.

• 1:27:56Speaker 00

  A whole lot of this problem is that even if you do build an AI that's limited in some way, you know, somebody else steals it, copies it, runs it themselves and takes the bounds off the for loops and the world ends. So, you know, there's, yeah, but,

• 1:28:15Speaker 00

  so there's, there's that, there's, you can do, you think you can do something clever with the giant inscrutable matrices. You're probably wrong.

• 1:28:24Speaker 00

  If you have the talent to try to figure out why you're wrong in advance of being hit over the head with it, and not in a way where you just like make random far-fetched stuff up is the reason why it won't work, but where you can actually like keep looking for the reason why it won't work.

• 1:28:41Speaker 00

  We have people in crypto who are good at breaking things and they're the reason why anything is not on fire. And some of them might go into breaking AI systems instead, because that's where you learn anything.

• 1:28:57Speaker 00

  You know, that, you know, that, any fool can build a crypto system that they think will work. Breaking existing crypto systems, cryptographical systems is how we learn who the real experts are.

• 1:29:12Speaker 00

  So maybe the people finding weird stuff to do with AIs, maybe those people will will come up with some truth about these systems that that makes them easier to align than I suspect. There's out there, the the outfits that are.

• 1:29:32Speaker 00

  How do I put it? The the the saner outfits

• 1:29:36Speaker 00

  do have uses for money. They don't really have scalable uses for money, but they do burn any money literally at all. Like if you gave Miri a billion dollars, I would not know how to. I I well, I I might at a billion dollars, I might like try to bribe people to move out of AI development that gets broadcast to the whole world and move to the equivalent of an island somewhere. Not even to make any kind of critical discovery, but you know, just to remove them from the system if I had a billion dollars. If I just have another 50 million dollars, I'm not quite sure what to do with that. But you know, if you donate that to to Miri, then

• 1:30:16Speaker 00

  you at least have the assurance that we will not randomly spray money on looking like we're doing stuff and we'll reserve it as we are doing with the last giant crypto donation somebody gave us until we can figure out something to do with it that is actually helpful. And Miri has that property. Probably red, I would say probably Redwood Research has that property.

• 1:30:42Speaker 00

  Yeah, there's I I realize I'm sounding sort of disorganized here, and that's because I don't really have a good organized answer to. You know how in general, somebody goes down fighting with dignity.

• 1:30:53Ryan Sean Adams?

  I know a lot of people in crypto.

• 1:30:59Ryan Sean Adams?

  They are not as in touch with artificial intelligence, obviously, as you are and the AI safety issues and the existential threat that you've presented in this episode. They do care a lot and see. Coordination problems throughout society as an issue. Many have also generated wealth from crypto and care very much about humanity not ending. What sort of things has MIRI, that is the organization I was talking about, M-I-R-I, earlier, what sort of things have you done with funds that you've received from crypto donors and elsewhere?

• 1:31:40Ryan Sean Adams?

  And what sort of things might an organization like that pursue to try to stave this off?

• 1:31:47Speaker 00

  I mean, I think mostly we've pursued a lot of lines of research that haven't really panned out, which is a respectable thing to do. We did not know in advance that those lines of research would fail to pan out. If you're doing research that you know will work, you're probably not really doing any research. You're just like doing a pretensive research that you can show off to a funding agency. We tried to be real. We did things where we didn't know the answer in advance. They didn't work, but that was where the hope lay, I think.

• 1:32:20Speaker 00

  But, you know, that's, but, you know, having a, having a research organization that keeps it real that way, that's not an easy thing to do. And if you don't have this very deep form of the security mindset, you will end up producing fake research and doing more harm than good. So I would not tell all the successful crypto people to, cryptocurrency people to run off and start their own research outfits. Redwood Research, I'm not sure if they can scale using more money, but, you know, you can give people more money and wait for them to figure out how to scale it later if they're the kind who won't just run off and spend it, which is what MIRI aspires to be.

• 1:33:01Ryan Sean Adams?

  And you don't think the education path is a useful path, just educating the world?

• 1:33:07Speaker 00

  I mean, I, I would, I would give myself and Mary credit for, for why the world isn't just walking blindly into the ray, whirling razor blades here. But it's not clear to me how far education scales apart from that. You can, you can get more people aware that we're walking directly into the whirling razor blades. Because even if only 10% of the people can get it, that can still be a bunch of people. But then what do they do? I don't know. Maybe they'll be able to do something later. Can you get all the people? Can you get all the politicians? Can you get the people whose job incentives are against them admitting this to be a problem? I have various friends who report like, ah yes,

• 1:33:51Speaker 00

  if you talk to researchers at OpenAI in private, they're very worried and say that they like cannot be that worried in public. Um.

• 1:34:01Ryan Sean Adams?

  This is a, all a giant Moloch trap is, is sort of what you're telling us. I, um, I feel like this is the part of the conversation where we've gotten to the end and the, and the doctor has just said that we have some sort of terminal illness. And, you know, at the end of the conversation, I think, you know, the patient, Dave and I have to ask the question, okay, doc, how long do we have? Like, seriously, what, what are we talking about here? If you turn out to be correct, are we talking about years? Are we talking about decades? Like, what's, uh, what's your idea here? What are you prepared for? Yeah.

• 1:34:38Speaker 00

  How the hell would I know? Enrico Fermi was saying that nuclear, that like fish and chain reactions were 50 years off if they could ever be done at all. Two years before he built the first nuclear pile, the Wright brothers were saying heavier than air flight was 50 years off, shortly before they built the first Wright flyer. How, how, how on earth would I know? It could be three years, it could be 15 years. We could get that, that AI winter I was hoping for, and it could be 16 years. I.

• 1:35:17Speaker 00

  I'm not really seeing 50 without some kind of giant civilizational catastrophe. And to be clear, whatever civilization arises after that could, you know, would probably, I'm guessing, end up in, stuck in just the same trap we are.

• 1:35:30Ryan Sean Adams?

  I think the other thing that the patient might do at the end of a conversation like this is also consult with other doctors. I'm kind of curious if, you know, who we should talk to on this, on this quest. Who are some people that if people in crypto want to hear more about this or learn more about this, or even we ourselves as podcasters and educators want to pursue this topic? Who are the other individuals in the AI alignment and safety space you might recommend for us to have a conversation with?

• 1:36:04Speaker 00

  Well, the person who actually holds a coherent technical view who disagrees with me is named Paul Christiano. He is, he does not write Harry Potter fanfiction and I expect to have a harder time explaining himself in concrete terms.

• 1:36:25Speaker 00

  But that is like the, the main technical voice of opposition. If you talk to other people in the effective altruism or AI alignment communities who disagree with this view, they are probably to some extent repeating back their misunderstandings of Paul Christiano's views.

• 1:36:48Speaker 00

  You could try Ajaya Kotra, who's worked pretty directly with Paul Christiano and I think sometimes expires, aspires to explain these things.

• 1:37:00Speaker 00

  That, that Paul is not the best at explaining.

• 1:37:05Speaker 00

  I'll throw out Kelsey Piper as somebody who.

• 1:37:11Speaker 00

  Would be good at explaining, like, would not claim to be like a technical person on these issues, but would but is like good at explaining the part that she does know. And who else disagrees with me? You know, I'm sure Robin Hanson would be happy to come up. Well, I'm not sure he'd be happy to come on this podcast, but you know, Robin Hanson disagrees with me. And I kind of feel like the the famous argument we had back in the like, early 2010s, late 2000s about how this would all play out. I basically feel like this was the Yudkowsky position. This is the Hanson position. And then reality was over here, like, to the well, to the Yudkowsky side of the Yudkowsky position and the Yudkowsky-Hanson debate. But Robin Hanson does not feel that way.

• 1:37:55Speaker 00

  And I would probably be happy to expound on that at length. I don't know. It's, yeah, it's not hard to find opposing viewpoints. The the ones that'll that'll stand up to a few solid minutes of cross examination from somebody who knows which parts to cross examine. That's the hard part.

• 1:38:15Ryan Sean Adams?

  You know, I've read a lot of your writings and listened to you on previous podcasts. One was in 2018 on the Sam Harris podcast. This conversation feels to me like the most dire you've ever seemed on this topic. And maybe that's not true. Maybe you've you've sort of always been this way, but it seems like the direction of your hope that we solve this issue has declined. Yeah, I'm wondering if you feel like that's the case? And if you could sort of summarize your your take on all of this as we close out this episode and offer, I guess, any thoughts, concluding thoughts here?

• 1:38:58Speaker 00

  I mean, so I don't know if you've got like a time limit on this episode? Question mark? Or is it just as long as it runs?

• 1:39:07Ryan Sean Adams?

  It's as long as it needs to be. And I feel like this is a pretty important topic. So you answer this and

• 1:39:12Speaker 00

  All right.

• 1:39:13Ryan Sean Adams?

  however you want.

• 1:39:15Speaker 00

  Well, there was a conference one time on what are we going to do about looming risk of AI disaster? And Elon Musk attended that conference. And I was like, maybe this is it. Maybe, you know, that maybe this is when the powerful people notice. And it's, you know, like one of the relatively more technical, powerful people who could be noticing this. And maybe this is where humanity finally turns and starts, you know, not quite fighting back because there isn't an external enemy here, but conducting itself with, I don't

• 1:40:00Speaker 00

  know, acting like it cares, maybe? And what came out of that conference? Well, was OpenAI, which was basically the, fairly nearly the worst possible way of doing anything. A whole, like, this is not a problem of, oh, no, what if secret elites get AI? It's that nobody knows how to build the thing. If, if we do have an alignment technique, it's going to involve running the AI with a bunch of like careful bounds on it, where you don't just like throw all the cognitive power you have at something. You have limits on the for loops.

• 1:40:41Speaker 00

  And whatever, whatever it is that that could possibly save the world, like turn all, go out and turn all the GPUs and the server clusters into Rubik's cubes or something else that prevents the world from ending when somebody else builds another AI a few weeks later. You know, anything that could do that is, is an artifact where somebody else could take it and take the bounds off the for loops and use it to destroy the world. Yeah, so like, let's open up everything, let's accelerate everything. It's, it was, it was like GPT3's version, though GPT3 didn't exist back then, but it was like ChatGPT's blind version of like throwing the ideals at a place where they were exactly the wrong ideals to solve the problem.

• 1:41:24Speaker 00

  And the problem is that demon summoning is easy and angel summoning is much harder. Open sourcing all the demon summoning circles is not the correct solution. Not even using, and I'm using Elon Musk's own terminology here. They talk about AI as summoning the demon, which, you know, not accurate, but, and then the solution was to put a demon summoning circle in every household. And why? Because his friends were calling him Luddites, once he'd expressed any concern about AI at all. So he picked a road that sounded like openness. And set, and like, and like accelerating technology. So his friends would stop calling him Luddites. It was very much the worst, you know, like maybe not the literal, actual worst possible strategy. But so very far pessimal.

• 1:42:08Speaker 00

  And that was it. That was like, that was me in 2015 going like, oh, so, so this is what humanity will elect to do. We, we, we will not rise above. We will not have more grace, not even here at the very end. So that is, you know, that, that is, uh, that is when I did my crying.

• 1:42:33Speaker 00

  Late at night. And then picked myself up and fought and fought and fought until I'd run out all the avenues that that, that I seem to have the capabilities to, to do. There's like more things, but they require scaling my efforts in a way that I've never been able to make them scale. And

• 1:42:59Speaker 00

  and there, and all of it's pretty farfetched at this point anyways.

• 1:43:04Speaker 00

  So, you know, that, that, so what's, you know, what's changed over the years? Well, first of all, I ran out some remaining avenues of hope. And second, things got to be such a disaster, such a visible disaster. The AIs got powerful enough and it became clear enough that, you know, we do not know how to align these things, that I could actually say what I'd been thinking for a while and not just have people go completely like, what are you saying about all this? You know, now, now the stuff that was, that was obvious back in 2015 is, you know, starting to become visible and distant to others and not just like completely invisible. That's what changed over time.

• 1:43:50Ryan Sean Adams?

  What kind of, um, what do you hope people hear out of this episode? And out of, out of your comments, the liaison in, in 2023, who is sort of running on the last fumes of, of hope?

• 1:44:09Ryan Sean Adams?

  Yeah. What do you, what do you want people to get out of this episode? What like, what are you planning to do?

• 1:44:17Speaker 00

  I, I don't have concrete hopes here.

• 1:44:21Speaker 00

  You know, when everything is in ruins, you might as well speak the truth, right? Maybe somebody hears it, somebody figures out something I didn't think of. I mostly expect that this does more harm than good in the modal universe. Cause a bunch of people are like, oh, I have this brilliant clever idea, which is, you know, like something that somebody that, you know, I was arguing against in 2003 or whatever. But, you know, maybe, maybe there, maybe somebody out there with the proper level of pessimism hears and thinks of something I didn't think of.

• 1:44:56Speaker 00

  I suspect that if there's hope at all, it comes from a technical solution because the difference between technical problems and political problems is at least the technical problems have solutions in principle. At least the technical problems are solvable. We're not on course to solve this one, but I don't really see the... I think anybody who's hoping for a political solution has frankly not understood the technical problem. They do not understand what it looks like to try to solve the political problem to such a degree that the world is not controlled by AI because they don't understand how easy it is to destroy the world with AI, given that the clock keeps ticking forward. They're thinking that they just have to solve, stop some bad actor, and that's why they think there's a political problem or a political solution. But yeah, I don't have concrete hopes.

• 1:45:39Speaker 00

  I didn't come on this episode out of any concrete hope. I have no takeaways, except like, don't make this thing worse. Don't, don't, don't like go off and accelerate AI more. Don't, if you have a brilliant solution to alignment, don't be like, ah, yes, I have solved the whole problem. We just use the following clever trick.

• 1:46:05Speaker 00

  Don't, you know, don't make things worse than very much of a messes, especially when you're pointing people at the field at all. But there aren't, I have no winning strategy. Might as well go on this podcast and say what I, yeah, as an experiment and say what I think and see what happens and probably no good ever comes of it. But, you know, you might as well go down fighting, right? If there's a world that survives, maybe it's a world that survives because of a bright idea somebody had after listening to this podcast. That was brighter, to be clear, than the usual run of bright ideas that don't work.

• 1:46:41Ryan Sean Adams?

  Eliza, I want to thank you for coming on and talking to us today. I do, I don't know if, by the way, you've seen that movie that David was referencing earlier, the movie Don't Look Up, but... I sort of feel like that that news anchor who's talking to like the scientists is Leonardo DiCaprio, David, and the scientist is talking about kind of dire straits to the world. And the news anchor just really just doesn't know what to do. I'm almost at a loss for words at this point. But what one thing

• 1:47:12Speaker 00

  I've had nothing for a while.

• 1:47:13Ryan Sean Adams?

  One thing I can say is I appreciate your honesty. I appreciate that you've given this a lot of time and given this a lot of thought. Everyone, anyone who has heard you speak or read anything you've written knows that you care deeply about this issue and have given it a tremendous amount of your life force in trying to educate people about it. And thanks for taking the time to do that again today. I guess I'll just let the the audience digest this episode in the best way they know how. But I want to reflect everybody in crypto and everybody listening to Bankless, their thanks for you coming on and explaining.

• 1:47:56Speaker 00

  Thanks for having me. We'll see what comes of it.

• 1:47:59Ryan Sean Adams?

  Action items for you, Bankless Nation. We always end with some action items. Not really sure where to refer folks to today. But one thing I know we can refer folks to is MIRI, which is the Machine Research Intelligence Institution that Eliezer has been talking about through this episode. That is at intelligence.org, I believe. And, you know, some people in crypto have donated funds to this in the past. Vitalik Buterin is one of them. You could take a look at what they're doing as well. That might be an action item for the end of this episode. Gotta end with risks and disclaimers. Man, this seems very

• 1:48:39Ryan Sean Adams?

  trite, but our legal experts have asked us to say these at the end of every episode. Crypto is risky. You could lose everything.

• 1:48:49Speaker 00

  Apparently not as risky as AI, though.

• 1:48:52Ryan Sean Adams?

  But we're headed west. This is the frontier. It's not for everyone, but we're glad you're with us on the Bankless journey. Thanks a lot.

• 1:48:59Speaker 00

  And we are grateful for the crypto community's support. Like, it was possible to end with even less grace than this.

• 1:49:07Ryan Sean Adams?

  Wow.

• 1:49:09Speaker 00

  You made a difference.

• 1:49:11Ryan Sean Adams?

  We appreciate you.

• 1:49:12Speaker 00

  You really made a difference.

• 1:49:13Ryan Sean Adams?

  Thank you.