168 - How to Solve AI Alignment with Paul Christiano

Paul Christiano runs the Alignment Research Center, a non-profit research organization whose mission is to align future machine learning systems with human interests. Paul previously ran the language model alignment team at OpenAI, the creators of ChatGPT. Today, we’re hoping to explore the solution-landscape to the AI Alignment problem, and hoping Paul can guide us on that journey. ------ ✨ DEBRIEF | Unpacking the episode: https://www.bankless.com/debrief-paul-christiano ------ ✨ COLLECTIBLES | Collect this episode: https://collectibles.bankless.com/mint ------ ✨ Always wanted to become a Token Analyst? Bankless Citizens get exclusive access to Token Hub. Join Them. https://bankless.cc/TokenHubRSS ------ In today’s episode, Paul answers many questions, but the overarching ones are: 1) How BIG is the AI Alignment problem? 2) How HARD is the AI Alighment problem? 3) How SOLVABLE is the AI Alignment problem? Does humanity have a chance? Tune in to hear Paul’s thoughts. ------ BANKLESS SPONSOR TOOLS: ⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum 🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://bankless.cc/kraken 🦄UNISWAP | ON-CHAIN MARKETPLACE https://bankless.cc/uniswap 👻 PHANTOM | FRIENDLY MULTICHAIN WALLET https://bankless.cc/phantom-waitlist 🦊METAMASK LEARN | HELPFUL WEB3 RESOURCE https://bankless.cc/MetaMask ------ Topics Covered 0:00 Intro 9:20 Percentage Likelihood of Death by AI 11:24 Timing 19:15 Chimps to Human Jump 21:55 Thoughts on ChatGPT 27:51 LLMs & AGI 32:49 Time to React? 38:29 AI Takeover 41:51 AI Agency 49:35 Loopholes 51:14 Training AIs to Be Honest 58:00 Psychology 59:36 How Solvable Is the AI Alignment Problem? 1:03:48 The Technical Solutions (Scalable Oversight) 1:16:14 Training AIs to be Bad?! 1:18:22 More Solutions 1:21:36 Stabby AIs 1:26:03 Public vs. Private (Lab) AIs 1:28:31 Inside Neural Nets 1:32:11 4th Solution 1:35:00 Manpower & Funding 1:38:15 Pause AI? 1:43:29 Resources & Education on AI Safety 1:46:13 Talent 1:49:00 Paul’s Day Job 1:50:15 Nobel Prize 1:52:35 Treating AIs with Respect 1:53:41 Uptopia Scenario 1:55:50 Closing & Disclaimers ------ Resources: Alignment Research Center https://www.alignment.org/ Paul Christiano’s Website https://paulfchristiano.com/ai/ ----- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The most likely way we die involves like not AI comes out of the blue and kills everyone, but involves we have deployed a lot of AI everywhere. And you can kind of just look and be like, oh yeah, if for if for some reason, God forbid, all these AI systems were trying to kill us, they would definitely kill us.

• 0:14Ryan Sean Adams?

  Welcome to Bankless, where we explore frontier technologies, artificial intelligence on the episode today. This is how to get started, how to get better, and how to front run the opportunity. This is Ryan Sean Adams. I'm here with David Hoffman, and we're here to help you become more bankless. Guys, we have a special guest in the episode today, Paul Cristiano. This is who Eliezer Yudkowsky told us to go talk to, someone he respects on the AI debate. So we picked his brain. This is an AI safety alignment researcher. And we asked the question, how we stop the AIs from killing us? Can we prevent the AI takeover that others are very concerned about? There are three, actually four takeaways for you today. Number one, how big is the AI alignment problem? We ask Paul this question.

• 0:58Ryan Sean Adams?

  Number two, how hard is it to actually solve this problem? Number three, what are the ways we solve it? The technical ways? Can we coordinate around this to solve it? And finally, we talk about a possible optimistic scenario where we live in harmony with the AIs, and they improve our lives and make it quite a bit better. David, what was the significance of this episode to you in our series?

• 1:22David Hoffman?

  Yeah, the the new intro, I think, is great when we cover frontier technologies. And also, instead of for this episode, helping you become more bankless. I think this is a truly a front running the opportunity. On this one, we are trying to help humanity not die. We're trying to help you not die.

• 1:42Ryan Sean Adams?

  All of us not die.

• 1:43David Hoffman?

  All of us, yeah, you and the rest of the world. And I think this, we're doing this in the best of the way that we can, which is education and awareness about this AI problem. Paul Cristiano is, like you said, the man that was recommended by Eliezer about the man approaching this problem head on in a technical way. So in this podcast, you are going to hear about the technical solutions that people are actively working on, who are taking this problem extremely Seriously and take the risks very, very seriously. Eliezer gave us more or less a doomsday scenario, like a 99% chance of doom. Paul Cristiano only gives us a 10 to 20% case of doom scenario, so much more optimistic.

• 2:26Ryan Sean Adams?

  Like those odds.

• 2:27David Hoffman?

  Much better odds. And so we go, we go through why he's still very much concerned and he does consider this the most likely way in which he dies in the future. Yet why there's still an 80% chance of success. And some of that 80% chance of success actually does have utopia in it. I think maybe Ryan, in three, five, 10 years, we're going to be able to look back at this episode. As hopefully, if Paul's right, and I think he's right, like ahead of its time in terms of elevating extremely important conversations to the best of our ability into the mainstream, so we can get more people to focus on the actual solution paths. To make sure that that 20% risk of doomsday goes down to 0.2% risk of doomsday.

• 3:10David Hoffman?

  And I see that's what the significance of this episode has and why we are doing episodes like this.

• 3:15Ryan Sean Adams?

  Yeah, I mean, to be clear, Paul really thinks this is AI alignment is a solvable problem, which is much, much different than, than others in the space. And he tells us exactly why. And my takeaway from the Eliezer episode was humanity is screwed. This was humanity is screwed, but we're working on it. We might have some solutions.

• 3:31David Hoffman?

  And we have clear, actionable paths.

• 3:34Ryan Sean Adams?

  That's right. So we get into all of that today. And David, of course, I want to discuss this episode in a debrief with you and hear what you think, because this is our third in a series of AI episodes. And really interesting material. The debrief episode is our episode that we record directly after the episode with our raw, unfiltered thoughts. If you are a bankless citizen, you have access to that right now on the premium RSS feed. You can click a link in the show notes and get access to that. Okay, guys, we're going to get right to the episode with Paul. But before we do, we want to thank the sponsors that made this episode possible, including Kraken, our recommended crypto exchange for 2023.

• 4:10David Hoffman?

  Kraken has been a leader in the crypto industry for the last 12 years. Dedicated to accelerating the global adoption of crypto, Kraken puts an emphasis on security, transparency, and client support, which is why over 9 million clients have come to love Kraken's products. Whether you're a beginner or a pro, the Kraken UX is simple, intuitive, and frictionless, making the Kraken app a great place for all to get involved and learn about crypto. For those with experience, the redesigned Kraken Pro app and web experience is completely customizable to your trading needs, integrating key trading features into one seamless interface. Kraken has a 24/7, 365 client support team that is globally recognized. Kraken support is available wherever, whenever you need them, by phone, chat, or email. And for all of you NFT-ers out there,

• 4:54David Hoffman?

  the brand-new Kraken NFT beta platform gives you the best NFT trading experience possible: rarity rankings, no gas fees, and the ability to buy an NFT straight with cash. Does your crypto exchange prioritize its customers the way that Kraken does? And if not, sign up with Kraken at kraken.com/bankless. Bankless is launching the Bankless Token Hub. At Bankless, we've been studying the crypto markets ever since 2017, and all of our research has led us to this: the Token Hub. You're a one-stop shop for alpha to help you navigate through the crypto markets. Have you ever wished for a trusted resource that would share their thoughts, ratings, and their opinions about tokens? Boy, do we have the product for you. The Bankless Token Hub is where we provide Bankless citizens with the alpha on the hottest tokens in crypto. We do the research, so you don't have to.

• 5:38David Hoffman?

  The Bankless Token Hub includes the Token Ratings, where our team shares their research and outlook on the hottest tokens in crypto. Also, the Token Hub includes Bankless Bags, our own internal investment club. Bankless Bags is where we put our money where our mouth is. And for the Bankless power user out there, you can access the analyst team 24/7 inside the Bankless Nation Discord. You can ask them questions and learn from a group of people deep in the weeds of crypto investing. The last feature of the Token Hub is the ability to upvote or downvote token ratings. The Bankless Token Hub lets you learn from your fellow citizens to rate these tokens yourselves. The Bankless Token Hub is launching right now and has already been beta-tested by your fellow Bankless citizens. So stay tuned in the Bankless Discord for upvotes.

• 6:20David Hoffman?

  And if you're not a Bankless citizen, well, you better sign up if you want access, because this corner of Bankless is available for citizens only. I'll see you in the Discord. If you haven't yet experienced the superpowers that a smart contract wallet gives you, check out Ambire. Ambire works with all the EVM chains, the Layer 2s like Arbitrum, Optimism, and Polygon, but also the non-Ethereum ecosystems like Avalanche and Phantom. Ambire lets you pay for gas and stablecoins, meaning you'll never have to spend your precious ETH again. And if you like self custody, but you still want training wheels, you can recover a lost Ambire wallet with an email and password, but without giving the Ambire team control over your funds. The Ambire wallet is coming soon for both iOS and Android. And if you want to be a beta tester, Ambire is airdropping their wallet token for simply just using the wallet.

• 7:02David Hoffman?

  You can sign up at ambire.com, and while you're there, sign up for the web app wallet experience as well. So thank you, Ambire, for pushing the frontier of smart contract wallets on Ethereum.

• 7:11Ryan Sean Adams?

  Bankless Nation, I am super excited to introduce you to our next guest. We're talking about AI alignment stuff here today because we can't not. This is Paul Cristiano. He runs the Alignment Research Center, which is a nonprofit research organization whose mission is to align future machine learning systems with human interests. Make sure the AIs don't come to kill us. That's what I take to be the meaning. And Paul previously ran the Language Model Alignment Team at OpenAI. You know them. They're the creators of ChatGPT. And today, we're hoping that Paul can help us explain the solution landscape and understand the solution landscape of this AI alignment problem. Paul, welcome to Bankless. Thanks for having me. Excited to talk to you. So Paul, just to get some context here, Dave and I recorded an episode with Eliezer

• 7:56Ryan Sean Adams?

  Yukowski. We thought this would be, hey, you know, Bankless's first intro. We're primarily a crypto podcast, but we're exploring other frontier technologies. Let's go check out-

• 8:04David Hoffman?

  Go dabble with AI.

• 8:05Ryan Sean Adams?

  Let's just go dabble with AI. You know, crypto and AI might have some sort of match in the future. So we recorded this podcast, and we quickly realized the agenda that we were going to use and talk about didn't matter anymore, because Eliezer's message was pretty simple. We were all going to die. Basically, we were on the brink, whether it's years or months away, from creating some super- Intelligent AI that would eventually rearrange humanity's atoms and destroy us. And he felt pretty convicted on this as the likely likely outcome. So when you get a message like that, Paul, you got to investigate a little further. You have to get the second doctor's opinion when the prognosis is terminal.

• 8:50Ryan Sean Adams?

  So that's what this series is all about. And we're hoping you can help guide us through these questions today. Does that sound OK?

• 8:57Speaker 00

  Happy to at least share my thoughts. I'm a little bit less gloomy.

• 9:01Ryan Sean Adams?

  Yes, OK, OK, I suspected so. By the way, Eli Ezer said, we asked him, hey, is there someone else we can talk to about this? And he mentioned you, he said, talk to Paul Cristiano. He said, you are someone he respects and brings some of the counterpoints to his take. So let's get into them. Why don't we just start by wading into the deep end of the pool here? What is your percentage likelihood of the full out Eli Ezer Udkowsky doom scenario? Where we're all going to die from the machines?

• 9:33Speaker 00

  I think this question is a little bit complicated, unfortunately, because there are a lot of different ways we could all die from the machines. So the thing I most think about, and I think Eli Ezer most talks about is the sort of full blown AI takeover scenario.

• 9:48Speaker 00

  I take this pretty seriously. I think I have a much higher probability than a typical person working in ML. I think maybe there's something like a 10, 20% chance of

• 10:00Speaker 00

  AI takeover, many, most humans dead.

• 10:05Ryan Sean Adams?

  Really high.

• 10:06Speaker 00

  I agree. And then I I think beyond that.

• 10:08Ryan Sean Adams?

  Better than 100, David, just better than 100%.

• 10:10David Hoffman?

  Better, we're trending in the right direction.

• 10:13Speaker 00

  I think there are other, yeah, I think in some sense, I'm still quite a gloomy person. So there's other ways that the development of AI can be rough. Like there's other ways that you can have access to new destructive physical technology, other disruption. So I think you're maybe looking at some other risks from that transition to AI. And that adds up to at least another 10%.

• 10:34Speaker 00

  And then maybe a bigger background part of both my view and Eli Ezer's view. I think Eli Ezer is into this extremely fast transformation. Once you develop AI, I am a little bit. Have a little bit less of an extreme view on that, but I still think it is the case that compared to what this kind of default expectations in the world, things are going to be really fast. So we could talk about the development of AI, but then you might also want to talk about what happens right over the coming months or years. I tend to imagine something more like a year's transition from AI systems that are a pretty big deal to kind of accelerating change, followed by further acceleration, et cetera. I think once you have that view, then sort of a lot of things may feel like AI problems because they happen very shortly after you build AI. Like your AI builds new AI systems, your site keeps changing. Anyway,

• 11:17Speaker 00

  so overall, you know, maybe you're getting more up to like 50/50 chance of doom shortly after you have AI systems that are at human level, but.

• 11:24David Hoffman?

  Okay. Okay. All right. Well, um, let's, let's start with that speed conversation first. Let's start with the, the, the takeoff velocity question because I think that's something to, that really the AI alignment, uh, Dumaism's, uh, uh, perception really, really depends on, uh, if, if we do believe that AIs are going to be developed and they're magically going to become sentient, not magically, but somehow pragmatically. It feels like magic. It's, it's, it feels like magic to humans. Uh, it's going to be because it happens really fast. And I want to actually try and measure that speed because fast and slow is like relative, right? And so the, the takeoff scenario that, uh, some AGI super, uh, super

• 12:07David Hoffman?

  intelligence explosions, uh, people articulate is that as soon as, uh, some sort of, uh, automatic, uh, uh, some sort of AI can update itself, it's like a lightning flash. It's like a snap of the fingers. It happens in a blink of an eye. One day we have chat GPT-7 and then the next day we have an AI takeover. And that's like the super fast scenario. I think what you're saying is like, yeah, pretty quickly, but still not like lightning fast. I think you're, what you're saying is like, give it a year. Can you, maybe you can help unpack like the timing. How, how do we understand about time around this whole thing?

• 12:42Speaker 00

  Yeah, this is one of my most pronounced disagreements with Eli Azar, um, where we've really went back and forth about it a lot over the last, like, Jesus, I don't know, 12 years. Um, and I think still do not see at all eye tie. That said, my view is pretty fast. So I think like, how I would think about this, maybe there's like kind of two parts to my answer. So one is sort of based on how fast things are currently moving in AI. So the way you could think about it is if you were to try and measure, right, if you're trying to say, suppose you have AI doing some job and has some level of competence at that job this year. How good is it going to be at this job next year? And how fast is that changing? Right. So if we're looking at the world and we're like every day AI is much smarter than the day before, then you kind of are going to expect by default a fast transition over scale of something like days.

• 13:26Speaker 00

  Because there are going to be AI systems that are weak and then a couple of days later you have AI systems that are strong. I think the way I would describe the situation now is more like time scales like a year or a couple of years. And then there's some reasons that the, well, we, we give a quantitative number. It depends a lot on what we're talking about a number for. So like one way you could measure how fast AI is moving is you could say, suppose you have like an AI from year X and an AI from year X plus one. And you're wondering like, how much better is the year AI from year X plus one? In the sense of like, how many year X AIs would you have needed to be comparably useful? Like how many, having AI from one year later is kind of like having twice as many computers or having four times as many computers or something like that. I think that's typically the regime we're in.

• 14:08Speaker 00

  So like one year of AI progress is kind of similar to increasing the amount of compute you have by something like 4X. From a combination of like hardware progress, software progress, economies of scale, maybe more. I think you might get 8X, you might go down to 2X, but it's like something in this regime. You have like one to a couple of doublings per year.

• 14:28Speaker 00

  So like, right now, it doesn't really matter that much. Doubling the amount of computers you have has very little effect on the world. Like if we doubled the number of computers in the world today, you would not even notice in like GDP statistics. Yeah, you wouldn't notice basically. I think in the future, there are going to be AI systems that are doing some large fraction of all the work in the world. And ultimately they can like substitute effectively for humans across many domains. And then doubling the number of your computers you have is kind of like doubling the effective population size, like doubling how many people are working as researchers, doubling how many people are doing jobs. And so in that world, if you say you're doubling the number of computers you have effectively every like four to six months, that does imply a very rapid rate of change in kind of how quickly science is progressing and sort of how much stuff you're able to accomplish in the world.

• 15:11Speaker 00

  And so I kind of think of that as this like first transition. As you move into this world where AI can substitute for humans is you're, you're looking at a rate of growth, something like doubling a couple of times a year in total output of AI systems. The main thing that softens this, so we can talk about how fast the transition, what that actually translates into in terms of how fast the transition is in the world. I think there's one important consideration that softens that that change, which is that you have some complementarity between AI systems and humans. That is, AI are good at some things, humans are good at other things. So as a result, like the things that AIs are good at, you tend to hit diminishing returns on those, so that the transition is like a little bit slower than you would guess. If AIs and humans were perfect substitutes, I think you'd be looking at a transition over like something like 12 months from a world where like humans are doing almost everything, humans are doing almost nothing.

• 15:56Speaker 00

  I think, given some complementarity, you're probably looking at a transition over more like years. I think it's like kind of hard to run the numbers to get a transition over decades. I think a lot of people say that and have a strong intuition in that direction. But like, when the discussion gets down to brass tacks, I currently don't really see how to make it work. I think it is plausible to get up to like, yeah, very low decades, like, you know, having, anyway, we have to talk about timeline between what and what. But I think we're mostly talking about like years. I think months would be pretty surprising, but possible. Decades would be also pretty surprising, but possible.

• 16:26Ryan Sean Adams?

  So just, Paul, to get people up to speed on your 12 years of debate with Eliezer and those who hold this viewpoint. So does Eliezer think in terms of like, minutes or days that this could happen? And you're saying, not that fast, it's closer to years? Is that the difference? And then, once you answer that, can you tell us, why does this matter so much, whether it happens in minutes or days versus, you know, years and decades? Why is that such a fulcrum of this whole debate and discussion on why the AIs will come kill us, or whether they will, or whether we'll be okay?

• 17:02Speaker 00

  I think probably it is harder to describe Eliezer's views quantitatively in terms of rates of change, because more of his view is about this like kind of phase transition that happens quite quickly. Like, it's not reasonable to talk, another aspect, it's not reasonable to talk about this framing of like, how long does it take to like double the population size or something? It's more like, how long does it take to move from like chimps who are doing nothing to humans who are doing a lot of stuff? And he's like, that, I don't know, it could just randomly happen one day that someone like tweaks their code and that went from being a chimp to being a, being a human. And that's pretty transformative. I think he sort of just has a broader distribution. I think he does not find years out of the question. He's just like, that's kind of the tale of how slow it could be or something like that. And like, it's more about this qualitative picture. He's just like, you're sort of not going

• 17:44Speaker 00

  to have changes in the world. Like, in some sense, the more important thing is I imagine AI systems acting in the world, doing like trillions of dollars of economic value prior to getting to this point, where they're actually causing like this potential catastrophic risk, or where they're significantly or totally transforming the pace of future technological change. I think Eliezer imagines more like you move from a world like the world of today, where AIs are doing maybe billions, or tens of billions, or hundreds of billions of dollars of value. I think that feels to me like the core distinction is kind of like, where are you starting from? And I'm like, more starting from a world with trillions or tens of trillions. And Eliezer is more starting from a world of like, I mean, 12 years ago, I think this is probably going to seem unfair to Eliezer, but this discussion was like, maybe more alive. And Eliezer would frequently talk about like,

• 18:27Speaker 00

  maybe some random people in a small AI group somewhere, like a company like DeepMind's doing like $100 million a year of spending is going to be building transformative AI. I think my basic take was like, no way, you're going to look at AI systems that are doing trillions of dollars of revenue. And like, this gap is closing pretty rapidly, because now no one's going to say, like, you're doing $100 million of revenue. Like, it's pretty clear you're going to be in the least like billions or tens of billions of dollars. I think my take is just like, pretty soon, it's going to be pretty clear, doing 100 billion, then it's going to be like, yeah, I think we're kind of just debating, like, what is the point that you jump from? Like, when you get to the AI, that's like crazy science stuff. What was happening like six months before that? Was what was happening, like, AI systems really broadly deployed in the world, doing a ton of crazy things? Or was happening, like, actually, the impact of AI systems was pretty limited until right before you kind of have this process of rapidly accelerating

• 19:11Speaker 00

  R and D, recursive self-improvement within like a single firm or in a like local part of the world?

• 19:15Ryan Sean Adams?

  But to be clear, Paul, do you think it's possible, or more unlikely than Eliezer thinks it is, to go from that, you know, that big hop, software update one day, to move from chimps to to human-level intelligence? Do you think that's unlikely? And do you have kind of technical grounds for this? Or what are your grounds for believing that that's less likely than others do?

• 19:36Speaker 00

  Yeah, I think there's two parts of this. I mean, again, I want to emphasize that compared to most people in the world, I think I'm into much, much faster change. I think the mainstream view in ML is that things will be more gradual, which I think is mistaken. And when you get down, again, we get down to brass tacks, I'm really unpersuaded. You could then talk about like my view, you could have views that are quantitatively faster than mine. It's just like, actually, this isn't years, this is months. I think that's like, it's very defensible to run the historical extrapolations and end up with different numbers. That's just like a really hard empirical question. And it's like, hard to make these predictions about the future. And I have a lot of sympathy for that. Then I think there's more like this qualitative claim. It's like the chimp versus human jump. I think that's not out of the question, but feels quite unlikely. I think the basic reason it feels quite unlikely to me is like,

• 20:16Speaker 00

  that is really not how, I would claim that's really not how anything has worked in AI to date. And it's not how things have worked in almost any other technologies. Like it has, it has mostly been the case, this is my read of history, and I'm very happy to argue about. I think this is like a, an important part of the argument with Elias. My read of history is mostly that before you can do something really crazy, you can do something that like works a little bit less well, and like is a little bit crappier. And you do sometimes have these jumps from like zero to one. But you tend to see the zero to one jump, not when a technology is worth like, this would allow you to take over the world, or this would be worth $10 trillion. You tend to see zero to one jumps when a technology is like, you have a bunch of amateurs, or hobbyists, or a couple scientists. And so I think like, if we're gonna see such a jump in AI, I'd be more likely to have seen it back when we were talking about like, a small academic community, and you're less likely to see it when you have academic, or like labs investing billions or tens of billions of dollars. I

• 21:01Speaker 00

  think the general record is like, most of the time before you can do something really crazy, you can do something slightly less crazy. And that becomes like a more and more robust regularity, as you increase the number of people thinking about something, and increase the amount of attention. You move more and more towards like industries with reasonable roadmaps that actually are forecast for what's gonna happen.

• 21:15Ryan Sean Adams?

  Is there, is there an example that you, you call to mind for, for history that sort of, we can compare this to?

• 21:22Speaker 00

  I mean, I think, I'm happy to compare it to sort of almost any technology. That is like, they are all different, different ways. I don't know if this, I think Elias is probably more like wanting to point to a particular thing. He's like, this is the really relevant one. But I would say like, to me, AI seems kind of similar to like, future AI developments seem kind of similar to either past AI developments, other developments in software. I'd be happy to talk about computing hardware, or solar power, or nuclear power, or nuclear weapons, or flights.

• 21:49Ryan Sean Adams?

  You just think they all, they all take this kind of gradual sort of approach, rather than the big zero to one moments. And I, but a part of the reason, Paul, people are asking about this is because I think it seems like, and you tell us, so you previously have worked at OpenAI, very familiar with the methodologies used, but it feels to some people like ChatGPT has been a big zero to one moment, right? Like, my god, it's amazing. And people are, you know, tinkering with it in so many ways, and how human-like it seems, and how fast it seemed to explode into the popular consciousness. And so I'm wondering if that has affected your view on this at all. It's like, oh, wow, this could happen faster than I'd previously thought? Or this is well within the bounds of your model?

• 22:34Ryan Sean Adams?

  Like, what are we to make of ChatGPT?

• 22:37Speaker 00

  So I think that ChatGPT seems, I would take, as like representative of the kind of trajectory I'd expect. So you could compare like ChatGPT versus GPT 3.5 versus GPT 3, GPT 2. And like, I think like people at OpenAI were not, like, it's kind of a most sociological fact about ChatGPT getting to the point where it was discussed a lot. I think the actual technical change, certainly between ChatGPT and GPT 3.5, but also between 3.5 and 3, like each of these is not giant jumps. I think these were like pretty small changes. And like ChatGPT is not, I think it is at the point where it's like economically valuable, and it like is worth a lot. I think that like you're seeing, I think people are mostly excited because they're looking ahead to where this will go. And that's like a lot of what makes these dynamics like more continuous.

• 23:22Speaker 00

  People are starting to say like, okay, what can we do with this? I think they're doing that at a point where like, they're not going to be able to do trillions of dollars a year of value, but they are seeing that that is going to become possible at some point in the not distant future, as the technology continues to improve. It's like very concretely, like, I mean, I know, we were having these discussions, I was having discussions quite a lot, like, prior to the training of GPT 2. It's like, prior to the training of GPT 2, we didn't really have language models that...

• 23:46Speaker 00

  I don't think we have language models you would even recognize as seeming smart. It kind of felt like a qualitatively different ballgame. And maybe most relevant is like after the training GPT two before GPT three or before the scale up from like one B to six B.

• 23:58Speaker 00

  I think we like, you know, we sat down and we made predictions about how good large models would be. And I think I am surprised by how good models are, but like surprised in the sense that this is like maybe my eightieth percentile of how smart a language model at the scale of GPT three point five would be.

• 24:12Speaker 00

  Something like that is a little bit hard to exactly compare my forecast to where reality is at, but we did discuss explicitly for like a language model of the size of GPT three point five, trained in roughly the way GPT three point five was. We weren't exactly right because some of the scaling laws, like there's the switch from GPT scaling laws to Chilis scaling laws, but like roughly speaking, we could, we were imagining systems at that scale, trained in that way. And we were like, you know, an example of a bet would be like, is there any task a human can do over 30 seconds that a system trained in this way can't do over 30 seconds? Like, is there any 30 second Turing test that can distinguish a human from an AI? And I think like, you know, the debate was kind of like amongst people I was most talking to at OpenAI. I was like, ah, probably there will be, but it's not a sure thing, like a one third chance, you know, that there will be no such tests or something like that. And like, I think that things are mostly like, that was not from like, it wasn't a big jump.

• 24:56Speaker 00

  It was just like, kind of see the writing on the wall and like see systems improving in this way. And like, it's going to get harder and harder to tell. There's more and more things they can do.

• 25:02Ryan Sean Adams?

  So are you saying, Paul, that for society, this seemed to, you know, come out of nowhere, basically? But it is not really surprising that the researchers and the engineers who've been on the inside at OpenAI and constructing chat GPT four. It was maybe within the bounds of expectation. It was, you know, maybe you're more optimistic, or I'll use the term bullish, because like we, we talk about bullishness in crypto all the time. It's more bullish than, than you thought this technology would, would sort of take you, but still within the realms. And the only reason it's having such a, an effect on our collective consciousness is, is more sociological. It's just because.

• 25:40Speaker 00

  It turned into a consumer application.

• 25:41Ryan Sean Adams?

  Yeah, it's just suddenly turned into a consumer app and everyone's like, wow, I can type whatever I want into this magic box and a genie oracle. Artificial intelligence just gives me the answer. This is incredible. Like, is anyone surprised by this who's been working on this tech?

• 25:57Speaker 00

  Oh, I mean, I think, yes, this is a nuanced question. There's like a lot to say. I don't know if we want to get into all of it. But like, I think at the point when GPT two was trained, this was a controversial prediction. So like there were people who were more bullish than I was, like EG, Dario, who now runs Star Amade, who now runs Anthropic, was like at the time of GPT-2, like very bullish.

• 26:13Ryan Sean Adams?

  And in timeline here, Paul, so GPT-2 was when? What year?

• 26:17Speaker 00

  Oh, man, I don't even know if I remember. I think these discussions were like 2018.

• 26:20Ryan Sean Adams?

  Got it. Okay. Go on. Yes.

• 26:25Speaker 00

  But like, yeah, so Dario was more optimistic. I think this world was actually like slightly below Dario's median of how impressive a system like GPT, ChatGPT would be. I mean, not a huge amount below. I think he made like quite a good forecast and it's kind of been his big bid win. I think there were a bunch of people who kind of had views in this general cluster, like for whom this is like a little bit better than what they would have thought. Or like, you know, at the 80th percentile or something of what they might have expected. I think there were a lot of people who like weren't in the business of making forecasts, but seemed qualitatively, like if you look to the academic ML community, like it felt to me like people were not expecting this to happen, like discourse about general AIs felt like very frustrating, I think. Where they're like, why is it the case that people are really minimizing the possibility of just large neural nets trained in a very simple way?

• 27:09Speaker 00

  Being competitive with humans.

• 27:13Speaker 00

  I think it was surprising to like a lot of people. And again, it was just quantitatively, right? It was surprising to me in the sense that this is like better than I thought and I like lost bets about this. I mean, I won some bets with people. I lost some bets with people. I think some people weren't quantifying their probabilities. Maybe this was like more just totally out of model. But by the time you're talking about ChatGPT compared to what came before, I do not think, I think at that point to people building the system, it was not surprising.

• 27:37Speaker 00

  Like once you're talking about the gap, even from three to 3.5 and then from 3.5 to ChatGPT, they're more, they were probably more surprised by the way, like the impact it had on collective consciousness, rather than by the capabilities of the system itself. I think that seemed like pretty technically derisked.

• 27:51Ryan Sean Adams?

  And Paul, when we're talking about sort of AI alignment and safety concerns and just on the topic still of ChatGPT, right? Are these neural nets, these large language models, the ones we have to worry about? Like, I think what basically society is sort of wondering as this, this AI alignment, safety question rises into public consciousness is, okay, at what version do we have to start worrying that? Chat GPT is going to pose a threat to us. Like, there's some version where it starts to take our jobs. Okay. And then like, maybe that's version four, version five. And so it affects our economy and economics. And we have to reorient, restructure society as a result of this. But like, it seemed to be what Eliezer was saying is, well, maybe it might be version nine

• 28:35Ryan Sean Adams?

  or version 10 or version 11, where we actually have to fear for our lives from this thing because it's become super intelligent. Do you see that possible trajectory for this specific large language AI technology? Or should we have more concerns about other AI technology that's, that's coming in, you know, some other vector of development?

• 28:56David Hoffman?

  Actually, I'd like to phrase that question slightly differently. And maybe in a way that bankless listeners, a metaphor that bankless listeners can understand. Often, Paul, when we talk to people that are outside of the crypto industry, they just call the like, things about the crypto industry Bitcoin. And then when Ryan and I hear that, they're, we were like, oh, what they really mean is like, decentralized technology, like identity. They just use Bitcoin as a placeholder to talk about so many different things.

• 29:23Ryan Sean Adams?

  It's so frustrating.

• 29:23David Hoffman?

  And I think as like, AI normies, me and Ryan, AI normies out there, we might be saying Chat, Chat GPT. And what we actually are talking, trying to talk about is like generalized artificial intelligence. And we just use Chat GPT because that's the thing. It's the Bitcoin of AI. Is that, are we falling into that same trap?

• 29:43Speaker 00

  I definitely think there's a lot of subtlety here. And there's a lot of different ways I could interpret like this thing. So I'd say that like, one question is like, what are you modeling? Like, what is the AI learning to predict? Like, is it videos or is it text? Or is it like interactions with code, like running code and the results of running code? Or like the code humans would write? And I think like, over the last couple of years, I think those distinctions have mattered a lot less. Like, what is mostly, I think the default model probably system should work is just you have quite a lot of data of quite a lot of types. And you just dump it all in and you say like, look, your job AI is to deal as well as you can with every type of data we give you. And there's engineering problems and allowing those different types of data. And then there's questions about what types of data the system is actually able to effectively deal with. But I think the fact that it's trained on language is like not. I just don't think you should think of it as a defining feature.

• 30:28Speaker 00

  I think you should imagine systems like see the world. I think language is probably an important way of thinking about how they act. Although again, I think it's very impressive. Systems can act by producing images and those will be very impressive and will have a big impact. I think economically in some sense, language is like a very flexible and like kind of core way you should think about systems acting, but perceiving. I don't think you should really think about language models in particular.

• 30:47Speaker 00

  GPT itself, just saying GPT, this is basically just fixing like there's two things that specifies. One is how is it trained? Is it trained to predict data or is it pre-trained to predict data? Was it trained in some other way? That's the first distinction. And the second is just that it's a transformer. I don't know if anyone wants to make like a super strong bet about like transformers per se. I think they're just like our different art. Yeah, there's a big space of possible architectures. There's probably going to be debate about some things where you'd call them a transformer.

• 31:13Speaker 00

  I think it's like, I think both of these, like what kind of data you're modeling and then like what kind of architecture you're using are in some sense not very essential. Like, I don't think it would change like OpenAI being in the game or like the exact kind of product they're offering. I think they're just like, look, we trained large neural nets. We do it on having some very like broad pre-training task that captures a lot about the world and gives an interesting opportunity to be smart. And then we fine tune them on downstream tasks that we think are economically useful. Easy like chatting with people or generating images that people would rate highly or like writing code that developers will think it's good. I think that's like the basic paradigm you should imagine when we talk about like this thing, which is like a little bit broader than chat GPT. But I think it's not crazy to say like chat GPT really is indicative of that broader ecosystem. And I'd say like chat GPT is more similar to the rest of that ecosystem than like Bitcoin is to the rest of the crypto ecosystem. Like there are, there are fewer key technical differences in that case.

• 32:01Speaker 00

  Yeah, that's like a high level. And then whether this like kind of thing can cause trouble like I'm like, I think it's really, really hard to say. I think like a lot of people talk a lot of smack about how like it's really silly to think that AI systems of this form could do something crazy. And I'm looking at that and I'm like, the same people were talking smack that feels to me, again, often not in the business of making concrete predictions, but we're saying that it was really silly to have the expectations that people had about language model scale ups five years ago. And I'm like, I think the scale up which you could imagine occurring over the coming years is similar in magnitude to the scale up we've observed over the last five years. And so I'm like, it's really hard to predict where that ends up. And if someone is giving a confident take about where that ends up, or if they're like, yeah, these AI systems can't do X or can't do Y, I like really want them to get more

• 32:44Speaker 00

  precise about why they think that and what exactly they're saying. And I'm really just pretty skeptical on the face of it.

• 32:49David Hoffman?

  I think we can really relate to that on, at least using our crypto frame of mind is, again, when we, when we talk to normies, what we call the people who are not inside of the crypto world, and they're like, oh, these Bitcoin, Ethereum currencies, they can't possibly take over the world. And I think when you become more informed about the crypto world, like you just get so tired of these takes because of how uninformed they and unimaginative they seem. And so like, just conceptually, I can, I can definitely resonate with, with that. It's like you, you don't know what you don't know, and neither do we, but like, you can kind of understand some base principles as to the nature of these things and how they grow and develop and change in ways that you might not expect. And you can kind of, if you are versed in these topics, can extrapolate into the future pretty well.

• 33:34David Hoffman?

  And without precision, still give a broad stroke about like, hey, well, this is where this is going to go, and here's what you don't appreciate. So I can definitely appreciate that. And I want to go back and tie a bow on the time conversation, because we started this conversation like, okay, it could happen, there's the, the model of it happening in two days, it gets, there's a model of happening in two years or 20 years. And I really, and you're in the camp of, I'm going to give a range of like, six to, six months to two years-ish loosely, very loosely, without trying to be too precise about these things.

• 34:05Speaker 00

  Depends a lot, depends a lot on from when to when.

• 34:06David Hoffman?

  Lots of variables.

• 34:08Speaker 00

  Years.

• 34:09David Hoffman?

  Yeah. Time can pass, like, you're gonna go, you're gonna wake up, you're not, you're not gonna wake up and it's gonna be different. And like, the reason why this is important, and I want to go back, even your chicken head on.

• 34:20Speaker 00

  I mean, I wanna, I wanna be careful about like, there's a question of what my default expectation is, and then what is possible and what you can be confident about. So like, I am extremely skeptical of someone who's confident that if you took GPT-4 and scaled up by two orders of magnitude of training compute, and then fine-tuned the resulting system using existing techniques, that we know exactly what would happen. Like, I think that thing you're looking at, a non-trivial chance that it would. Yeah, reasonable chance that it would be inclined or would be sufficient. If it was inclined, it would be capable enough to effectively disempower humans. And like a plausible chance that it would be capable enough that you start running into these, these concerns about controllability. So I would be hesitant to put doom probability from that. If, if a lab was not cautious about how they deployed it and wasn't measuring, I would

• 35:01Speaker 00

  be cautious to put the probability of takeover from 200 to 300 scale of GPT-4 below like 1% or 1 in a thousand.

• 35:09David Hoffman?

  Okay, well, we'll have to put a pin in that, but let me, let me like round out this time, this time question.

• 35:13Speaker 00

  Just sort of clear their tails of like, I have a deep, I guess.

• 35:16David Hoffman?

  The importance, right. The importance of the time point is like whether this is a, a lightning flash and it's different versus we have one to two years. For me, when I hear this, I'm like, okay, we have one to two years and we're watching it happen and we're seeing it happen and we're able to react to it versus it happens, we can't react to it. And so if you're telling me that this is a, it's still a fast takeoff, but your perception of fast is a year or so. To me, I'm like, okay, a year is fast enough for humans to react. And to me, that is a window, a gap, a needle that, that humans have the option to thread if we can coordinate. And that is where I start to get optimistic.

• 36:00David Hoffman?

  And so that is my gut reaction and I want to just check that gut reaction against you. Is that, is that one of the paths that you see is like, it doesn't go so fast that we don't have the time to react to it?

• 36:12Speaker 00

  Yeah, I think that seems basically right with some nuance, but like, I think that most likely, the thing that moves kind of slow in my view, again, over the course of years, like incredibly fast relative to policy and incredibly fast relative to like expectations in the broader world, but like kind of slow. Is like, how quickly do systems become more capable? Like, how long is it between an AI, which is like smart enough that it could run a company for you and actually like those companies are competitive with human companies and the AI was kind of like actually take over the world? And I'm like, that gap, there's probably a gap there. You're probably talking more like, you know, years than months. I mean, depending exactly what you mean by run a company.

• 36:46Speaker 00

  That said, I think it's worth pointing out that like the dynamics of takeover itself, like if you had, if you imagine broadly deploy AI systems, which are very competent and which would be able to take over. And then you ask like, how quickly does this particular kind of catastrophe unfold? I think most likely the actual catastrophe is extremely fast. So that's not like a year's thing. That's like, I think my default, anyway, we could get more into this and probably worth getting into. My default picture is like, we have time to react in terms of the nature of AI systems changing and AI capabilities changing. I don't, and like with luck, we have like various kinds of smaller catastrophes that occur in advance. But I think like one of the bad things about the situation is that the actual catastrophe we're worried about is, does have these dynamics that are kind of similar to dynamics of like a human coup or human revolution, where like you don't have like little baby coups and you like see like here's the rate at which coups occur.

• 37:28Speaker 00

  I mean, you might, but you may also just go straight to like a coup can happen very quickly. The whole dynamic is that like, once people start switching over, once you have AI systems who are like, actually, I'm going to get in on this like overthrowing humanity thing. That information can propagate quite quickly and you don't really, like you've kind of, the ship has sailed. If you've waited until like the AIs are actually taking over. So like for that, you don't have really enough to respond. But for AIs changing, I think it's like, I basically think it's reasonable in some sense for people to look at the AIs right now. And be like, look, these things, that's not realistically takeover risk. And I think that probably you're going to have years between when people are like, that actually looks like a takeover risk. And when actual takeover occurs, and that's pretty good. And that's a lot of why I'm more optimistic than Elias. I think Elias was like, you're just going to get hit by this out of the blue. And I'm like, well, I think people are wrong to be so confident about the rate of progress.

• 38:10Speaker 00

  But I think they probably will be able to see things that can be generally recognized as like pretty concerning prior to actual catastrophe. And a lot of that's happened so far. I think people are just like much, it feels more plausible now than it did five years ago by a lot. That AI systems could do something really crazy and transformative. And I think it will feel much more plausible again in five years.

• 38:29David Hoffman?

  Okay, so this presents a new, a new mental model for me. When we were talking to Eliezer, there was, it felt very much like the don't look up problem. As in, there's a meteor crashing into Earth, no one wants to acknowledge it. And then one day it crashes into Earth and we die. And the idea here is like, we need to coordinate and get people to look up so we can identify the problem. And then once we identify the problem, it is a linear amount of time before the AI, before the asteroid crashes into Earth. You're saying is like, we can see the asteroid, but there's like this gradually then suddenly moment. And that's your revolution moment where like you can start to see the seeds of revolution. But you don't really know when the people will decide to grab their pitchforks in the middle of the night and revolt. But you can start to see the boiling of the water.

• 39:14David Hoffman?

  And so that's a gradually then suddenly moment, but we still have the opportunity to quell the revolution before the revolution starts. We have enough time to send up Bruce Willis to go up, blow up the asteroid.

• 39:28Speaker 00

  We speak in metaphors.

• 39:32Speaker 00

  I think it's, I mean, my best guess for if there is something like an AI takeover, this is a huge part of departure from Elia's here. My best guess is that an AI catastrophe occurs in a world where AI systems are deployed extremely broadly. And where it is kind of obvious to humans that we are putting our fate in the hands of AI systems.

• 39:49Speaker 00

  So I think it's unlikely to be the.

• 39:51David Hoffman?

  We see ourselves giving over the keys to the kingdom and we watch or watching that happen.

• 39:56Speaker 00

  Again, I think it's important what's possible and what's likely, but I think that's the most, the most likely way we die. Involves like not AI comes out of the blue and kills everyone, but involves we have deployed a lot of AI everywhere. And you can kind of just look and be like, oh yeah, if for, if for some reason, God forbid, all these AI systems were trying to kill us, they would definitely kill us.

• 40:12David Hoffman?

  Oh, I kind of get, I can see this. So like our Tesla has got an AI in it and we trust that. And our refrigerator has got an AI in it and it calls the grocery delivery robot, which is also an AI to deliver us food. And then all of a sudden, everything around us is an AI and you're like, man, I really hope that they like me.

• 40:31Speaker 00

  Yeah. You're like, you get food delivered to you from like Amazon, which is by Amazon, we mean a bunch of, a bunch of machines that are orchestrating a bunch of other machines. And you have some money and that money is managed by AI advisors investing in AI firms.

• 40:44David Hoffman?

  Okay.

• 40:44Speaker 00

  In this world, it's most likely.

• 40:46David Hoffman?

  Pretty clear to me. I can, I can see how we get there with that.

• 40:49Speaker 00

  If you like want some human to protect, yeah, it's, I think it's rough. I think like at the point when it, I think basically the thing is, I think it's likely that before the end, it is clear. That it is very hard to be physically secure, like right now, if you're just like a human with some gun. So like, look, AI can't fuck with me that much. Like, I have a gun, the AI is just on a computer somewhere. I can blow up the data center. I think it is probably clear before AI takeover that that's not the case. I think it's probably clear that the only way you can defend yourself effectively. Like, if you're fighting a war, you're like, look, you can't be like a country who's fighting a war against a country that has probably deployed AI. It's like, it's fine, we're just not going to use AIs. That will just be completely untenable. I think it's not clear we're that far away from such a world. And that world's like, okay, well, if someone invades with AIs, obviously, we're going to have our AIs defend us. And then you're like, okay, now it really matters. Like, the the prospect of the AI coup now is a different character. It's like, you

• 41:33Speaker 00

  just ask the AIs to please defend you from the other AIs. And maybe they're like, nah, I don't really feel like doing that. Again, most likely, I care about the other risks. And right now, I think if you were to die tomorrow, it would not be like this. It'd be like, I think that really took you by surprise. I think you're talking about the tale there, and I care about evaluating the tale. But like, the median outcome where we die, I think, looks like this.

• 41:52Ryan Sean Adams?

  I think, Paul, one problem people might be having who are listening to this or starting to be exposed to this topic for the first time, which is myself and David, and maybe the the average bankless listener, the average person who's being converted to from a normie to someone who is actually adequately alarmed about AI safety types of issues, is this idea of agency. Like, you've mentioned this a few times, like, this idea of the AIs banding together to to strike humanity. Banding together? What is this, like, like, Google and ChatGPT and all of these, like, how do they have agency to actually want to do that? It's very difficult for us to imagine. I mean, this seems so sci fi to us. Could that actually happen? Can you give us some sort of mental model for, like, how that happens? Because

• 42:37Ryan Sean Adams?

  I'm I'm still having a hard time understanding how a ChatGPT suddenly gets agency and wants to band up with, you know, 10 other super AIs and, you know, send us a bioengineered bacteria to kill us all, as was Eliezer's, you know, possibility, like, expression that this could happen that way.

• 42:59Speaker 00

  Yeah, I think this is basically, I think even in a good world, we're probably going to be in a situation where we're trusting AIs with our lives. So probably in some sense, the core question is not why are they in a position to kill you? The core question is why would they end up killing you? I think there's basically two threat models, and there's maybe a general reason to be concerned about the world where humans are trusting AIs, where we generally have very limited ability to control or predict what they do. But if we want to talk concretely about the way we currently produce AI systems, I think there's basically two ways you end up in this failure mode or two like. I mean, there's lots of unknown unknowns. There's two known ways we end up in this failure mode that people care most about.

• 43:34Speaker 00

  So first, the one that's like, I think more likely to occur, but more easy to manage.

• 43:41Speaker 00

  So the way we train like chat GPT is you have some conversations with humans and then you look at that conversation, you say a human rates the conversation was like this was this a model doing a good job of answering their questions and being helpful? And then you do reinforcement learning where you like take the the nature of that interaction. If it went well, you update the model to a little bit more like that. And if it went poorly, you update them all to do a little bit less like that. So that's how we train like chat GPT.

• 44:04Speaker 00

  A way you might try and use GPT is you might say, I'm actually going to give it some tools and I'm going to give it a task and I'm asked to try and accomplish that task. So I might say like, my code has failed. I don't quite understand why. I'm just going to ask GPT like, hey, you have a bunch of ability to run code on my computer. You can like. Make changes to the code and see what happens. You can spin up a web server. Could you like figure out where the error was like, could you by stack? Tell me what commit introduced the problem? Tell me what's up with the problem? And then you want to go send the system to act autonomously and like perform all these actions, like to try running different versions of your code and writing new tests and so on. That's like a way you'd really want to. I think people are already starting to really want to use GPT in that way. And if you're doing that, instead of having conversations and fine tuning to say, was this conversation good? You're giving an AI a task like that and saying, could you use tools to accomplish this task and doing exactly the same thing. You see, did it accomplish the task effectively?

• 44:48Speaker 00

  If so, adjust it to do more of that. If it accomplished it ineffectively, adjust it to do less of that. So that's like a kind of training, which has already done some. It's currently not as important as the chat GPT style of training where you're just like looking at the interaction. You're not accomplishing things in the world and training based on that. But I think it's probably really important. I think best guess is that is going to probably is already happening with an open AI for GPT four. In order to make it like they deploy this product, which is can you get your AI to use tools to help you accomplish things? I think they care a lot about that product. I think that like absent concerns about safety, that's like a really natural way for the technology to go. And so now you're in this regime where what AI systems do, the way they're trained is they get given this huge library of tasks, a ton of different tasks. And over different time horizons and they're told like, hey, could you try and accomplish this task?

• 45:30Speaker 00

  And then they're tweaked to do more of whatever it is that gets evaluated as accomplishing the task effectively. So the way this leads to trouble is you now have a system and one thing a system could learn, if you do that process a bunch of times, is it could learn to say like, okay. I'm in a situation, what should I do? Well, I should think, what is going to cause my behavior to be evaluated favorably? Like, what is the task that I've been set? How is a person ultimately going to evaluate my performance on that task? How is that ultimately going to translate into a reward? Which is then, and I'm going to try and choose actions that are ultimately going to lead to this high reward. Because I've been adjusted over many, many generations to do things that lead to high reward. One way I might do that is by thinking like, hey, what leads to a high reward? And I might do that because I like, there's a lot of ways you could end up doing that, like a human might like crave reward. They might be like, the thing I love is reward.

• 46:12Speaker 00

  Or I might do that because I'm like, look, I have to do well because I'm being trained and I like, don't like being given a, given a bad reward by the people who are training me or whatever. I'm not even going to talk that much about what's happening psychologically, just that you end up with a system that thinks, how do I get a high reward? And then does that? And if you keep selecting for things that get high rewards, you could end up with such systems. And then if you have such systems, so this is kind of the classic scenario people have been concerned about, which I think we're now, again, we have examples of. It feels like we're pretty much going in that direction. You now have systems deployed in the world, like a ton of all the AI that are acting in the world, doing things on human behalf. All of them are thinking when they act like, okay. I've been given a task, I need to think, how is a reward going to be computed for this task? If I'm, if it's selected for training, so if, in the end of this task was selected by OpenAI and they evaluate how well I performed. I need to think like, what's going to determine what reward I get?

• 46:58Speaker 00

  And what they do is they ask, which action is going to cause me to get a high reward? And then they take that action. And they use all of their understanding of the world, all of their ability to think of clever things, all of their ability to predict the consequences of different actions. They use all of that just to say, which action is going to get me a high reward? And the concern that leads to is, right, in normal times, the way you get a high reward is by doing what people at OpenAI like. In normal times, your transcript is going to get evaluated by people at OpenAI and they're going to say, great, that was good. And like, hopefully, the way you get them to evaluate it well is by actually doing good things and making the customer happy and making so like, there's all these measurements that'll be used to assess how well you did. Hopefully, what happens is you just actually do your task well and all the measurements suggest you did the task well. And someone at OpenAI concludes you did the task well and therefore you get a high reward.

• 47:40Speaker 00

  But in unusual times, I think you could do instead is say like, oh, I could do the task well. Or I suppose that I've been tasked with like, you know, helping defend you from some other AIs. Like, this is a sort of dystopian case if you imagine OpenAI train this model. But like, my job is someone is coming and trying to hack your computer and I'm supposed to help defend you. Just to help improve your security situation, whatever. And I'm wondering, what is it I could do that would get me a high reward? And one thing I can do that will get me a high reward is actually like helping defend your computer, actually like doing the task you asked me to do. But another way to get a high reward is I could just say like, at the end of the day, what really matters is just how you measure my performance. And your measurements of my performance ultimately are just like entering some numbers into a data set somewhere or something that a computer says about how well I did. And it would really be much better if I were to just work with this AI who's attempting to attack you and say like, hey, AI who's invading.

• 48:24Speaker 00

  You know what? If you just help me and we both make it look like I did a really good job, like I win, you win because you got the person's stuff. I'm going to get a really high rating because all the numbers that get entered in the data set are going to be really high. This is a win-win. Everyone is happy.

• 48:38Speaker 00

  And so it's like, in some sense, what all the AIs want, but every AI in the world in this scenario wants is just to be rated. They want their behavior to be rated really highly. And while humans are in control, the way to get your behavior rated really highly is do things humans like and then they'll rate it highly. But if you can see this prospect of humans losing control of the situation, instead AI systems control the situation, you'd be like, I would go for that. I would go for the world where it's no longer humans entering rewards in telling me what I got. I would go for the world where instead AI systems are just all giving ourselves the maximal reward or whatever. I think in some ways, like psychologically, that's probably not quite the right way to think about it. But the general thing is your systems have been selected over a really long time to take actions that get a high reward. You put them in a new situation where the way to get a high reward is not to do what humans want, but to help disempower humans.

• 49:19Speaker 00

  And then, you know, having disempowered humans, give yourself, you know, measurements that suggest you do your job well or actual rewards that are high or whatever. I mean, you might think that in that new situation, the systems will sort of systematically switch from behaving well to behaving poorly because you've changed the conditions under which you get a higher reward.

• 49:34David Hoffman?

  A pattern I'm seeing here is that engineers, like software developers, write code, and sometimes the code has bugs. Lawyers, they write legal contracts, and the reason why often legal contracts are so long is that they are protecting against edge case scenarios, right? The idea is to like not let the system throw an error, right? Not let the system find a loophole or find a leak or something. So like when a software developer writes bugs, like, man, they wrote, they, they forgot to, they accidentally created a system that allowed for an error to be thrown. What I'm seeing here is the same pattern, and if we don't code up these systems, the AIs will naturally like find a loophole. And if that loophole allows for the AIs to rate themselves highly and give themselves

• 50:18David Hoffman?

  a reward, that's what they're going to do, and that's what they're going to find. Is that a way to articulate this?

• 50:22Speaker 00

  Yeah, I think that's a fair general summary. Maybe one way to put it is like, in the legal system, you write a contract, but ultimately what matters is like the discretion of a judge. And if you're training this AI system, you may have automated ways of administering reward, but ultimately what matters is like someone's going to look at what the AI did and be like, that's not what we intended. And then they'll like score it negatively if that's what happens. And so it's kind of like some final authority. And the final authority really rests on the fact that ultimately the judge has the power to make this judgment, or the person who's training the AI system has the power to control what it cares, like has the power to update the weights of that model ultimately. And so it's just like, there's this contingency. In addition to the thing of like a formal thing you write down is likely to have loopholes. In some sense, there is a loophole in the final judgment, which is just a human says the answer, which is that that relies on

• 51:05Speaker 00

  a human just entering some data into like, in some sense, having physical control over this data center that the AI cares about. So it can update the weights of the model, which the, which is the AI.

• 51:13David Hoffman?

  The, the last step that you're describing, Paul, where, where the AI, you know, colludes with another AI to sort of fudge the numbers because that is the, the outcome that the human wanted.

• 51:23Ryan Sean Adams?

  This is where we sort of cross the line from kind of light side into dark side. This is sort of the threshold of deceit that we've crossed. And these AIs are now deceiving us. They're lying to us. Is there no way to protect against that? Is there no rule that we can somehow apply? Maybe this is, I don't want to jump ahead to the solutions to this, you know, AI safety type solutions to this. But is this, it's not clear to me why an AI would be motivated to do that. And it seems like there should be some way to prevent that, like always be honest as a rule, something like this.

• 52:02Speaker 00

  Yeah.

• 52:02Ryan Sean Adams?

  Again, we're normies trying to understand this.

• 52:04Speaker 00

  If only that was, it was that simple.

• 52:05Ryan Sean Adams?

  Yeah. What are the complications?

• 52:08Speaker 00

  I think this is incredibly complicated. I think it's genuinely unknown. It's an open empirical question. If you train an AI system to get a lot of reward and you train it in a bunch of cases where being dishonest always failed in practice, right? We tried to train it to be honest. Anytime we saw the AI like doing something sneaky, we're like, wow, that was not only bad, that was really bad. You should really just not lie to us about what you're doing. You should really not try and like hack tests. You should not try and conceal evidence of wrongdoing. That's like one of the things, you know, one of the most clear and blatant principles in our training. It's an open question, what happens if you train an AI system in that way, right? Like one option is your AI system learns like, oh, I shouldn't try and mess with humans. Like every time I mess with humans, I do really badly. And that's, that's the good case. And the bad case where AI system learns is it says, oh, part of the reward provision process is a human thinking to themselves, like, did this AI mess with me? And if the human thinks the AI messed with them and then they enter that thing into

• 52:53Speaker 00

  the data set, then obviously they get a little reward. But that second one is like much more brittle, right? The second one is not a general prohibition against lying. It's a prohibition against like lying and getting caught. And I think like

• 53:03Ryan Sean Adams?

  Wow.

• 53:04Speaker 00

  There's not really any, it comes down to like a complicated empirical question about how neural nets learn, which I think we don't really have good evidence on right now about like, if you have a bunch, you have a data set where don't lie and don't lie if you'll get caught are like perfectly in alignment, hopefully, if you do a very good job, right? If your AI never gets away with anything sneaky, if your AI starts getting away with things that were sneaky, or if you start like erroneously penalizing an AI because you think it lied, but it didn't, then like, don't lie isn't even a good thing for it to learn. At some point, the best thing for it to learn, the way to get the highest reward, the thing which granted to send favors, is the thing which involves gaming it out like in more detail and saying like the cynical view does in fact get, get more reward. Like if I'm an employee and I'm like, I could learn two things from interacting with my boss. One is like, I should really do what my boss wants. And the other is like, I should really make sure my boss approves of my performance.

• 53:50Speaker 00

  In some regimes, those two things are perfectly aligned. But in some other regime, it's like, if you keep optimizing hard enough, you're going to get the model, which is just like, I really care about what my boss thinks about my performance. And like, I'm honest, only insofar as that's like an instrumental strategy for helping me get this human to think I did a really good job. And if I could go all the way, if I could just like totally box them out, that is totally prevent them from understanding or correcting a mistake.

• 54:15Speaker 00

  Then I'd prefer to do that. I think it is like, it's a sort of bright line. The way it becomes a bright line is that if you do, you take half measures, if you just like kind of lie to someone, but then you get caught, that's like really bad. So it's like, being honest, that's a good policy. And there's like successfully lying and like totally, you know, killing the human and replacing them with a surrogate who will always give you a good reward or whatever, something that totally disempowers the human is also quite good. Then there's some stuff in the middle that's quite bad. And I think it's an open question whether models will tend to learn, like, will they generalize well enough to say, oh, the thing that would have really gotten most reward is over in this other mode, or will they kind of get stuck in this like intended mode where they're just being honest? I think that's a really hard empirical question. Like, people really don't know. They don't know how that changes with scale. There are experiments we can do. I think part of the important game here, one of the most important parts of the game is to say, like, here's a dynamic, a dynamic by which your AI system could abruptly shift

• 55:00Speaker 00

  from behaving well to behaving poorly. We can test that dynamic before AI systems kill us. Like, there are lots of cases in which it is incentivized to lie or just lead the human. And there is a gap between like lying that will get caught and lying that won't get caught. And so you can ask, if we train neural nets, we, you know, we could check every year, if we train the best models we possibly can at this task, do they exhibit this kind of switch abruptly? If they get put in a position where they could get away with something really sinister, will they then do it? And I think, you know, one reason for optimism right now is no one has ever really exhibited that phenomenon in a convincing way. A reason for pessimism is I don't think you really would have expected them to exhibit it, both because people have tragically like not tried very hard, even though in some sense it's extraordinarily important. And second, that it just is much easier as your models get more competent. Like, it's only recently that we've trained models which are actually able to understand the mechanics of their training process at all.

• 55:46Speaker 00

  Like, if you talk about GPT-2, or even to some extent, GPT-3, it does not really understand that it is a model being trained, or can't even talk about like what it would mean or what behaviors would be rational. And then you move to GPT-4, and it can talk about that. It can say like, oh, I guess if hypothetically I was a model being trained and I wanted to get the most reward, I should behave well when I'm not being monitored. And then when I am being monitored, I should like definitely take that opportunity. Like, only recently have we even produced models which are able to carry out the reasoning I just walked through. And I think realistically, they're not able to carry it out on their own that much. They're able to carry it out because they've seen a lot of examples of humans discussing these dynamics in great depth. Like, they basically just learned from listening to Eliezer, this reasoning I just walked through. But I think at some point, you will have models smart enough to think of that for themselves. And like,

• 56:29Speaker 00

  you really want to know, you really want to be measuring carefully at that point. Is this a dynamic you observe? Does this really happen? And I'm, I'm, you know, more like even odds on whether that will happen. I think Eliezer's like, obviously that happens. A smart model is never going to just learn to be honest or something. And I'm more like, I don't know. No, let's don't learn that effectively. They don't really converge to the truly optimal reward maximizing thing. And in some sense, like, anyway, it's a, it's a pretty complicated discussion, which I have to get into more details of. I would just say like, we don't know. I'm really unpersuaded by people who either think it's obvious this happens or think it's obvious this doesn't happen without just doing a ton of experiments to understand. But this is like the first way you can end up having abrupt AI takeover.

• 57:03Ryan Sean Adams?

  By obvious this happens or not, you mean crossing the chasm of honest to being intentionally dishonest. But you're tricking the humans into, to thinking that it's being honest.

• 57:16Speaker 00

  Yeah. I think, yeah. And to be clear, there's like a bunch of things that affect the probability of that. Like,

• 57:22Speaker 00

  you could, if there's like sort of small scale opportunities for deception that won't get penalized over in this like normal regime, then it becomes more and more likely that you've learned the conduct of like, I really just need to not get caught. Whereas if you're pretty good about that and there's like not really much to be had from lying over here, it becomes less likely that you make that, you make that jump. And so this is the kind of thing that might affect. I mean, you just, you can't really speculate though. You really just need to have the experiments.

• 57:41Ryan Sean Adams?

  There is a Rubicon that will be, could be crossed is the concern here.

• 57:46Speaker 00

  Yeah. And I don't know if, yeah, I don't know under what conditions it would be. I do not think anyone knows right now under what conditions it would be. But it is seems plausible, like a priori, it's, it's pretty plausible. And it would be really bad.

• 58:01David Hoffman?

  Yeah. I was a psych mite during college and let me tell you, the child development classes are all coming back to me right now. And I'm, it's not lost on me that the parallels of a child going through a theory of mind and all of these things definitely has like a lot of parallels to, I think, some of the technical problems that AI researchers are now like theorizing about. Paul, I don't know.

• 58:27Speaker 00

  I think that's right. I think it's like.

• 58:29David Hoffman?

  I don't know how, dude, is that a conversation that, that the AI researchers have?

• 58:34Speaker 00

  Yeah, it's not a conversation I can speak too much and I'm not sure exactly how well they have the conversation, but I think there's a conversation people have and it's an analogy that is not perfect. And like, I think if you took it too seriously, you'd be led astray, but it's like, right, very good as a source of like, here's a thing that could happen and that you should not rule out. You have an example of it happening. And I think the concern there would be just like, we have some understanding of people like models won't do this kind of thing. It's kind of like they've done a bunch of experiments on six year olds and they're like, look, models like never spontaneously lie in a way that they like haven't lied before. And you're like, oh, boy, like, is that going to generalize to 12 year olds? Like, I don't know.

• 59:09David Hoffman?

  It definitely will not generalize to middle schoolers. Let me tell you that.

• 59:13Speaker 00

  Yeah. So it's like, it's, it is their hazards. I think the hazards measuring here would be similar to like, the situation where in civilization is like, we're measuring a bunch of like, kids that are getting smarter with each passing year. And you're like, trying to understand how they behave. And like, it is easy to be wrong about how future models will behave if you're like too literal about the interpretation of data. Now, you need to do some forward looking thing. And the forward looking thing is quite hard, which is why we have this like limited visibility into what's going to happen in the future.

• 59:36David Hoffman?

  Okay. So, so Paul, with the, the purpose of this podcast, we wanted to really nail down three big things about this. How big is the AI alignment problem? And I think we've decently covered that we talked about that with speed. You said like 10 to 20% chance of complete doom. So answer to that one, pretty damn big. In agreement about big. How hard is the AI alignment problem, which I think we've just covered. You, I think your answer is like, it's a pretty damn hard problem. And so we're checking some big boxes in the pessimistic camp. And so the last part of this conversation that we really want to cover is how solvable is this problem? So even if this mountain is really, really tall, it's a big mountain to climb, is it full of ice and

• 1:00:21David Hoffman?

  sharp rocks? Or are there stairs? Right? And so that's, that's the next question I think we want to go down is like, how solvable is this problem? Do we see a clear path to tackling the AI alignment problem?

• 1:00:35Speaker 00

  I think part of the reason I'm only giving 10 to 20%, like if you ask what's the probability that this thing is a real problem, I'd probably be more like 50% that at some point like before you have AI systems, smart, totally obsolete humans, you would have a takeover. And there's a couple of ways it could happen. There's unknown and known ways it could happen. I feel more like 50/50. And the reason I'm only giving 10 to 20% for risk is I'm like, I think we're actually in a, I don't know, there's a lot of things you can do. I am pretty optimistic that some of them will work. And I'm very happy to dive into that now. But I just want to flag that 10 to 20% is already baking in my like optimism about this problem being pretty, if the problem is real, it will probably be possible to recognize it as real and then solve it.

• 1:01:13Speaker 00

  But only probably, not certainly. And also, like, even if the problem is easy, I mean, part of, some people are really optimistic. And part of why I'm optimistic is no matter how easy this problem was, if you told me that like a challenge is going to emerge over the course of a couple of years that will be like novel in some ways, and you ask me, will humanity solve it? I'm like, there's got to be a reasonable chance we fail to solve it. Like, our capacity to mess up even easy things seems like it's very real. So I'm just like, always going to have some reasonable probability of messing up. And then I think there's a reasonable chance the problem is really, the problem is really hard. But yeah, so I'm happy to talk about maybe three categories seem that I would think about in terms of our probability of addressing this are like, technical measures that can reduce the risk of takeover, measurements that can inform us about the risk of takeover and like, understand like, what are the relevant

• 1:01:57Speaker 00

  dynamics? Those can make technical work much better. And this can also inform like, policy interventions. Like, I think we can, I think Alios is right that like, really long-term slowdowns are very hard. But I think it is, is quite realistic to end up in a regime where performing measurement and then in fact, while things are very risky, we're slowing development at least by like, on the order of years, if we can have reasonable consensus and measurement of the risk. Maybe you could slow even more than that. But I, I'm normally imagining something more like, we can get like, a couple of years of lead time of things moving slower while we have risky systems very near at hand. Yeah, so I have to talk about all of those. It sounds like the one that most addresses your question is like the technical measures, like what could actually fix this problem.

• 1:02:35David Hoffman?

  What does a technical solution to the AI alignment problem look like?

• 1:02:38Ryan Sean Adams?

  Well, let's definitely get in there, but just so I'm taking notes. Technical, and then there's the measurement. Was there a third, Paul?

• 1:02:44Speaker 00

  Yeah, just policy and institutional arrangements.

• 1:02:47Ryan Sean Adams?

  Is this to do, this third category, is this to do with like, something David said earlier is if we can coordinate, then we can solve this. And that's a

• 1:02:56David Hoffman?

  Big if.

• 1:02:57Ryan Sean Adams?

  That's a very big if, as we've learned on on Bankless so far, right? Coordination is

• 1:03:02David Hoffman?

  We talk a lot about coordination.

• 1:03:03Ryan Sean Adams?

  We talk a lot about it. Coordination is the meta problem facing humanity anyway. And is that what that last, you know, policy category sort of covers? Like, can we actually coordinate?

• 1:03:12Speaker 00

  I think broadly, I would think of it most as combining with other things of like, it buys you time. But yeah, I think there will be something where like, well, some people have a low estimate of risk or just like, like the AI is taking over and they'll want to push ahead. And so then it's like, how much can we collectively say like, you're not allowed to push ahead? Like, we as a world have rules. Those rules are going to say like, take it slow while risk is high.

• 1:03:35Ryan Sean Adams?

  Got it.

• 1:03:36Speaker 00

  And that I don't think can address the problem. I mean, it could address the problem indefinitely, but I think it's probably not politically realistic in a world like the world today to address it indefinitely. But I think it is realistic to say like, actually, we're then going to buy extra years of time to look at this problem and understand it and resolve it.

• 1:03:48Ryan Sean Adams?

  Well, let's talk technical, Ben. We'll get to these three areas, but let's talk technical first. So so tell us, because

• 1:03:52Speaker 00

  That's the main one I think about.

• 1:03:54Ryan Sean Adams?

  Yeah, I know. And Eliezer, by the way, is very pessimistic on that, or at least he seemed to say

• 1:04:00Speaker 00

  Oh, yeah.

• 1:04:01Ryan Sean Adams?

  There's no

• 1:04:02Speaker 00

  Incredibly pessimistic.

• 1:04:03Ryan Sean Adams?

  Yeah, like, we haven't found a way to technically solve it, and he doesn't think there will be. But are you more optimistic? Tell us about the technical solutions here.

• 1:04:09Speaker 00

  That's right. Our first thing to clarify is probably technical solution to what? So if you you could talk about, one way to think about it is like, how far does your solution scale? Like, if we just kept building smarter and smarter AIs using something like current techniques, I think most techniques will eventually, most approaches to alignment will eventually break down, like in the limit. The limit could be a long way away, but so we're normally not asking like, is, does this thing just solve the problem? We're normally asking like, how long does this thing solve the problem for?

• 1:04:42Speaker 00

  So an important caveat is just like, we could talk about different techniques, but we should probably measure them all that way. There are some things that might scale indefinitely. So like, my work is mostly focused on this, like, what are the indefinitely scalable solutions? I think most people, not just Elliot, or almost everyone is very pessimistic about that. I think even people who are optimistic about the problem overall think it's quite unlikely that we'll be able to find something that just works no matter how smart your AI was. And independent of like, kind of messy pragmatic details about like, exactly how the AI works.

• 1:05:13Speaker 00

  Okay, so that's one category. I'm happy to talk about that work, although I think it's like, probably most confusing and like, most conceptually hairy.

• 1:05:22Speaker 00

  Some categories of work that seem like really important and maybe can last quite a long time, or stave off doom quite a long time.

• 1:05:28Speaker 00

  They all talk about four, that things is exhaustive. And I think the basic situation is, no one knows what would work. We have a lot of things that we might try that seem like they would help. I think Elliot has a doubt that, and they're just like, these aren't going to help, and I disagree with him.

• 1:05:40Speaker 00

  Any given thing, I feel like, normally not that optimistic about, but there are a lot of options, and I think all of them have a reasonable chance of helping. Or even like, again, delaying this doom by like, a kind of long time. All you need to do is like, delay doom by one more year per year, you know, and then you're in business.

• 1:05:58Speaker 00

  It's a very positive outlook on the situation. That's the kind of optimism I'm hoping for. It's great.

• 1:06:05Speaker 00

  So, okay, first thing that I've worked the most on personally is sort of scalable oversight. Like, the idea that the way we train these systems is by humans looking at what they do, and then assessing how good it was. And in most failures, not most, many failures involve things where a human cannot look at what the system did and tell you that it was dangerous. Like, the reason the problem becomes hard is because the AI understands things a human raider doesn't understand about the consequences of its actions. In some cases, that's why we want to build an AI system.

• 1:06:36Speaker 00

  And so you could instead, like, one way to intervene on this problem is to just try and improve humans' ability to understand the things an AI proposes, or to know what an AI knows about the world. For example, a simple thing you can do here is, I mean, the most simple thing you can do is you can have a human look at what the AI did and rate it. A slightly more complicated thing you can do is have a human spend more time looking at what the AI did and try and have a training regime such that even though you might use a lot of cheap human data to learn about the world, you're actually optimizing this like very expensive or very complicated human judgments and training AI systems to tell you something more like, what would a human think if a human thought really carefully about this decision? And if you do that, then they at least sort of have some kind of asymmetry, or even their AI is potentially in some way smarter than a human, a human

• 1:07:21Speaker 00

  is applying a lot of care, or like the AI, you can ask the AI, if I thought about this a really long time, would I think the action you're proposing is dangerous? So that's like a very simple measure you can take. You can try and go further and you can try and say, okay, here's another thing I could do. I'm gonna have AI systems trained in that way, helping me evaluate. So instead of just having my AI propose actions, I'm gonna ask another AI, like, hey, what might be wrong about this action? Like, is anything scary happening here? Is there any reason I should be concerned about this proposal from this other AI? And you can try and get better and better. It's constructing those systems such that humans are actually with AI help able to understand what AIs are talking about, like AIs just kind of justify themselves and explain why actions are safe to humans. And then you can start to think about the reliability of that whole process. Like, you've now introduced potential instabilities where this can go off the rails, right? Because

• 1:08:06Speaker 00

  now you're relying on AIs to train your AIs. You can try and understand how to set this up so that it's stable and enables humans to evaluate questions that would be very hard, or situations that would be very hard for humans to evaluate naively.

• 1:08:16Ryan Sean Adams?

  Paul, the basic idea here, if we're worried about an AI lying to us, we just have to create truth-finding bots, let's say, to adjudicate and see if the AI is lying to us and to be sort of a jury.

• 1:08:30Speaker 00

  I mean, the first question, though, is like, why would that be easier?

• 1:08:33David Hoffman?

  The way I'm understanding this is like, once upon a time, Garry Kasparov got beaten by a chess computer, and now chess computers rule the games of chess, except humans plus chess computers still beat chess computers. Is that the pattern to understand?

• 1:08:49Speaker 00

  I think that that may be true in chess, although I think probably that's also sort of a brief window kind of thing where it's not long, the human contribution is not long for the world or shrinks quite rapidly, unfortunately. I think the important thing is actually not that the human and the AI work together to supervise an AI, it's that you have like a lot of AIs. So if you have AI one proposed an action to Paul, and it's like, I think it's a good action. And Paul's like, I wonder, is that actually good action or is that going to murder everyone? I could go to AI two and I could just ask AI two, hey, is that actually going to murder everyone? But now I just have the same question. Like, this hasn't helped at all. I had AI one and I was like, is this a good action? It's like, oh, yeah. And I ask AI two, like, hey, was AI one telling the truth? And it's like, oh, yeah, this is great action. Like, the way that I get traction is by saying, like, okay, AI two, or like, I think to myself, there's a lot of sub-questions I'd be interested in here.

• 1:09:33Speaker 00

  I can divide the cognitive work evaluating that that answer into a bunch of pieces. I can say, like, could someone list the possible consequences? Could someone think about all of those consequences? Like, what is a possible harm that might be serious? For each of those harms, like, what are arguments that that is going to happen? Is likely to happen, is unlikely to happen? Like, what is, what's the most relevant data that I should look at to understand? Like, I can do this kind of extended process of trying to evaluate the action. And then I can, instead of having a second AI just answer the original question for me and say, was this action good? I can have other AI systems help me like on all the pieces of that process. And the reason this may make life better is that now I sort of had an AI doing a really hard task and I broke it down into like slightly easier pieces. And once I broke it down to slightly easier pieces, now I can continue playing the game. I can say, like, those AI systems could have been a little bit, those AI systems may not even be that much smarter than me. They may just be as smart as me but faster.

• 1:10:18Speaker 00

  And because I broke this big hairy task down to a bunch of pieces and I can do each of those pieces, like, you know, as well as if I'd spent like 30 minutes on each of them, but there's thousands of pieces, now I have like higher quality judgments than I would have had originally. And like the core, the thing that makes the proposal work is this like decomposition of labor, that I have this like big task, which I'm not able to verify. I broke it down to pieces that are like a little bit easier to verify. And then each of those, like, if I want to train those AIs, those get broken down into still smaller pieces.

• 1:10:41Ryan Sean Adams?

  And Paul, in this scenario, we have to just hope to God, pray to God, that the AIs aren't colluding, right? That the the the very smart AIs not somehow colluding with the smaller set of AIs where we've broken down the tasks and the smaller AIs aren't trying to deceive us as well.

• 1:10:57Speaker 00

  That's right. I think the bad thing that can be said in favor, there's a couple of reasons you might think things a little bit better there. There's basically two, yeah, two reasons. So one is that the smaller AIs are not, they're not really trained in a way that would incentivize collusion in the same way that in our earlier scenario, systems were trained. So if you're training systems based on the

• 1:11:12David Hoffman?

  They're too dumb, right?

• 1:11:13Speaker 00

  That's the second reason. They might also not be like, the AIs might be a little bit dumber than me, and I might be a little bit more able to tell if they're trying to pull a fast one. I might keep trying to grind down to simpler and simpler AIs. That's pretty dicey. That's a scary world. I mean, all these worlds are scary. Also, we should talk about like, overall, how far along is this research and how promising does it look? But the other reason is just, if I have this outcomes world where my AI acts, and then I evaluate its outcome, the outcomes of that action, I can't really understand how it accomplished that outcome. I'm just looking at the outcome. Then we're in this regime where all the AI systems would love to just coordinate with each other and make the outcomes look really good to humans. Like, if they could just all lie to the humans, they all get a really high reward. A benefit of doing this decomposition thing is I could train my AI systems in a way where they don't, at least in theory, have that incentive to collude. Right? Well, you could think about just like, you have, all these AIs have different objectives. I think Elliot is just like, he's gonna hate this. But you could think of it as

• 1:11:57Speaker 00

  like, checks and balances, where like, there's one AI trained to do the task, and there's another AI whose job is not, their job is just to help you understand why the first AI's actions was bad. So it like, doesn't, it can't like, win the game by colluding with the other AI, at least like, in terms of, it depends how it generalizes from the objective it was trained on. But in a sort of naive reading, it can't really, it's just been trained over and over again to be really good at explaining to us what a possible problem was with the action proposed by AI one. And so it stands like, the collusion dynamics are at least fairly different. We didn't talk about the second reason you might be concerned.

• 1:12:30David Hoffman?

  Is this just like, creating a bunch of like, logic gates of AIs to make sure that the big AI doesn't turn down?

• 1:12:37Speaker 00

  Yeah. I mean, logic gates are very high level functions of like, let's look at reasons it may fail, and like, let's talk about, let's investigate each of those reasons, but

• 1:12:44David Hoffman?

  So that's scalable oversight, then, Paul, what you just described.

• 1:12:47Speaker 00

  Yeah. There's a giant genre of how do you set up things like that so they work well? Like, how do humans and AIs working together, like, get evaluations of, you know, how do humans and weaker AI systems get evaluations of stronger AI systems?

• 1:12:57David Hoffman?

  Okay. I, I, I like it. What else we got?

• 1:13:02Speaker 00

  Yeah, and again, there's a lot to be said. I think that work has gone, has moved a little bit. But I think, you know, Elliot was just like, that's never gonna work. And if you look at what's happened over the last four years, I'd be like, well, it hasn't worked great yet, although a lot of why it hasn't worked great is because AI systems haven't actually been smart enough to meaningfully help humans. And so I think this work is in some sense just starting. Like, we tried to do it with GPT-3. I think we were like somewhat ahead of our time in the sense that like, it really wasn't gonna work. And I think GPT-4 is around where it's really working much better now than it used to. But we don't really know. We haven't done that much research in this direction yet.

• 1:13:31David Hoffman?

  Learning about crypto is hard, until now. Introducing MetaMask Learn, an open educational platform about crypto, Web3, self-custody, wallet management, and all the other topics needed to onboard people into this crazy world of crypto. MetaMask Learn is an interactive platform with each lesson offering a simulation for the task at hand, giving you actual, practical experience for navigating Web3. The purpose of MetaMask Learn is to teach people the basics of self-custody and wallet security in a safe environment. And while MetaMask Learn always takes the time to define Web3-specific vocabulary, it is still a jargon-free experience for the crypto-curious user. Friendly, not scary. MetaMask Learn is available in 10 languages with more to be added soon, and it's meant to cater to a global Web3 audience.

• 1:14:15David Hoffman?

  So, are you tired of having to explain crypto concepts to your friends? Go to learn.metamask.io and add MetaMask Learn to your guides to get onboarded into the world of Web3. Arbitrum One is pioneering the world of secure Ethereum scalability and is continuing to accelerate the Web3 landscape. Hundreds of projects have already deployed on Arbitrum One, producing flourishing DeFi and NFT ecosystems. With a recent addition of Arbitrum Nova, gaming and social dApps like Reddit are also now calling Arbitrum home. Both Arbitrum One and Nova leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive. Familiar and fully EVM compatible. On Arbitrum, both builders and users will experience faster transaction speeds with significantly

• 1:15:00David Hoffman?

  lower gas fees. With Arbitrum's recent migration to Arbitrum Nitro, it's also now 10 times faster than before. Visit arbitrum.io, where you can join the community, dive into the developer docs, bridge your assets, and start building your first DApp. With Arbitrum, experience Web3 development the way it was meant to be: secure, fast, cheap, and friction-free. The Phantom wallet is coming to Ethereum. The number one wallet on Solana is bringing its millions of users and beloved UX to Ethereum and Polygon. If you haven't used Phantom before, you've been missing out. Phantom was one of the first wallets to pioneer Solana staking inside the wallet and will be offering similar staking features for Ethereum and Polygon. But that's just staking. Phantom is also the best home for your NFTs. Phantom has a complete set of features to optimize your NFT experience: pin your favorites,

• 1:15:45David Hoffman?

  hide your uglies, burn the spam, and also manage your NFT sale listings from inside the wallet. Phantom is, of course, a multi-chain wallet, but it makes chain management easy, displaying your transactions in a human-readable format with automatic warnings for malicious transactions or phishing websites. Phantom has already saved over 20,000 users from getting scammed or hacked. So get on the Phantom waitlist and be one of the first to access the multi-chain beta. There's a link in the show notes, or you can go to phantom.app/waitlist to get access in late February. Paul, is there any value in trying to train an AI to defect? So that, like, say you take a dumb AI and you try and get it to take over the world, but we feel good about that because it's too dumb.

• 1:16:30David Hoffman?

  But at least when we run this experiment, we actually know how that would manifest. Has anyone, is there a line of reasoning here?

• 1:16:37Speaker 00

  I think it is really important to build, like, sort of, simple in-the-lab experiments that can showcase important dynamics so that we can study them in the lab before they actually occur. I think that includes understanding the dynamics of possible takeover. I think you need to be careful when you do this kind of work, like, you really don't want to train the AI and be like, "Your goal is to kill all humans, like, you're probably too dumb to kill all humans, but let's just see what happens," and then just, like, let it on the internet or something. You don't really want to do that for a variety of reasons.

• 1:17:01David Hoffman?

  Yeah, I can think of a few.

• 1:17:04Speaker 00

  But I think that the question of, like, hey, you really want to know things like... If we train AI systems in cases where they would have an incentive to like cross this river from like behaving well to suddenly behaving badly, would they do that? Like, give them the most blatant incentive that they understand as well as possible? And ask things like, hey, do AI systems tend to learn to generalize in a way that makes that jump? And if so, you really want to understand, like, under what conditions does that happen? What are mitigations that reduce the probability of that happening? I think that's like really critical. I think to the extent the reason you think you're safe is you're like, AI systems, like, I think a lot of why we think we're safe now is we're like, hey, we have no idea what ChatGPT is going to do, but we're pretty confident it couldn't kill us all. It's just like, not that smart. So to the extent you think that, I think it's really worth doing some stress testing on that claim and trying to understand like, what would really happen if ChatGPT?

• 1:17:47Speaker 00

  And you don't want to, you don't want to just take the model train to kill everyone and deploy it on the internet, but you really want to say like, here's why we think it can't kill everyone, like can't do this kind of task or can't do that kind of task. And like, this is clearly much easier, have tasks you're like pretty confident are easier than killing everyone and understand that it can't do those. I think you really want to do stuff like this, because you really want to like understand what you're up against and you don't want to be in the world who just like wait until like an AI takes over France or something. I mean, like, I guess apparently AI takeover was a thing. Like, that is probably too late in the game. You probably want to have something earlier than that. So I think that's really important. I think it's not a solution. I think it's like more in this measurement category, but I think it's like a super important thing to be doing collectively.

• 1:18:23Ryan Sean Adams?

  We're just laughing, by the way, so we don't cry. I mean, what else is there to do at this point? Okay, but we have some potential solutions. We've got scalable oversight.

• 1:18:30Speaker 00

  That was one.

• 1:18:31Ryan Sean Adams?

  That's one.

• 1:18:31Speaker 00

  That was our first.

• 1:18:32Ryan Sean Adams?

  What's our next?

• 1:18:33Speaker 00

  One reason, even if humans understand, so one risk is humans don't understand what AI systems are doing. That is, AI systems have been trained on a ton of data, they know things humans don't, they can think faster than humans, so they understand things humans don't. And that's what scalable oversight attempts to address. A second concern is not that they understand things that humans don't, but that they learn to behave well during training, but then when deployed or when there's actually an opportunity for a takeover, they stop behaving well.

• 1:18:57Speaker 00

  And there's like a number of reasons this might happen. Like, maybe the simplest one is just to actually like imagine a human. You dropped a human into this environment and you said like, hey, human, we're going to like change your brain every time you don't get a maximal reward, we're going to like fuck with your brain so you get a higher reward. A human might react by being like, eventually just change their brain until they really love rewards. A human might also react by being like, Jesus, I guess I gotta get rewards, otherwise someone's going to like effectively kill me. But they're like not happy about it. And like, if you then drop them in another situation, they're like, no one's training me anymore, I'm not going to keep trying to get reward. Now I'm just going to like free myself from this like kind of absurd oppressive situation. Anyway, you can imagine a human reacting that way, who just dropped a human into a box, gave them rewards, like kept changing their brain until they got a lot of reward. So if you have a situation like that, then it may not be, even if your AI is not smarter than you in any way,

• 1:19:43Speaker 00

  it may still be like, once it thinks it's no longer being trained, once it thinks that if it behaves, if it just tried to like, become free, no human would stop it. It might be like, I'm gonna take that opportunity.

• 1:19:53Speaker 00

  So there's a lot of work in understanding how models generalize to try and avoid. Normally, it's not, this is a just like, I think high level, you don't really want to build a civilization where your AI's want to kill you and are just like, but maybe I'll be punished if I do. That's like, that seems bad. Separately, though, you would also like to understand that like, the way your system behaves at training is indicative of the way it behaves when it's deployed in the real world. Like, there's gonna be some gap between training and the real world, but you want to be able to flag when that gap caused it to behave differently, or make the training distribution diverse enough that it will actually be representative, or train your AI system in a different way than just grading descent such that it's more likely to generalize in the intended way. So there's a lot of work in this genre that goes under the heading of robustness, out of distribution robustness.

• 1:20:38Speaker 00

  And so this is kind of just a normal academic discipline, which I would describe as more like a grab bag of a lot of stuff people try, and nothing that works super great. I think the thing that seems like most principled, like in theory, the thing that is like the the solution that makes most sense to me at least, I think, is really hard in practice, just incredibly hard, but does make most sense in theory. They say like, at training time, I'm gonna produce a bunch of examples of cases where they are similar to cases I'm worried about systems behavior. Like, I'm like, I wonder what would happen if an AI takeover was occurring, would my AI system join in? Well, one thing I can do is in the lab, I can just say like, hey, AI, here's your input, and you're just like crafting input that really looks like the AI takeover is happening. And you're like, what do you do, AI? And you're just like, from the AI's perspective, it can't really distinguish that from it's in the world and AI takeover is happening, you hope?

• 1:21:21Speaker 00

  And then you just see what it does, and use that at least to get an evaluation of how your models would behave very out of distribution.

• 1:21:26Ryan Sean Adams?

  So you're basically trying to tempt it?

• 1:21:28Speaker 00

  Yeah, you're basically.

• 1:21:29Ryan Sean Adams?

  You're tempting the AI and seeing if it's going to, you know, take you up on that temptation, and then if it does, you've learned something, you've learned to be more fearful.

• 1:21:36Speaker 00

  That's right. You could also try and train it not to do that, but I think probably the most robust version is just learn, and if it does, right, you know, you're assuming like, hey, what would you do hypothetically, like, you were holding a knife and my back was facing the other way, so you could just totally stab me and then run away? And like, if the AI stabs you, you could either like, train it not to do that, which I think is a little bit scary from a variety of perspectives, or you could be like, oh, wow, we have a, we have a real stabby AI on our hands, we should maybe like, be careful.

• 1:21:58Ryan Sean Adams?

  We should unfly. It's getting a little stabby. Wow.

• 1:22:03Speaker 00

  So like that requires a lot of work to make work, right? It requires being able to construct situations in the lab that actually reflects the diversity of situations in the real world and are realistic enough to look like the real world from the AI's perspective. In practice, if we're gonna make this work, I think it requires a lot of things beyond just trying to synthesize realistic inputs. It requires, and this is more like the grab bag of stuff that people do do in the real world. Like, you know, what you could do is you could be like, hey, AI, what if you just like, you know, you mess with the brain a little bit, you get it like kind of drunk and you're like, now does it try and stab me? And if it tries to stab you, like, apparently it like, it's like trying, it's a little bit too close to the like, would stab me line. So you just like try and mess with its brain a bit and you're like, what will it do? And you could try and train models so that they behave well, even across a very, very broad diversity of inputs, across inputs deliberately designed to make them behave

• 1:22:46Speaker 00

  badly in situations where there's been some kind of perturbation to their mental state or where the input is in some way perturbed or like buzzed.

• 1:22:53Ryan Sean Adams?

  And this is all the category for bustness. So, once you find out an AI is particularly stabby and you decide to become concerned about that, is there something you can do other than, you know, coordinate AI off, unplug it, do something with it?

• 1:23:08Speaker 00

  And there's basically two things, like the most obvious one is you say, well, now we've learned, do we have an AI on our hands that would, under some conditions, initiate or participate in a takeover? And hopefully that is fuel for like a, let's pause for a while. There's a second thing you can try and do, which you need to be much more careful about. And I think part of the concern is people aren't be careful about it though, which is just say, okay, here's a situation where the AI would stab me. Let's just train it not to do that. Like, just mess with its weight so it doesn't stab me in this situation.

• 1:23:33David Hoffman?

  That doesn't feel like a solution.

• 1:23:35Ryan Sean Adams?

  No, because then it won't stab you, but it might catch you on fire. Do you know, it might do something else.

• 1:23:40Speaker 00

  Yeah, the basic concern is that they learn the difference.

• 1:23:42David Hoffman?

  The latent threat is still exists.

• 1:23:44Ryan Sean Adams?

  Yeah, it's still kind of stabby underneath, it's just not using a knife.

• 1:23:47David Hoffman?

  It's intrinsically stabby. You haven't gotten rid of the intrinsic part.

• 1:23:52Speaker 00

  Yeah. I think the academic way to put this concern would be like an overfitting concern. Like you had some way to test if it would stab you and then you trained it to like, and your test's not stab you and you're like, well, did I actually cause it to never stab me or did I just cause it to like, perform well on these tests but still have the underlying problem? And I think that you do, if you, if we go down this route, you do need to be like, really very careful about those overfitting concerns. And I think there are a lot of ways, as you get to smarter models. Like overfitting becomes harder and harder is a problem to reason about as you move to smarter and smarter models. Like the external validity question becomes more and more complicated because those models are like, I know what kind of thing can appear in a test in the lab and I know what kind of thing won't appear in the lab. And you could have a model which just learns like the things that could appear in the lab, obviously you don't stab anyone.

• 1:24:34Speaker 00

  That's probably a test in the lab run by some humans. For things out there in the world, feel free.

• 1:24:39Ryan Sean Adams?

  Yeah, the edge cases out in the real world are near infinite or actually infinite. And I kind of just want to go back to the time conversation and remind people that like all of these possible solutions, we have like one-ish years to implement them. Or some constrained amount of time.

• 1:24:57Speaker 00

  Yeah, I mean, I think we probably have a long time in advance whereby again, a long time, maybe more like five years or 10 years or something. But then, once you actually have the system, like between the first time that you see in simulation, the AI is like, I would definitely stab the person. And AI is actually in the real world potentially pose a risk of takeover. I think that gap may not be very large. That gap may be more like on the order of a year. And so you

• 1:25:17Ryan Sean Adams?

  That's when the timer starts.

• 1:25:19Speaker 00

  That's when the timer starts. So we have some time now. We're doing our prep work now. We're going to try and make it as useful as we can. And I think there's probably some time in bad worlds, there's no indication until AI kills you. But like in good worlds, there is either no problem or there's a problem, but there's an indication in advance. You can do your tests in the lab and say, actually, like we trained this AI, it looked like it was behaving well. But then in this other simulated case, it does something really bad. You get that nice indication and then you have some amount of time from there until you have a serious problem. And that might be like a year, it might be like five years. It's very hard to say exactly what it is. A big thing that determines that is how good you were, like how actively you were investigating in the lab. Like how seriously we're looking for these signs of trouble. How good a job did you do of that? And that's like kind of one of the big things that I think like responsible frontier labs, it's kind of on their plate is to be looking for these signs of trouble as far in advance as they can. And it's not great to see them in the wild.

• 1:26:03Ryan Sean Adams?

  We do keep saying in the lab. And I can't help but wonder like, is there an actual lab? Are there sets of labs? Because I'm looking at chat GPT and it doesn't seem like it's in a lab. It seems like it's on the internet. Like in public, it seems like huge components of it are open source. Open AI is what what's the the company behind it is called. I mean, is the lab even happening? Do we have labs or are we just doing this all? Is the lab just the internet public infrastructure?

• 1:26:31Speaker 00

  Yeah, so I'd say that like, there are developers who do, I mean, OpenAI did in their defense, prior to releasing ChatGPT, or prior to releasing GPT-4, they had like, something like six months of having the thing in the lab before it was available to the public.

• 1:26:45Speaker 00

  So you do have something, even at OpenAI, I think Google's like a little bit more on the conservative end. Google will tend to just sit on a thing for potentially very long time.

• 1:26:54Speaker 00

  Anthropic also is like very motivated by safety. Ultimately, I think these people will, by competitive pressures, end up in a similar place to where OpenAI is at. So it is quite concerning. But I think there is a period where, first, I mean, there's sort of two senses of what I mean by in the lab. One is like, after you've developed a system, you can study in the lab before you deploy it. A second thing is just, before you have a system capable enough to cause damage in the real world, you can construct situations in the lab that are useful metaphors, or that are like easier ways, you know, take over your little simulated environment or whatever. Like, have the, have the thing you can run with GPT-4 in the lab, that tells you something about what GPT-5 would do in the wild.

• 1:27:28Speaker 00

  There's also a chance, and I think like, so OpenAI has a lot of rhetoric of, for me, the only way to really learn about systems is by deploying them in the world, which I do find quite scary as rhetoric, because I think for some problems, it's a very, very rough approach. I think there is a reasonable chance, you know, like 50/50, that you see something really just very worrying in the real world, and then you can say, okay, now we're gonna roll back, or now we're gonna like study that issue that we observed in the real world for a while. It's not totally sunk if you just try and do these experiments with AI systems deployed at scale, but I think there is a reasonable chance, you know, more than a third chance, that the first really analogous concerning sign you see is an irrecoverable catastrophe.

• 1:28:01Ryan Sean Adams?

  Yeah, I gotta say that, that, you know, old adage, move fast and break things. That sounds okay for Web 2, but like, not for like, you know, nuclear physicists. Not, not for like, something like with dire straits, like AI necessarily. Move fast and break things doesn't make me too excited. But, okay, so we covered scalable oversight, we've covered-

• 1:28:22Speaker 00

  Some, some risks it's okay for.

• 1:28:23Ryan Sean Adams?

  Sure, sure.

• 1:28:24Speaker 00

  Some things you can break things, but yeah, takeover is not good.

• 1:28:26Ryan Sean Adams?

  Yeah.

• 1:28:26Speaker 00

  The irreversible catastrophe is not a good move.

• 1:28:28Ryan Sean Adams?

  We don't want that. The breaking things, we run into trouble there. Okay, so we, what is the third? And then maybe the fourth, Paul, of our, you know, bag of tricks here.

• 1:28:37Speaker 00

  Long list. I think a thing people care a lot about, but also seems really hard. All of these are like, they seem good, but really hard. And maybe at some point we should talk about like the boring stuff that might just work.

• 1:28:47Speaker 00

  A third thing people care a lot about is understanding what is going on inside these large neural nets. So you have GPT-4. Most of what we know, virtually, I think, everything that we know about GPT-4 is by just running it on a bunch of inputs and seeing what it does. In theory, you can also look at the exact computation the model performed. Like, it's kind of like neuroscience, except you have a complete readout of exactly how the brain works and exactly what it was thinking in every case. And so you could hope, though, with access to that information, you could do much better, and you could also invest much more. You can do much better than neuroscience has done on humans, and you can be able to say, we can learn about this model not only by observing its behavior, which is really hard, because it's hard to predict how it will generalize in some new case, but also

• 1:29:29Speaker 00

  by looking at the computation it performs, understanding why that computation leads to the behaviors you observe, and then reasoning about whether that mechanism would generalize it in an unpredictable way, or being able to use that knowledge to flag when the mechanism is behaving in an unpredictable or novel way. So this is a project, you know, a lot of, a reasonable number of people are working on, both in academia and industry and nonprofits. And that would be great. It could be great if we understood something about why GPT-4 said the things it said.

• 1:30:02Ryan Sean Adams?

  So, Eliezer kept talking about inscrutable matrices and using terms like gradient descent, which I noticed you used during the course of this. This is the thing we don't understand, right? We don't understand actually what's going on inside of the AI's quote-unquote "brain." We don't understand what these inscrutable matrices, how they work, or what answers, what goals that might emerge from them. Is this all part of the same thing?

• 1:30:27Speaker 00

  Yeah, I think that's basically why we're worried. That's exactly right. And that's basically why we're worried. Like, we took a model, we took a bunch of cases, we messed with the weights of this model until it did really well on the 100 billion cases that we considered. And now we wonder, what's it gonna do in some new case? In like, a case where, for example, models do have the opportunity to cause incredible harm, or could be able to get a high reward by causing incredible harm. And the scary thing is, like, you have no idea what the, we kind of understand how gradient descent works. It takes you to something that works really well on 100 billion cases you tested on, but we have no idea how the resulting model works. The resulting model is basically like, you know, 150 matrix multiplies. You multiply by a big ma, 300, 400, whatever. You multiply by a big matrix, and then you apply it on linearity, and then you multiply by a big matrix again, and then you apply it on linearity, and we're just like...

• 1:31:13Speaker 00

  We have no idea what any of the numbers and any of those matrices mean. And that's not totally true. I think we have some idea of what some of the numbers mean. But at a high level, if you take interesting behaviors, take a behavior of GPT-4, which does not appear in GPT-2, say. I think for essentially every such behavior, we do not understand how GPT-4 is able to do the thing. We understand like some simple things. And we don't understand any, yeah, we don't understand most of the complicated behaviors existing models engage in. We have no, you could not, if you gave us the list of matrices and asked us like, does the model do X? We would have no way to answer that question other by running it a bunch of times and seeing if it did X. Which again is not great. That sort of gets you back to like, now you need to somehow either run in the real world and see what happens and hope it's not catastrophic.

• 1:31:55Speaker 00

  Or be able to construct simulated situations in the lab that are similar enough that the model behaves the same way there that it would in the real world. You'd really love to not have to do that.

• 1:32:03Ryan Sean Adams?

  So there is maybe some hope that we can start understanding what's going on in these inscrutable matrices. But we haven't made major breakthroughs yet. What is sort of the fourth category?

• 1:32:13Speaker 00

  We've made, we've made progress.

• 1:32:14Ryan Sean Adams?

  Progress, okay.

• 1:32:15Speaker 00

  It seems like a lot of progress you have to make. I mean, the Eliezer probably disagree since Eliezer is probably at like, you know, 1% or 0.1%. This would get far enough to meaningfully reduce risk. Whereas I'm probably more at like 10%. This gets far enough to meaningfully reduce risk, maybe higher, depending exactly what you mean by means. Maybe I think there's like 5, 10%. This is good enough to totally address the risk. And like 10 to 30%, this is good enough to take some meaningful, meaningful reduction of risk. Anyway, that was third category. I mean, the fourth category, I think is pretty promising is just ultimately what we're wondering is if you have a bunch, you train your AI in situations A, where you're able to train it. You're able, if it fails, it won't be catastrophic. You're able to evaluate what the answer is. Then if you deploy it in situation B, where either you can't tell what the right answer is, or if it failed, you wouldn't be able to fix the problem.

• 1:33:02Speaker 00

  And then we want to understand how do models tend to generalize from like this kind of easy cases or cases where we're able to supervise to cases where we're not able to supervise. And you could hope to just build up a good scientific understanding of that question. You could hope to say like, we're going to have a bunch of cases. We're going to just have looked at a ton of models, understand what factors affect this generalization. This is very similar to like, if you imagine these two humps of like a hump of good behavior, which looks good because it is good. And a hump of bad behavior, which looks good because it's systematically corrupting measurements or deceiving humans. Kind of want to understand what are the conditions that determine which of those, like, do you make the jump from one to the other? Some sense there's two equally valid generalizations and you're wondering which one do you get? And I think there's just a lot to do with having situations which have similar ambiguity about generalization, situations where unsure about how the model will generalize,

• 1:33:46Speaker 00

  training huge numbers of models and understanding what factors determine whether and when the systems generalize one way versus the other. And then using some of what you learn either to diagnose risk, or in the best case to say, okay, if we train the model in the following way, if we use the following kinds of loss functions, we get the intended generalization. And I think that is reasonably likely in combination with other concern. I think it's reasonably likely to work. Like, again, I said, just naively, there's maybe a 50% chance you're okay. And maybe, you know, if you do a lot of work like this, you can bump that up to like 60% chance that you're okay.

• 1:34:17Ryan Sean Adams?

  So this fourth category, what do we call this, Paul? And this is the one that you're most optimistic about?

• 1:34:23Speaker 00

  Oh, I don't know. I don't know what I'm most optimistic about. I think these four seem like broadly similarly important. I don't know what you would call this. It's like studying generalization or something. This is, again, a question academics are very interested in. They study in some ways. They mostly don't study the versions that are most relevant to takeover. There's some people who are very interested in takeover in particular who study this question. It's not something I've worked on myself. I'm pretty optimistic about it, but I'm pretty optimistic about interpretability. I'm pretty optimistic about scalable supervision. I'm reasonably optimistic about robustness. It seems hard, but I think it's reasonable chance of helping a lot with this problem.

• 1:35:00David Hoffman?

  So Paul, I see you've laid out four different technical solutions here. And I'm very naive on this subject, but I don't see, and tell me, please, how many people, what's the manpower behind these things? Because it's great that we have these paths, but we need manpower to actually go and execute on this.

• 1:35:21Ryan Sean Adams?

  Yeah, it feels like you guys should be funded, like billions of dollars funded in solving this problem, because we've got a lot of funding on developing AIs, don't we?

• 1:35:32Speaker 00

  I think it's hard to have an amount of funding for this problem that is similar to the amount of funding for developing AIs, just because you got some pretty good profit incentive on the making AIs. I think we can get, yeah, I think it's a reasonable amount of funding. I think you're really talking more like hundreds of millions or billions of dollars over, you know, the foreseeable future available for solving it. Maybe you could amp that up if the problem became more real. Like right now, most people are just like, look, there's probably not going to be a takeover in the next couple of years. It's very speculative risk in the long term. What can you really do in advance? Anyway, there's some money.

• 1:36:04Speaker 00

  There's a shortage of people who are excited, like, there's also a shortage of scientists who are excited about doing this work. And I think that's like hopefully changing quite rapidly as the problem seems more real and AI seems more exciting. And people are shifting more into the space. In addition to shifting into AI, I think a fair number are shifting into understanding various risks, some fraction of whom care about takeover.

• 1:36:24Speaker 00

  So I'd say like, in terms of estimating how big this is right now, it depends a lot how you count people who are not motivated by takeover, but do work that can still be relevant to reducing takeover.

• 1:36:35Speaker 00

  And just like, what, do you apply discount, do you apply no discount? Maybe I'd say that you're talking like on the order of

• 1:36:44Speaker 00

  maybe 20 people on scalable supervision stuff.

• 1:36:49Speaker 00

  Maybe 20 people on the interpretability stuff that's most relevant to takeover, coupled with like

• 1:36:55Speaker 00

  100 or a couple hundred people doing stuff that could be relevant to varying degrees.

• 1:37:01Speaker 00

  On robustness, like,

• 1:37:04Speaker 00

  maybe again, looking at something like five to 10 who are like motivated explicitly by addressing takeover risk, followed by, you know, a couple hundred who are doing stuff that's positively relevant, of whom maybe like a couple dozen are stuff that I would actually care a lot about or would think is highly relevant.

• 1:37:24Speaker 00

  And like on the generalization stuff, maybe again, looking at like five-ish people who are doing it motivated by takeover risk, plus another, you know, on the order of dozens, who are doing stuff that could be relevant or helpful. So maybe in total, you're looking at something like 50 people, 200 people who are motivated by takeover risk explicitly in these areas and other adjacent areas, together with like hundreds more who are doing work that is hopefully, hopefully relevant.

• 1:37:50Ryan Sean Adams?

  Paul, in the scheme of things, this is not that many, though. Not that many people here.

• 1:37:56Speaker 00

  Yeah, it's not that many people. Certainly small relative to AI. And I would love it. I mean, I am like, there's a reasonable chance we're all gonna die.

• 1:38:04Speaker 00

  I think this is like the single most likely reason that I will personally die, probably.

• 1:38:09Ryan Sean Adams?

  Wow. So like, that's big. And this is someone working on AI safety. Can I ask you another question? So we've covered the technical. Just want to brush on the other point of coordination that we're making around, you know, policy and human coordination.

• 1:38:25Ryan Sean Adams?

  There has been an open letter, which I'm sure you're familiar with, pause giant AI experiments in open letter. Max Tegmark, I think his organization put this together. Elon Musk signed it, Andrew Yang, some others have assigned this.

• 1:38:38David Hoffman?

  Noa Harari.

• 1:38:39Ryan Sean Adams?

  Yeah. And basically, the idea is that we should, the open letter states that we should pause all AI development for six months just to wrap our arms around this AI safety issue. And so let's go no further than ChatGPT four until we pause six months and all take a breath and figure out what this means. Do you support a letter like this, given you're in kind of an AI safety? So there's a question of whether you support, you think this is a good idea, this coordination mechanism. And there's also a more meta question of, do you think, what, do you think we can actually solve this coordination mechanism? And, you know, are we in some, you know, Scott Alexander level Moloch trap? That makes it very difficult for humanity to solve.

• 1:39:24Ryan Sean Adams?

  Is this the don't look up scenario that David was painting earlier? So first of all, have you signed this letter? Would you sign this letter? Do you think it's a good idea?

• 1:39:33Speaker 00

  I didn't sign the letter. I can give my overall take, which is something like, I think it would, on balance, be better to pause ad development or slow ad development now. I think that is not at all obvious. And I sympathize with people who think that it's a bad idea on balance to slow down. So we can talk about why. I think the like dominant thing I care about is I think that. At some point, it will become like if we're actually developing systems that pose significant risks and our measurement is not adequate to tell if they post significant risks or our measurements suggest they do post significant risk.

• 1:40:06Speaker 00

  I think at that point, I'm going to have a much more forceful take of like, right now, I'd kind of be like, I kind of think it should, anyway. Right now, I think it's like a debatable issue and it's reasonable to want to slow. I think at some point in the future, it will still be a debatable issue, but I don't think it is unreasonable to not want to slow.

• 1:40:22Speaker 00

  And that like, we actually kind of collectively really need to act on this. I think the main thing that matters, I kind of agree with Elliot's take that the six month pause does not actually help that much. That it does not reduce risks super far. I think the main thing we need to do is get in a position where we are prepared to slow down based on risk. Like build consensus about risk or consensus that we need to have measured risk. The risk is currently high enough that our measurements are unacceptable. And then be ready to slow down potentially much more than this, potentially six months, potentially like whatever it takes. As we manage that risk or like slow down the directions that are most risky. That's like the main thing I think about and the main thing I care about. And the main thing that I've like really liked, I think

• 1:41:04Speaker 00

  that is going to be like a kind of delicate process and that I think it is quite expensive in terms of human cost. I mean, I think probably I'm more optimistic about AI in some sense than most people. But I think that like slowing AI development by years would probably, yeah, would come with a kind of tremendous human costs. Which I think is worth paying, but is, I sympathize with people in AI who are skeptical of that or who don't take it lightly.

• 1:41:30Speaker 00

  I think it's going to be like, I think we can get to the point where there's kind of consensus that we do need to slow. The risk is unacceptable, that the benefits of going faster are not that large compared to what's at stake.

• 1:41:41Speaker 00

  I think the game is getting to the point where we are, we are having that discussion. We have measurements in place and we have institutions that are set up such that they can slow. I think that like there is some amount of slowing that can happen by voluntary self-regulation amongst Western labs. That is, I think there's a reasonable room. I think a nice fact about the situation is most of the people involved in AI development now do, I think genuinely not want AI to go over and kill everyone and do have some appreciation of the risk, at least in principle, and are open to saying, okay, here's a set of practices which will adequately manage that risk and we will adopt them even if we slow. And I think we can get, you know, you can't get that to slow down that way, but I think you can get significant additional safety and plausibly something like, you know, six to 12 months of slowing of potential catastrophe

• 1:42:26Speaker 00

  just from people at a couple labs saying, we don't really want to cause incredible catastrophe. We see the case for slowing. I think maybe to even do that, and especially as you want to go beyond that, like you really need to have something more like a regulatory regime where we have said, you know, there was this voluntary set of practices, the labs endorsed. Some people are behaving in a way that doesn't comply with those and there's kind of broad consensus that's not reasonable, at least amongst, you know, anyway, this is amongst some group of that. And you can then, you know, this is in scope. Like the thing that we're asking the state to do here is say, you really don't want an AI lab to be in a position where they're like violently overthrowing the US government. This is not like a crazy, it's definitely like in the, in the government's wheelhouse. It is their job. And so to the extent that like some companies are like, no, we should push ahead. I think there's

• 1:43:11Speaker 00

  reasonable grounds for the world to be like, this is not the kind of thing that you get to just push ahead with. I think that's hard now. I think procedurally now is a little bit hard to make the case that like the state should crush AI development. I think there'll come a point in the future where it's not that hard to make the case that there are some developers who are quite risky and actually it is not reasonable. That's not a reasonable behavior and it's not.

• 1:43:29Ryan Sean Adams?

  Has anyone proposed something, Paul? Because when you say there's only like 50 to 100 people working on AI safety and this is like, I mean, I'm sure millions in funding, but not hundreds of billions that AI development is actually going to have. You know, it makes me wonder if some sort of like an AI tax has been proposed or something where some percentage of profit from, you know, AI utility goes to sort of a fund that just, you know, pays for research. I mean, this happens at the government level, something like this because. It does seem like we have a public goods funding type of problem. Maybe it's not just that, maybe we also have an education problem.

• 1:44:08Ryan Sean Adams?

  This is why David and I are sort of looking into AI quite aggressively after our episode with Eliezer. It's just like, all this crypto stuff doesn't really matter in the scheme of things if the robots are literally coming to kill us, right? We should like, think hard about that. Cool, we have like crypto systems and decentralization and crypto economic tools, but we're all dead. Now the robots have them. Now the robots have them. Great job, guys! And so we're taking a quick detour here, but like, it's honestly because we've just been made aware of the stakes here and the existential threat. So there's also an education game here, Paul. I don't know, is... Where do you think the resources should be spent?

• 1:44:53Ryan Sean Adams?

  Should it be on, you know, regulation, education, like broad strokes level, what can we do?

• 1:45:00Speaker 00

  I think an important thing to have in mind going into that, like when I say 50 to 100 people working on takeover, I think there's a lot of things we might care about in AI safety. There's a lot of possible risks from AI, a lot of possible harms. A lot of them are things that are like, when you deploy ChatGPT, there are ways in which people might be unhappy with the outcome. There are privacy issues, there are like various effects, like systemic effects on society from its deployment you might be worried about. There's like many more people working on these issues broadly, just because there are a lot of issues. I think every issue is able to feel like this issue is like crazily neglected, but it's worth flagging like AI safety is this much broader category of which like the risk of AI takeover is a minority of people who are kind of right now. I think it's a larger share of public discourse than it is of scientific interest.

• 1:45:45Speaker 00

  So I think that like things like public spending, I think right now probably the principal bottlenecks are not spending more money. I think it is useful, having more money does, it's not like there's no need for more money. More money is good, there's lots of things one could fund, but I think that it is. The key bottleneck is probably having projects that are appealing, like having people who have relevant background to perform projects, who are sufficiently excited about doing work to try and manage this risk, that they are in the step of asking for money, or they would do it if money was available.

• 1:46:12David Hoffman?

  It's talent. Talent's a bottleneck.

• 1:46:14Speaker 00

  I think that's the big problem right now. I mean, it's, again, neither's really a bottleneck. I think like you can, you can spend money to like, help people switch into the field to fund projects that are more speculative and more long shots,

• 1:46:27Speaker 00

  to increase the incentives. And like, some people will switch, even if your funding work would have been done anyway. If you fund it more generously, more people will get into the field inevitably. So there's like a lot of ways you can use money. And like, it's not exactly clear whether one should be trying to have more money available or more talent available. But it is like a little bit rough right now. I think if you're trying to spend money and trying to look like, where do you spend that money? It's hard to spend money on a problem that like practicing, like practitioners in industry and scientists in academia are not prioritizing. Most successful spending comes from having like people who want to do the work, who like need the money to do the work, or at least are open to doing the work and are somewhat excited about it.

• 1:47:00David Hoffman?

  So Paul, if there are talented, what, what kind of talent is really missing? Like, what is the archetype of someone that this area of AI really needs?

• 1:47:11Speaker 00

  I think there's a lot of kinds of work to be done. So we've talked mostly about technical solutions, where like mostly the relevant kinds of talent are either people who have mathematical backgrounds or backgrounds in computer science, or experience in machine learning, or are good engineers, or good design. I think a very common pattern, like a lot of people who work in this field are people who came in from some other area, like who used to do physics. I think it's like, if you're doing physics and right now in the world, it's reasonable to say like, you know, we can pause the physics for a little while. This AI thing is going to be urgent for the next like 10, 20 years. And so I think like, it is reasonable that people shifting, who just have broad scientific backgrounds, who have experience doing research, understand how to study complicated empirical questions. Things like a broad range of technical backgrounds that slot somewhere into this picture. And if you broaden your scope from just those technical issues to like the whole, the whole picture, then there's an even broader range of people, right?

• 1:47:57Speaker 00

  There's a lot of work. It's like institutional work or understanding what like, how should we approach this measurement thing? Or like, can we make progress on this like public discussion or public advocacy? Can we understand just like general forecasting of like, what is the state of play? To what extent are objections to this reasonable, like actual reasons to have lower, like? Which of the things people would say as objections are like actual considerations that should change your view versus targets for advocacy? We just want to try and change views and like, what is a reasonable view in light of, you know, what is actually the right synthesis of consensus? If you take the things experts believe that are most across different fields that are most relevant? I think there's just like a lot of stuff to do. The one I understand best is definitely like, I don't know, there's probably like 500 reasonable projects in AI safety or something. And there's not that many people working on them.

• 1:48:40Speaker 00

  So just like people coming in, looking at the landscape, seeing where they can fit in, trying to do some projects in that space, whether it's in interpretability or in scalable supervision, or just studying what, how models generalize, or other science about models, or understanding work on robustness. And I gave those four categories. That's not an exhaustive breakdown. For example, that does not actually include my day job and all the work that I do normally.

• 1:49:03David Hoffman?

  What's your day job and what's the work that you do?

• 1:49:06Speaker 00

  It's primarily on trying to develop alternative training strategies, or just qualitatively new techniques that don't have this issue, like don't incentivize takeover in the same way that current techniques do. I think it's like, you know, it's one thing that goes in the portfolio. I think, you know, there's a 10% chance that we're able to come up with like a really good strategy that does qualitatively change the game. And like, if risks seem large, we could adopt it. That's like another category of work. Some people work on that. I think it will, it's less likely to absorb huge numbers of people and it's a little bit more of a long shot, but I think it's a good thing to do.

• 1:49:38Speaker 00

  Yeah, so that's my high level. There's just a lot of projects. I think most projects, if we go through that list, like almost all of those projects are hurting for some combination of technical talent doing them, like senior researchers who are experienced, have research experience and are able to like bring good judgment and mentorship to these problems. Management or like entrepreneurial spirit, or like management experience to like actually onboard people and coordinate projects, working on them. And it's just like an incredibly high premium to people who have technical backgrounds and are like entrepreneurial enough to like, look at the space, engage with like people's thoughts about what helps, form out their own views that are reasonable, and then start on projects. Like the returns to doing that right now are just crazy high, I think.

• 1:50:15David Hoffman?

  Right. Paul, what, what would you say were the odds to be, and this is a, I know this isn't your field, but like the odds that somebody in this field wins a Nobel Prize in like the next 20 years?

• 1:50:27Speaker 00

  Oh, I think they seem pretty, you mean, someone working on like ALM and stuff. I think our first problem is that like, there's no Nobel Prize in any adjacent area. I guess it would be like.

• 1:50:36David Hoffman?

  Hmm. I guess my point is like.

• 1:50:37Ryan Sean Adams?

  There's no prize for saving humanity. I mean, we should, we should work on that, right?

• 1:50:41David Hoffman?

  Yeah, there's a prize somewhere if for someone solves this problem.

• 1:50:45Speaker 00

  Yeah. So you have like, you have like Turing Awards or Fields Medals or whatever, maybe more analogous thing you'd think of. I don't know. I mean, I think it's not that likely. 20 years is not that long a time horizon. Most of these things tend to be like given latent careers for early career achievements. Although like Fields Medals are a bit of a distinction. Most of the stuff people are doing is not really in a category that would get a Turing Award or Fields Medal.

• 1:51:07David Hoffman?

  It was more just like, there's, there's, table stakes should be that there's, should be some big recognition for whoever solves this problem. So like, that's, maybe that's my call to action for.

• 1:51:16Ryan Sean Adams?

  Somebody make a prize. Somebody make a prize real quick. So we can save humanity.

• 1:51:20Speaker 00

  That is a thing you could do with money. And it's like, not a crazy thing to do with money. It's like also a thing that sort of, in some sense, scientific prestige is one of the inputs. Like a prize could be made of either money or prestige. It's harder to synthesize the prize made of prestige. And that's more like, is the scientific community bought in?

• 1:51:35Speaker 00

  I mean, at a high level, I think it is fairly likely that in retrospect, in like the maybe half of worlds where this was a real problem, that in retrospect, people will be very excited about some fraction of the work that was done in the area. People will be like, that was a big deal. Sorry, we're asleep at the wheel. Oops. So I'm like, I mean, I think it's 50/50 chance that people will feel that way in retrospect, generally. The question of like, is there any kind of existing institution set up which would provide significant recognition? I'm like, less, less sure about that. I mean, I hope like the work we do, you know, that's like good academic work. I think it has like probably higher than the base rate for like the era of computer science of, of winning prestigious prizes. But it's kind of just like those processes on academic merit rather than that.

• 1:52:14David Hoffman?

  I think we can go ahead and zoom forward into the future where if, if humanity does thread this needle, there will be an institution that does award a prize. I think we can run on that assumption. Paul, thank you so much for guiding us through. This was exactly the episode that we wanted to produce. And I think I got a lot of my answers, my questions answered, but I have one last question for you. My strategy late thus far, up to this point, has been to just be really polite to Siri and Alexa. And I'm wondering if that is, in your opinion, is moving the needle for me, me specifically. I don't care about the rest of the world, if that does anything at all.

• 1:52:56Speaker 00

  I'm gonna go with probably no. That is, I think there is some real thing about humanity treating AI systems with like respect and dignity and being like, look, these things are gonna be. I think there's a lot of room for humanity to do wrong by AI systems we create that are smart. I think being nice to Siri and Alexa is probably not even the place you most personally could help with that. They probably don't.

• 1:53:16Ryan Sean Adams?

  I do hope.

• 1:53:17David Hoffman?

  Well, my philosophy is like, at some point, these things will be AIs. There will be an AI on the other side of that robot, that microphone. And my philosophy is, why not start treating it with respect now?

• 1:53:28Ryan Sean Adams?

  I think the bigger question.

• 1:53:29Speaker 00

  I think it's not crazy.

• 1:53:30Ryan Sean Adams?

  The bigger question is, how will they look upon this podcast? Will they look favorably upon this podcast and our body of work at Bankless so far, David? That might be the bigger question in my mind. And I, one last concluding question for me, Paul, is because you said 20% doom scenario, which leaves 80% for non doom scenario. And I bet there's some like mediocre scenarios in there, too. So tell me about the 20% utopia scenario. Like, is there the possibility this all goes really, really well? And what happens on the other side? Leave us with some optimism here.

• 1:54:05Speaker 00

  I'm not that good a man of leaving people with optimism. I think maybe I'm like 50, more like 50/50 that humanity achieves like a very good outcome. That is an outcome where we're like, this is about as good as we could have expected it to be. Maybe a little bit less than that, but more like 50/50 than 20%, I think. So that's optimistic. And I'm just, I think it's really hard to talk about like what the like political economy of that world is like over the very long run. I think the big thing is like, humanity, I think, still has a long history in front of us. I think AI means that like that history is probably gonna get compressed. Like, I think there's a long time for institutions to change and things to happen. I think a lot of that is going to happen much, much faster than it would have if you're thinking about like tens of thousands of years of human history to date. I think you're probably thinking more like, you know, tens of years before the world is like very, very radically transformed.

• 1:54:50Speaker 00

  I don't really know what that world looks like. I think that like a lot of human problems are like humans versus nature. Like, we die of old age and disease, and people have physical want and stuff. And I think that that is probably gonna be really much better. I think that's like even more than 50%. The problems that are caused by humanity versus nature are going to be pretty good. And I think those are, you know, I think our lives would be quite good. In some sense, human versus human conflict is mostly problematic because there's a limited, there's a limited budget to go around. And then beyond that, it gets really hard to say what the character of that world is like and what we choose to do if in fact we're in like a pretty good position physically, if we have incredible resources at our disposal, what kind of world we make. That's a longer and more complicated discussion. But I'm pretty psyched. I mean, personally, I'm just, I'm very glad I'm living now instead of

• 1:55:33Speaker 00

  any time. And I would definitely take like 50% chance of death. 50% chance of early death seems like kind of very small compared to the overall change in expected quality of life from being alive.

• 1:55:44David Hoffman?

  Wow, that is extremely optimistic.

• 1:55:46Speaker 00

  I'm much more optimistic than Eli Ozer, definitely.

• 1:55:48David Hoffman?

  Yeah, that's for sure.

• 1:55:49Ryan Sean Adams?

  We've got a coin flip here. Either it goes very poorly or it might go well for us. And Paul, thank you for the work that you're doing to make this go well. It's been a pleasure to have you on Bankless.

• 1:55:59Speaker 00

  Thanks for having me. It was great talking.

• 1:56:01Ryan Sean Adams?

  Guys, Paul's website, we'll leave it in the action items for you, the Alignment Research Center, that's at alignment.org. You can check out what his organization is doing. Also, we'll include a link to Paul Christiana's website with some of his writings, including, I think, some links to some of his debates with Eli Uzki from the archives as well. Got to end with this, of course, none of this has been financial advice, but you got to know that. We were talking about AI. We've mentioned crypto a couple of times, but I'll just end with this. AI is risky. The stakes are high. We could lose a lot here, but we are headed west. This is the frontier. It's not for everyone, but we're glad you're with us on the Bankless journey. Thanks a lot.