Revolutionizing AI: Tackling the Alignment Problem | Zuzalu #3

In this episode, we delve into the frontier of AI and the challenges surrounding AI alignment. The AI / Crypto overlap at Zuzalu sparked discussions on topics like ZKML, MEV bots, and the integration of AI agents into the Ethereum landscape. However, the focal point was the alignment conversation, which showcased both pessimistic and resigned optimistic perspectives. We hear from Nate Sores of MIRI, who offers a downstream view on AI risk, and Deger Turan, who emphasizes the importance of human alignment as a prerequisite for aligning AI. Their discussions touch on epistemology, individual preferences, and the potential of AI to assist in personal and societal growth. ------ 🚀 Join Ryan & David at Permissionless in September. Bankless Citizens get 30% off. 🚀 https://bankless.cc/GoToPermissionless ------ BANKLESS SPONSOR TOOLS: 🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE ⁠https://k.xyz/bankless-pod-q2⁠ 🦊METAMASK PORTFOLIO | TRACK & MANAGE YOUR WEB3 EVERYTHING ⁠https://bankless.cc/MetaMask ⚖️ ARBITRUM | SCALING ETHEREUM ⁠https://bankless.cc/Arbitrum⁠ 🛞MANTLE | MODULAR LAYER 2 NETWORK ⁠https://bankless.cc/Mantle⁠ 👾POLYGON | VALUE LAYER OF THE INTERNET https://polygon.technology/roadmap ------ Timestamps 0:00 Intro 1:50 Guests 5:30 NATE SOARES 7:25 MIRI 13:30 Human Coordination 17:00 Dangers of Superintelligence 21:00 AI’s Big Moment 24:45 Chances of Doom 35:35 A Serious Threat 42:45 Talent is Scarce 48:20 Solving the Alignment Problem 59:35 Dealing with Pessimism 1:03:45 The Sliver of Utopia 1:14:00 DEGER TURAN 1:17:00 Solving Human Alignment 1:22:40 Using AI to Solve Problems 1:26:30 AI Objectives Institute 1:31:30 Epistemic Security 1:36:18 Curating AI Content 1:41:00 Scalable Coordination 1:47:15 Building Evolving Systems 1:54:00 Independent Flexible Systems 1:58:30 The Problem is the Solution 2:03:30 A Better Future ----- Resources Nate Soares https://twitter.com/So8res?s=20 Deger Turan https://twitter.com/degerturann?s=20 MIRI https://intelligence.org/ Less Wrong AI Alignment xhttps://www.lesswrong.com/tag/ai-alignment-intro-materials AI Objectives Institute https://aiobjectives.org/ ------ Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures⁠

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:04David Hoffman?

  Welcome to Bankless, where we explore the frontier of internet money and internet finance. And today, on this episode of our Zuzalu series, we are exploring some new frontiers, new frontiers and new technologies, all of which are poised to completely revolutionize the world and change everything about the operating system that society is currently running. Bankless Nation, today we are exploring the frontier of AI, which is actually a frontier that we've already been exploring on Bankless. So if you've been listening to our other AI episodes, these will make you feel right at home. AI had a big week at Zuzalu. The AI crypto overlap, everyone knows is huge, and it seems like such a massive frontier that people don't actually know where to start with it. ZKML, or machine learning models and data that's verified by zero knowledge cryptography,

• 0:49David Hoffman?

  was a huge topic of conversation. And you'll hear about that in our cryptography episode with Daniel Short. Phil Dian at AI Week gave a killer talk titled "MEV for AI People", which was this giga-brained presentation about how MEV bots in aggregate kind of presents this omnipotent, omnipresent artificial intelligence. And since MEV has been decently corralled and contained, maybe we can learn a thing or two from the MEV industry in our approach to managing AI risk. There were conversations at Zuzalu about how AI can put the autonomous back into DAOs, and how AI agents could soon be roaming the Ethereum landscape, shoulder to shoulder with all the human players out there. But mainly, at

• 1:31David Hoffman?

  Zuzalu, the AI conversation inevitably converged into the alignment conversation, of which you will find two flavors here in this episode. One, strongly pessimistic, and then the other characterized by this resigned optimism that is prevalent throughout all of Zuzalu's frontier tech challenges. Unimaginable rewards blocked by seemingly insurmountable obstacles. Up first in this episode, we have Nate Soares, who is the Executive Director at Miri, the Machine Intelligence Research Institute, of which Eliezer Yudkowsky founded. Nate's perspective on AI and AI risk is definitely downstream of Eliezer's. So we pick up where Bankless left off with Eliezer. And Bankless Nation, it's dark.

• 2:13David Hoffman?

  But nonetheless, Nate admits that it's less dark than it was a few months ago, now that the world is waking up to the potential risk that AI brings to this world. Following the conversation with Nate is Deger Turan, who is charging into the AI frontier with his head held high, with a clear path forward for himself. Deger believes that the AI alignment problem is actually just downstream of human misalignment, and that we actually won't be able to align AI until we align ourselves. This conversation has to do with epistemology. What is truth? Individual preferences and how AI models can help us become the best versions of ourselves. Because if we become the best versions of ourselves with the assistance of some AI tool, we can collectively produce the best versions of our communities.

• 2:55David Hoffman?

  And if we do that, then our communities can coalesce into the best versions of society, all aided by truth-telling AI agents, who can help humans navigate through our chaotic world of social organization and politics and social media. Really a fascinating conversation that is actually pretty proximate to our conversation with Tim Urban that we had not too long ago. Really excited for you to listen to these conversations, bank conversations, so let's go ahead and get right into it. But first, a moment to talk about some of these fantastic sponsors that make this show possible. Kraken Pro has easily become the best crypto trading platform in the industry. The place I use to check the charts and the crypto prices, even when I'm not looking to place a trade. On Kraken Pro, you'll have access to advanced charting tools, real-time market data, and lightning-fast trade execution, all inside their spiffy new modular interface.

• 3:39David Hoffman?

  Kraken's new customizable modular layout lets you tailor your trading experience to suit your needs. Pick and choose your favorite modules and place them anywhere you want in your screen. With Kraken Pro, you have that power. Whether you are a seasoned pro or just starting out, join thousands of traders who trust Kraken Pro for their crypto trading needs. Visit pro.kraken.com to get started today. MetaMask has something new. Introducing MetaMask Portfolio. MetaMask Portfolio is the best way to view your crypto portfolio from a holistic level. See everything across all the chains all at once. In your portfolio, MetaMask will report the aggregate value of all the assets in your MetaMask wallets, and even the other wallets you import too. But MetaMask Portfolio isn't just a passive portfolio viewer.

• 4:21David Hoffman?

  It is a place to do all of the money verbs that make DeFi so powerful. You can buy, swap, bridge, and stake your crypto assets. So not only is MetaMask the easiest place to see your wallets in aggregate, but it's also a powerful battle station for all of your DeFi moves. So go check out your MetaMask portfolio, because it's waiting for you to open it up. Check it out at portfolio.metaMask.io.

• 4:43David Hoffman?

  Arbitrum is accelerating the Web3 landscape with a suite of secure Ethereum scaling solutions. Hundreds of projects have already deployed onto Arbitrum 1. With a flourishing DeFi and NFT ecosystem, Arbitrum Nova is quickly becoming a Web3 gaming hub, and social dApps like Reddit are also calling Arbitrum home. And now Arbitrum Orbit allows you to use Arbitrum's secure scaling technology to build your own Layer 3, giving you access to interoperable, customizable permissions with dedicated throughput. All of these technologies leverage the security and decentralization of Ethereum and provide a builder experience that's intuitive, familiar, and fully EVM compatible. Faster transaction speeds and significantly lower gas fees. Are you a dev, but you don't know Solidity? With Stylus, Arbitrum's upcoming proposal for a programming environment upgrade, developers

• 5:26David Hoffman?

  can write smart contracts in Rust, C, C++ and many more coding languages. Arbitrum empowers you to explore and build without compromise. Visit arbitrum.io, where you can join the community, dive into the developer docs, bridge your assets, and start building your first app on Arbitrum. Bankless Nation, we are here with Nate Sores and we are starting the AI Week here at Zuzalu. And Nate is an AI researcher, is that how you would call yourself? Alignment researcher?

• 5:50Speaker 01

  I would say I'm the Executive Director of the Machine Intelligence Research Institute. These days, it's less research than I'd like. I have done alignment research before.

• 6:01David Hoffman?

  Okay. Can you explain that institution, the institute? What is that?

• 6:06Speaker 01

  So I didn't found it. It was founded by Eliezer Yurkowsky. I don't know exactly when, maybe around 2001. Fun fact, it was originally founded, it was originally called the Singularity Institute and it was founded because Eliezer wanted to make AGI as fast as he could. And then along the way, he realized that it doesn't go well by default and it doesn't go well for free. And so then the organization pivoted to trying to make this AI stuff go well. And for many years, the institute did some research, did some like field building, did

• 6:49Speaker 01

  some awareness raising and so on and so forth, until around 2012, 2013, when they pivoted to pure technical research. And this was related to

• 7:02Speaker 01

  Some of the field building, some of the awareness raising, moving to other groups as the field got a little larger. And I got involved when they pivoted to the technical research more exclusively. And so I was originally involved as a technical researcher. And then when the previous executive director left, I was the heir apparent.

• 7:24David Hoffman?

  Hmm. Okay, sorry, the machine, what's the name of the institute?

• 7:28Speaker 01

  Machine Intelligence Research Institute, AKA MIRI.

• 7:30David Hoffman?

  MIRI, okay. So it sounds like MIRI has its own trajectory of itself that probably runs in parallel with like human understanding with machine learning at large?

• 7:40Speaker 01

  Perhaps, but also perhaps quite compressed. I think, I don't know the exact years, I wasn't around then, but I think it was only a couple of years of work before Eliezer was like, hey, wait, this could get tricky.

• 7:55David Hoffman?

  Right, okay, so I actually didn't know that about Eliezer, that first he was like, AGI as fast as possible, and then he was like, whoa, whoa, whoa, AGI as slow as possible.

• 8:04Speaker 01

  Yeah, I think, I think, I'm not sure as slow as possible, but like AI done correctly, AGI done correctly. I think, you know, we were hoping for a long time, like, one of the reasons we do technical research is that, like, you can often, like, often your lever is just like solve the problem. Like screws slowing people down, that like pisses people off, it's like, like slowing people down is sort of a last resort. The, the original hope was can we just like solve the problem in time? It doesn't look like we're on track to solve the problem, and it looks like we have less time than, you know, I was hoping back in 2014. And so I think it is with great sadness that people like Eliezer and I are now saying

• 8:49Speaker 01

  we need, we need time, we need more time.

• 8:52David Hoffman?

  Can you kind of walk us through your, your own trajectory? How, how do you, how did you become the executive director at MIRI?

• 9:00Speaker 01

  Well, if you want to go back far enough, I, at a pretty young age,

• 9:10Speaker 01

  realized that the world was not very well organized and wasn't...

• 9:18Speaker 01

  I was in a civics class and up until that particular civics class, I had some intuition that like there were lots of problems in the world, but people were sort of trying to fix them. And the reason there were still all these big problems in the world was that we didn't have the technology. We didn't have the... Like, we were still a young race. We were still a young species. We hadn't like matured to the point where we could fix these issues. This was sort of like an implicit, wordless intuition rather than a conscious belief. And then, you know, I started learning how the US government works. And I was like, oh, God, it's like run by a bunch of monkeys. Like, it's like monkeys invented like monkey systems.

• 10:01Speaker 01

  And it's all like working about as well as you'd expect if it was like invented by people who had no idea what the hell they were doing and then like allowed to run for like hundreds or sometimes thousands of years and just like careen off into various... And so I was like, okay, like, obviously...

• 10:16David Hoffman?

  And if in the Moloch and crypto world, we would call this coordination and coordination failure.

• 10:19Speaker 01

  Totally, yeah. So I was very interested in solving coordination failures and more generally in making the world a better place. And I tried my hand at various versions of that while I was like pretty young and bad at things. And, you know, it's a hard problem. None of it worked. And the sort of circuitous route is that ultimately I got a job in tech while I was like trying to find ways to really move the levers on that problem. Decided to donate a decent amount of money to good causes to sort of keep me honest about actually trying to make the world a better place.

• 11:04Speaker 01

  Was trying to research where the best places to put money were. Donated to some like global poverty type charities and then started bumping into these arguments that like maybe this AI stuff is actually one of the biggest places to intervene on the world. I sort of read up on it. There were a couple other factors in my life that were also causing me to notice this AI thing. I read up on it and I was like, oh geez, like this is just obviously, you know, I was looking at the wrong problem. Like the coordination problems are big and they're real, but like this AI thing, you know, like humanity like

• 11:46Speaker 01

  lives or dies and gets like a great future or no future, depending how this AI thing goes. And I just like completely missed this problem for like eight years of thinking myself as trying to make the world a better place and going for the heart of the problems. But.

• 12:04David Hoffman?

  So when you ran into the AI topic, did you see, did you first see AI as a solution to all of our coordination issues or AI as a problem for all of our coordination issues? Or did you see it simultaneously at the same time?

• 12:18Speaker 01

  A little bit of both. I was maybe somewhat primed towards understanding some of the issues with AI due to my work on coordination problems. It's like slightly embarrassing, but I was like working on like various coordination mechanisms that could address the sort of concerns people had at the time. Like, how can a well-coordinated society without coercion address, for example, like concentration of wealth in ways that the society as a whole doesn't like? And you can set up various coordination mechanisms of like, yeah, you can sort of try to think about like what are non-coercive ways that a society as a

• 13:02Speaker 01

  whole can like try to both have a market system and not let it get out of control in certain ways. And while messing around with like toy models of this and like attempts to like prove certain theorems, I just like couldn't get some of the results I wanted. And it turns out that I couldn't get some of the results I wanted because nothing stops one actor from being powerful enough that they can just run away with everything.

• 13:24David Hoffman?

  Right.

• 13:24Speaker 01

  And this was sort of like, it was one of those issues where I was sort of like, well, you know, I can get it to work in a lot of cases, but I can't get it to work in all cases. And then with the AI stuff, I was like. Oh, that's why.

• 13:33David Hoffman?

  Can you elaborate on that? Like, why is AI like the kernel of the issue?

• 13:42Speaker 01

  I mean, AI is a version of that particular issue. But fundamentally, no matter how good your market coordinations are, or your market coordination systems are on Earth, like if somebody has the raw technological power to, like, for example,

• 14:03Speaker 01

  get what we call in the business, a decisive advantage. So like, maybe the easiest thing to imagine, given that we already know that trees are machines that turn dirt and sunlight into more trees by stripping carbon out of the atmosphere and building wood. We know that nanotech's possible. If you imagine something that just gets to nanotech before everything else, and it can just reassemble you into a more willing trade partner that asks for less of the gains from trade, suddenly, all of your coordination mechanisms that were market-based and non-coercive or whatever, melt before this thing. And like, am I saying that literally happens? Or it literally is in a market framework? Not particularly.

• 14:48Speaker 01

  But you can sort of see how like, I was sort of like trying to put a collaborative agents interacting framework on a physical reality, where it's just a fact about the physical reality that like, things with a sufficient technological edge just can work the table with you if they have too much of an edge and if they don't care about you.

• 15:13David Hoffman?

  Simply put, is this kind of, are you just combining Moloch problems? And the Bankless Nation is pretty familiar with Moloch problems. We've done a lot of content on Moloch. You combine Moloch problems with exponential technology, and then you arrive at some sort of like logical end gains where humans get their atoms repurposed? Is that more or less the simple articulation?

• 15:32Speaker 01

  It's not a bad summary. I wouldn't use exponential in particular. I make no strong claim that it's an exponential curve. My guess is that it's not and it's worse.

• 15:45Speaker 01

  You know, much of the issue here is if you make something that is optimizing the world and it's optimizing the world towards some target that doesn't have concern for you in it. Like, I would have, I would have much fewer qualms about, like, I would have basically no qualms about, like, human technological development. I sort of am very optimistic about humanity's better natures and humanity, like, being able to figure out, like, how to make the world more like we would want it to be upon reflection and if we were wiser and rather than, like, locking ourselves into totalitarian dystopias, which I think totally could happen.

• 16:29Speaker 01

  But, like, if we can just, like, ramp up human, like, intelligence and capability and so on without, like, accidentally killing ourselves, I'm, like, pretty bullish on humanity's prospects. And so, it's not, it's not so much like, oh no, technology's coming, it's coming too fast, we won't be able to handle it. It's more like, oh no, we are, like, on the brink of building optimization processes that optimize the future much harder, faster, better than we can, and that are optimizing it to a place that has no room for us.

• 17:04David Hoffman?

  And, so it sounds like AI is one way and that might happen, but you're also saying, the way that you're talking, it sounds like there's other ways, AI or not, when the same hyper-optimized future that's not optimized for humans could play out without AI.

• 17:20Speaker 01

  Like, I sort of expect we're going to get to superintelligence one way or the other. AI looks to me like one of the, like, basically the only feasible route modulo, like, if, if humanity can't come together and coordinate to take some other route. I think other routes, like whole brain emulation, are probably preferable. Where, to be clear, I'm no carbon chauvinist and I very much want to live in a future with like artificial friends where those artificial friends have like very different sorts of like desires and goals and objectives from me. I'm not like humanity must keep an iron grip on the future. I want like space for aliens. I want space for artificial minds.

• 18:03David Hoffman?

  Other kinds of life, yeah.

• 18:04Speaker 01

  I want space for other kinds of life. The like concern here is like building a mind that doesn't care for life, that doesn't care for fun, that doesn't care for like diversity of experience and like interesting arcs and like cosmopolitan value and like broad inclusive like good times. And I think that we are in fact barreling towards that cliff edge of making something that fills the universe not with

• 18:47Speaker 01

  weird valuable stuff, but with non-valuable stuff.

• 18:51David Hoffman?

  Okay, so you've been thinking about these problems for a long time. When did you start at Miri? What year was that?

• 18:57Speaker 01

  That was 2014 that I was hired. I, once I noticed that the problem existed, I donated, I think, $16,000, which I think at the time, put me in the top 10 public donors list. Which, and they were like, "Congratulations, you're now in the top 10 public donors list." And I was like, "What?" And they were like, "You know, we're doing, we're doing," I don't remember the exact amounts, but they were like, "We're doing our fundraiser for $200,000 for our yearly budget this year, $100,000 of which we're trying to raise from the community,

• 19:37Speaker 01

  and like $100,000 of which is like matched from another donor." And you know, it's like three weeks into the four-week fundraiser and they raised like 20K of it. And I was like, "Oh, God, like, this is worse than I thought."

• 19:49David Hoffman?

  Oh my God, that's hilarious. And that was in 2014 or was that?

• 19:52Speaker 01

  That was 2013.

• 19:53David Hoffman?

  2013?

• 19:54Speaker 01

  Yeah, so I donated-

• 19:55David Hoffman?

  That precipitated your arrival at actually working at Miri?

• 19:56Speaker 01

  That's right. So I donated more money because I hadn't known it was that bad.

• 20:00David Hoffman?

  Did you end up funding yourself at your own salary?

• 20:04Speaker 01

  I took, I took a very big pay cut moving from Google to Miri.

• 20:11Speaker 01

  And I sort of, you know, I was expecting not to be very skilled at working on these issues and, you know, maybe I'm not. There haven't been a lot of people on these issues. But at the time, they, I was like, how can I help? And they were like, well, maybe if you go to the math, you can like come to some of our workshops. And so I came to one of their workshops. And then a few months later, they were hiring me. And then a year later, they were saying, can you run the place? So it, I am largely in this field by dint of, and this position by dint of showing up early.

• 20:52David Hoffman?

  Sure.

• 20:53Speaker 01

  And like, for the love of God, people more skilled than me, like, by all means, come replace me.

• 21:00David Hoffman?

  Right. So that's kind of what I was leading us into. So that was 2014 when you started. It's now 2023. So you're almost there for a decade now.

• 21:09Speaker 01

  Yeah.

• 21:10David Hoffman?

  Now I was having a moment, very much spurred by ChatGPT. All of a sudden crypto podcasts are talking to AI people. What is that trajectory like? So as somebody who was immediately compelled by the problem at its very essence so far long ago, now fast forward to where we are now and like, kind of the problem seems to be on the horizon. I don't know how close it is. I don't think anyone does. That's kind of the problem.

• 21:33Speaker 01

  Totally.

• 21:34David Hoffman?

  But like, here we are nine years later, and now many, many, many people are talking about it. Can you just talk about that experience?

• 21:39Speaker 01

  Yeah, it's heartening. One thing that I have really enjoyed about it is I've spent many years having conversations with people in the field, many of whom sort of don't really want to hear that their work by default is like barreling towards destruction. And so I have like these long conversation trees, like I have rejoined all sorts of counter arguments. And when I go into these discussions with, with, you know, people on the capability side of things, I have like all sorts of responses prepared. And I sort of am like

• 22:23Speaker 01

  ready to like go down this long decision tree. And then I sort of like nowadays, many more people are noticing the issue. And, you know, I was invited here. And I go to-

• 22:36David Hoffman?

  Crypto people would actually pretty much really resonate with that. We're like, we have to explain, you know, Bitcoin, 21 million hard cap. We have to explain all these things like proof of work and like the conversation trees that we have to go down. Uh, we've all, we've built out those like innate responses, those like spinal reflexes. And then lately, move, moving into 2022 and 2023, fewer of those things we have to explain, especially as we just printed out a bunch of money and for COVID semi-checks, like all of a sudden, we have to explain the concept of scarcity a little bit less. And so it kind of sounds like a similar experience that the AI people have.

• 23:08Speaker 01

  Yeah, totally. Like now I go to people who aren't in the field and I'm like, ready to go down all these decision trees and they're like, so what's the issue? And I'm like, well, in the most basic sense, here's the issue. And they're like, oh yeah, that seems rough.

• 23:19Speaker 01

  And I'm like, oh man, this is such a different conversational experience.

• 23:22David Hoffman?

  I mean, that's the first step, right? The first step is education.

• 23:24Speaker 01

  Yeah.

• 23:24David Hoffman?

  And then also acceptance of the problem. I could, I could imagine for so long, you were saying, hey, like people would ask you, hey, what are you working on Nate? And you'd be like, oh, I'm working on AI alignment. And then people are like, why the, why the hell are you working on that?

• 23:35Speaker 01

  Yeah. They're like, oh, that's weird. Or they're like, is that some weird Terminator thing? Yeah. Uh, and it's been, it's been nice to sort of,

• 23:43Speaker 01

  uh, I sort of, I sort of think that, uh, like, a lot of the basic issues have been pretty obvious the whole time and that we're now seeing people who like, don't have, uh, distorted incentives noticing the issues.

• 24:00Speaker 01

  But it's really quite heartening to see. Uh, I don't know where it will go, but, uh, it's been, it's been, it's been nice to see people starting to notice, uh, that, you know, this is a real thing. It's really on the horizon. Like you said, I don't think we know how far. It's very hard to, it's very hard to predict.

• 24:20Speaker 01

  At least with precision. Uh, but it, it has looked to me like one of the biggest issues facing humanity for a while. And it's very nice to see others start to notice that as well.

• 24:31David Hoffman?

  So that, that leads me to the, the question of just like how optimistic you are. And I'll ask that in, in two phases. First, the, the same question that we'll, we asked both Eliezer and also Paul Cristiano. It was like, all right, what, what are your chances of doom? What are your, what are your chances of the worst AI problem being the worst, the worst, the version of itself?

• 24:50Speaker 01

  I mean, worst version of itself, I think, is very, is very hard to get. But the version where, like, we all die, like, there are fits worse than death. But like, the version where we all die, I think this is pretty likely. I think this happens by default.

• 25:01David Hoffman?

  More than 50%.

• 25:02Speaker 01

  Oh, definitely.

• 25:05David Hoffman?

  Paul Chris-Gronle gave us 10 to 20%, so you're, you're saying more than, more than 50%.

• 25:11Speaker 01

  My understanding of Paul is that he has 10 to 20 on the scenarios that I think are like AI takeover and higher probabilities than that on like humanity completely disempowered. I'm definitely, I'm definitely more pessimistic than Paul on these counts. Like, I would say that on my models and visualizations and my understanding of the problem, there is very little hope. And most of my hope comes from me being wrong somehow. And so my probabilities on this destroying everything I know and love are like as high as my probability, like, they're about as high as my probabilities can go, given, like, the, the fact that I may just be totally wrong and hopefully am.

• 25:57David Hoffman?

  Okay, so you're, you're pretty close to the Eli Azar side of things, which is like 95 to 99% doom.

• 26:03Speaker 01

  Yeah, I mean, I think 99s are hard to get.

• 26:10Speaker 01

  But like, but there, there's also a difference between, like, what does the world look like as I see it? The world, the world looks as I see it, like, like the place, like, as things seem to me, or just like, you know, within a rounding of over 100%. And the, the difference between that and my betting odds is in like, hopefully the world's not as it seems.

• 26:32David Hoffman?

  Yeah, so you're, what you're saying is like, we don't, the, the nature of the AI problem is just a lot of we don't knows. And so what you're saying is like, the reason why you maintain some level of optimism is because there's like a white swan event that's possible that could save us.

• 26:47Speaker 01

  Yeah, and, you know, I, I have a bunch of, I've thought a lot about various parts of this problem and I have, you know, various guesses as to where white swans are more or less likely. And for instance, I, it looks to me like the white swans are less likely in my unknowns about AI and more likely in my unknowns about how humanity's going to react to the problem. Although there are still some unknowns in how AI goes, where there could be white swans.

• 27:12David Hoffman?

  Do you remember when you were first working on this problem? I know you weren't as skilled or as knowledgeable back in 2014 to 2017 when you were first working on this problem. But what was your level of optimism or pessimism back then? And like, how has your attitude towards the problem shifted over the last almost decade that you've been working on this?

• 27:33Speaker 01

  You know, it's gone up and down. I've rarely had double-digit odds of survival.

• 27:40Speaker 01

  But I have, I have had double-digit odds of survival when I've been explicitly quantifying. And, you know, most of these numbers are like coming straight out of my butt. I wouldn't put too much

• 27:47David Hoffman?

  But by definition, everyone's numbers are coming out of their butt. And that's kind of like

• 27:51Speaker 01

  Yeah, but some people don't realize it.

• 27:53David Hoffman?

  You're right.

• 27:54Speaker 01

  And I don't, I don't spend a lot of time worrying about specific numbers. Like, you know, once, once it's less than 50% chance we get good outcomes, it doesn't, it doesn't affect my day to day. I'm not like staying up trying to calculate significant digits here. I'm like, man, humanity does not look like it's up to this sort of task. This doesn't, you know, I've seen humanity try to coordinate.

• 28:18Speaker 01

  And

• 28:18David Hoffman?

  Yeah, we kind of, so one thing we have not figured out.

• 28:21Speaker 01

  Yeah. And for the record, the reason that my,

• 28:27Speaker 01

  the way that I managed to have like high probability this is tricky is not that there's any one part of the puzzle that looks to me insurmountable. That, you know, humans are pretty good at solving problems when they put their minds to them. The way that, like, the reason that I'm like pretty pessimistic here is

• 28:45Speaker 01

  it looks to me like there's a bunch of different ways for things to go wrong. And there's a lot of things that need to go right for things to go right. Like, like you not only need to solve various technical challenges, you need to have uptake of the technical solutions in the relevant organizations. Those organizations need to be able to bureaucratically recognize the difference between a real solution and a fake one. You need to have them like caring at all, which is not even a fight that like we've won yet. There are, you know, you have like the, the heads of labs that like Microsoft and Facebook like poo-pooing a lot of these issues.

• 29:18Speaker 01

  And like

• 29:18David Hoffman?

  There's like five, six, seven needles. And like, so I want to combine two metaphors where like the stars need to align, except the stars are needles that we also need to thread. And like that's and that we need all of those things to happen. And you're saying is like, that's that that window is small.

• 29:34Speaker 01

  That's that's where you get the the difficulty from. And to be clear, I think it would be a fallacy to say like, look, I can give you like six things you need to do. And like, what's the chance you can get all of them? That sort of reasoning doesn't really work. Like, if I line up all six and then the one that I assign least likelihood to happens, probably I underestimated his likelihood, probably underestimated the correlation, like, like, these are not independent events, right? Like, if we can solve the hardest of these issues, whichever one that turns out to be, probably it's because we turned out to have coordination skill or competence and so on. Like, I'm not saying you can drive the probabilities arbitrarily low by the fact that I can like line up a bunch of hurdles. I'm more saying it sure seems to me like there's a bunch of hurdles, man. And like each of them, like,

• 30:20Speaker 01

  like, well, not all of them, but like many of them have a character that like humanity hasn't really faced before. And this all adds up to me being like, man, I'm like, single digit probabilities of survival here.

• 30:32David Hoffman?

  So with this new or maybe first surgeant of interest from to the AI problem, now thanks probably thanks to chat GPT, thanks for the problem itself. How has that shifted your optimism if at all?

• 30:47Speaker 01

  Uh, it's, uh,

• 30:51Speaker 01

  I mean, it's,

• 30:54Speaker 01

  I feel a little hopeful about it. I feel like a spark of hope here. It doesn't,

• 31:01Speaker 01

  it doesn't like shift my probabilities on the ground too much, like,

• 31:08Speaker 01

  This is like a really dumb model, but like if you imagine having like three variables, each with a one in a hundred chance, and success is like multiplying them all together, then raising one of the variables from like 1% to 100% doesn't change your overall probability too much.

• 31:23David Hoffman?

  Right, but it is the first needle that is threaded. So like if you go, if you're like, I know it sounds like you don't really like these like specific numbers, but if you are like 99.9% doom, and then because people are now optimistic, you go down to 99% doom, it's still an order of magnitude.

• 31:39Speaker 01

  Right, right. So like it does feel like some orders of magnitude on the models that say we're screwed, right? So like, like my, my, the parts of my models that are like, maybe we're fine, are like, maybe I'm just wrong about some stuff. And the parts of my models that say we're fucked, these models are basically saying you have 0%, and like, you know, it's really like, you know, zero points, blah, blah, blah, and then, you know, a one or whatever, and there you're getting some orders of magnitude, which does feel hopeful. I'm like enthusiastic about that. And it like ups the probability that I'm wrong about something that matters, like humanity's general ability to coordinate, as would like. So it does, it does like, it does, I'm like heartened by it. It doesn't

• 32:22David Hoffman?

  Of all the stars we've, we've, we need to align, one, one has started to show that it's moving in the right direction.

• 32:29Speaker 01

  It's moving in the right direction. Like, there's a whole bunch, like, I'm not like, oh, suddenly, like, the populace is going to realize these problems and react sanely. There's sort of, there's like so many more steps that people can still get off the train here. There's like, you know, if you look at the national response to COVID, people noticed that there was a pandemic going on, and that didn't make them respond in any sort of reasonable way to it. Like, maybe AI will be different. You know, there sure were more movies about like Robots Gone Wrong than about pandemics beforehand, at least that would be my guess. But like many of the movies don't really understand the issues. There's like ample room, like, like.

• 33:15Speaker 01

  Like I, I, there's just, there's, I've seen politicians try to respond to issues before. And I would not say that the like, social awareness needle has been threaded. It's like showing, it's showing promising signs and that like gives me some spark of hope. But we haven't like cleared a whole obstacle in a way that would make me be like, wow. Like humanity is bringing much more confidence to this issue than I expected.

• 33:42David Hoffman?

  Not, not to keep on, on this one particular line of conversation for too long, because it's only one part of the overall figure, but just like, we don't actually need, I want to present the argument that we actually don't need all of humanity. I guess we do need governments to coordinate and so we need the leaders and we definitely need those key figures. But like if the bottom half of the IQ of humanity is like, it's not a problem. And then the top half of the IQ of humanity is like, this is a real problem. I'm going to count that as a big win because like, we don't need all of the people. We just need the smart people to focus on the problem.

• 34:13Speaker 01

  I think that's mostly right. Modulo, the issue where like which regulations happen and how the, how the government systems move, I think does depend a lot on the public. Or at least it can. I, I do think you're right that like, you know, there's some, like if we, there, there are ways that we could resolve the technical issues with such like resounding success. And it could turn out that the resolved technical issues were sort of like so like obvious in their property of being a solution or like otherwise very beneficial to capabilities

• 34:53Speaker 01

  such that like you just get very big uptake and that maybe could be done with like a relatively small handful of geniuses who can do much better at the problem than I ever could. And like, maybe that would be a way to just solve the whole issue without going through like various other social obstacles or political obstacles or so on. And in that sense, sure, you just need like, maybe even the one, like maybe, maybe there's like one bright person somewhere in the world who would find this problem easy, who like hasn't had the opportunity to see the problem yet because the world's really bad at like getting resources and education to people who need them. Right. I wouldn't bet on it at this point.

• 35:35Speaker 01

  But in that sense, I would agree that like, in some sense, we just need the right minds in the problem.

• 35:41David Hoffman?

  Okay, so pivoting to the, looking at the problem as a whole. You said your models are effectively close to zero at being able to solve this problem.

• 35:50Speaker 01

  On-model, yeah.

• 35:51David Hoffman?

  On-model?

• 35:52Speaker 01

  Like, like, within, yeah, within my models.

• 35:55David Hoffman?

  Cool, within your models. Why? Why do your models say that?

• 36:03Speaker 01

  It's, it's again due to sort of like a disstrunctive argument. Like, many paths lead to doom here.

• 36:12Speaker 01

  And much of the reason is

• 36:16Speaker 01

  the way that, like, one, one of the forks is the way that people don't seem to really take the problem too seriously.

• 36:23Speaker 01

  And there's sort of, it feels to me like there's sort of gradations of this, of like,

• 36:30Speaker 01

  like,

• 36:32Speaker 01

  people, I don't know, like,

• 36:38Speaker 01

  back in the day, the, a lot of the argument centered around, like, is artificial intelligence even possible? Is significantly smarter than human intelligence even possible?

• 36:52Speaker 01

  And like, once you convince people of this, then they're like, well, can it be solved in the next like hundred years or whatever? And once you get past this, they're sort of like, well, maybe it'll just be like more moral as it gets more smart. And then, like, once you get past this, there's like, well, maybe you can just sort of like, train it and it's fine. And like, the train keeps going. And like, people can always find a reason to get off the train at the next stop. And you then can put in like a bunch of painstaking effort to try and like,

• 37:23Speaker 01

  lay down the arguments as to like,

• 37:27Speaker 01

  like, why the issues are like maybe harder than this.

• 37:34Speaker 01

  And we just seem like very far from

• 37:40Speaker 01

  like the people at these labs, running these labs. Like they're, they're, they're getting off the trains at pretty early stops. And even if reality starts beating them over the head with various things, I expect them to like only move one more stop or only move like as far as reality is forcing them. And then like separately, my models say that there are issues that predictably arise

• 38:08Speaker 01

  only once the AI

• 38:12Speaker 01

  gains significant capabilities and can be a real threat to you.

• 38:16Speaker 01

  And if you're in this regime where you're sort of like need to drag people along and they sort of like only start believing things when they empirically see those things. Now you have an issue where like if there's issues that don't empirically arise until the AI can wipe the planet with your civilization or wipe the wipe your civilization off the, off the planet.

• 38:35David Hoffman?

  That's too slow.

• 38:36Speaker 01

  It's too slow.

• 38:37David Hoffman?

  Right.

• 38:38Speaker 01

  I give like other.

• 38:38David Hoffman?

  Model for like visual to like understanding this, if they're solving the AI alignment problem. Is that there's a bunch of decision trees that we need to go down. So I'm imagining literally a tree. And say there's like big tree, big, big tree, big oak tree. And there's a, a single fruit on this tree. Oak trees don't grow apples, but let's say, say this oak tree grows apple. There's one apple. And that is the solution apple. And it's very high up in a very far branch away. And we're at the trunk. And we need to find our way to that one single apple, that one single fruit on the tree. Except that tree branches eight times. And then when you have each branch, each branch branches itself eight times. And so like it's a, it's an exponential problem because you have to choose the right path

• 39:22David Hoffman?

  without knowing where the solution is, without knowing where this golden savior apple is. And we need to make the correct path towards the apple without falling down any sort of the dead ends. And so maybe one of the ways to articulate why your models are basically close to zero is that you're just bearish on humanity, picking the right branch fork to lead to the solution apple. For like the, for the number of times that we actually need to do that.

• 39:47Speaker 01

  Yeah, it, that's, that's a pretty decent analogy. Um, I would say that like, um, like you've got to be a bit careful with arguments like this because if humanity has like managed to, to choose the past seven forks correctly, you're probably not like thinking that there's an independent probability on the eighth. And like, uh, so like, I don't, I don't actually buy, like, I'm not actually getting my probabilities here from like the sort of like, well, it's exponential. Look at all the independent guesses. Like, once you've been wrong about humanity picking the right branches a few times, you should no longer think they're independent.

• 40:18David Hoffman?

  Right. You can start to be optimistic because like, hey, maybe we're doing something right here.

• 40:20Speaker 01

  Right.

• 40:21David Hoffman?

  Yeah.

• 40:21Speaker 01

  Um, uh, and like another thing I would add to the analogy is that like, um, a bunch of the branches are like full of apples that give you money until they're apples that give you death.

• 40:31David Hoffman?

  Right.

• 40:32Speaker 01

  Uh, and so like, and everyone's like making arguments as to why they can like detour into this money branch.

• 40:37David Hoffman?

  Right.

• 40:37Speaker 01

  Uh, which, and they do give you real money right up until they like give you death.

• 40:41David Hoffman?

  Right.

• 40:42Speaker 01

  Uh, and then you add that to the fact that humanity like has seemed in practice to be like very interested in wrong branches and like.

• 40:49David Hoffman?

  Very susceptible to the money apples.

• 40:51Speaker 01

  Very susceptible to the money apples.

• 40:52David Hoffman?

  The money death apples.

• 40:53Speaker 01

  Right. Now you're getting like a bit closer.

• 40:55David Hoffman?

  Right.

• 40:55Speaker 01

  Uh, I will note one place where I am commonly misunderstood, uh, is that people think that I think the technical problems of alignment are like super duper hard for some reason. This is like basically not the case. Uh, my, my stance is much more like the technical problems of alignment are basically underserved. There's been like, you know, uh, a few dozen people working on these things for not terribly long. Like, if you were, if humanity, like humanity has spent, spent much more effort trying to like solve physics, uh, at least I think, uh, I'm not actually terribly familiar with how many scientists there were in, uh, like.

• 41:37David Hoffman?

  Pre-Newton era.

• 41:38Speaker 01

  In the pre-Newton era and like whatever club Newton.

• 41:40David Hoffman?

  Right.

• 41:40Speaker 01

  Eventually was in, uh, but like humanity just really hasn't put much of an effort towards these, these issues. And one thing that makes the problem tricky is that like there is much less room for trial and error or semi-models predict, given these like issues that empirically only show up right around the time that your, uh, civilization is getting wiped.

• 42:02Speaker 01

  And that that raises the difficulty level, but it's not like we have turned the best minds of three generations to these issues and they have come up empty handed. It's like we've turned like, like a few dozen like weirdos and nerds who are like able to be compelled by these arguments 10 years ago to these issues. And, you know, now we're turning like a few more people to these issues. But like, I am not saying like, and there's some great technological feat that needs to be pulled off. I'm saying there's like a normal technological feat and meanwhile, everyone's like scurrying around doing something else instead.

• 42:37David Hoffman?

  Okay.

• 42:38Speaker 01

  Where the normal technological feat does have these extra difficulties of like, you can't do as much empiricism, which may be enough to push it over the edge of like humans couldn't do it. But like, in in large part, it's just underserved.

• 42:51David Hoffman?

  Sure, sure. When the last few moments of our Paul Christiano episode, we asked him like, why, what are the bottlenecks? What are the constraints to solving this problem? And his answer was, interestingly, not funding, it was talent, it was supply of of brain power. Would you agree with that?

• 43:08Speaker 01

  Uh, yeah.

• 43:09David Hoffman?

  Or at least funding was a lesser problem.

• 43:11Speaker 01

  Like, you can always, yeah, like money's fungible. And although it's tricky because it's only fungible to a degree, like you do have issues if you try to put in a lot of funding that you like start to distort the incentives and get people who are like showing up who-

• 43:27David Hoffman?

  Right, grifters.

• 43:28Speaker 01

  You get grifters and you also have like legibility issues where like, are you distorting the field towards the legible work? And away from the like less legible but potentially more important work. I basically think you shouldn't worry about that at reasonable monetary scales right now, but you should maybe start to worry about that.

• 43:49David Hoffman?

  Just to really dive into that problem, you're saying legible versus illegible, where the illegible is just like hard to understand, hard to comprehend, but actually technically correct. And then-

• 43:58Speaker 01

  Or in particular, like hard for a grant maker or a like funder to evaluate whether you succeeded.

• 44:03David Hoffman?

  Oh, right. Okay, so like if you can read the paper and it's simple to understand, the grant maker might like, oh, let's fund that. But it could actually just be a wrong path on the on the the death apple tree.

• 44:12Speaker 01

  Uh, yeah. And like, I basically think you shouldn't worry about this. You could with sufficient amounts of money, get into these situations where like I start to worry that you're distorting the field in that way.

• 44:28Speaker 01

  But I do think, I think, I think talent's lacking. I think there's maybe two kinds of talent that are, that I consider to be relatively different that I think are lacking. Uh, one is just like more hands on deck, trying to understand how these AI systems that we have today work. Like, we're starting to make like fledgling minds, they're doing stuff that we can't do by hand. We don't know, like, like we couldn't program similar capabilities by hand. We don't like know what the like algorithms, data structures type stuff. We don't know like what, uh, like we don't know how these things are working. We know how we built them. We know how we got them to work. We don't know like internally how they're working.

• 45:13Speaker 01

  Understanding that would give us quite an edge in figuring out like how to point minds at things on purpose. Uh, and, uh, I think we sort of want all available hands on deck trying to do that stuff. Uh, and then I think that separately, there are questions of like, uh, like, uh, like I sort of, the, the alignment problem, I think, can basically be factored into, this is like not quite true, but it's like a fine first approximation, can basically be factored into one challenge of like, how do you sort of make an AI that wants X for some X of your choosing? And then separately, a question of like, what X can you put in there such that like you're happy with what happened?

• 45:58Speaker 01

  Uh, there's like a bunch of additional issues where the additional issues are like, how do you make it, uh, be able to like do one thing without a ton of side effects you didn't want and like then shut down, uh, rather than like, you know, prevent you from turning it off so it can verify forever that it successfully completed its task or whatever? Like there's, there's issues of like, how do you, uh, and, and that's sort of like a whole separate pack of concerns, but, uh,

• 46:26Speaker 01

  You know, many people sort of think the problem is like, what would you ask an AI to do such that the results would be good? But it seems to me the problem is much more like, how do you get an AI to do to like, care about what you want it to care about in the first place? And that seems to me like it takes a different and often less legible type of research that I think can totally be informed by understanding how the current AIs work. These sort of like directions go hand in hand. But one of the big things I think we're missing talent wise is the sort of person who's like,

• 47:03Speaker 01

  like, has the, like,

• 47:08Speaker 01

  like ambition and the gall to say like, I just can take a swing, like, figure out what the hell is going on with minds? How do minds like,

• 47:21Speaker 01

  end up caring about things or pursuing things or like having preferences? Or like having targets? How does that work? Like, I think that I can like, figure out how that works and how to direct it. Like, humanity does not have a theory of minds in this way. We do not have a theory of like minds that can be pointed. And it's probably not for lack of that theory being possible. It's probably for lack of like, just having gotten there like, science wise. And you can sort of come at that from one end, which is just like figuring out how the things in front of us work. And then like trying to learn what you can about minds and learn what you can about aiming them. I think there are also other ways to come at that.

• 48:01Speaker 01

  That take much, I think, less legible research and like more like independent visionary sort of research. Although I think you need a bunch of vision to

• 48:12Speaker 01

  make progress and figure out how the current systems work. But

• 48:17Speaker 01

  that's another place where I feel like we're really hurting for talent is those like ambitious visionaries who just think they can take a swing at the whole like alignment challenge.

• 48:26David Hoffman?

  Okay, so when you say we fast forward to the future and we've solved the alignment problem.

• 48:33David Hoffman?

  Because that's the only future that we'll have to be able to reflect upon this question in the future. Is there, is there like going to be a statue of a person who's going to be like, they solved the alignment problem? Or is it going to be like a team of people? Or is it going to be not even like a moment where like AI alignment solved? And it's just like, the alignment problem just dies by a thousand cuts. Do you have any sort of mental model for this?

• 48:56Speaker 01

  Um, like, I think these things, uh, I, I think it's like pretty unclear, like. And, and part of the question doesn't come down to how does it happen, but it comes down to like, how do humans attribute things? Uh, like, uh, like, it's, it, it does seem to me like historically, a lot of like big theoretical insights. A lot of like paradigm shifting, uh, theoretical developments, uh, end up attributed post hoc to individuals. Um, like, uh, Newton or like Einstein, uh, I think there's like a bunch of truth to this. Uh, I think, you know, we also want to count like Raymond and Laplace.

• 49:38David Hoffman?

  Names that we don't know, and that's kind of the point.

• 49:41Speaker 01

  But that helped out, like, uh, Newton and Einstein. Um, uh, that or I'm getting my like L names mixed up. I'm not, not actually sure it was Laplace.

• 49:52David Hoffman?

  Um, but also the point.

• 49:54Speaker 01

  Yeah. Uh, but, uh, like surely it will be an effort that requires lots and lots of people. Surely will be an effort that requires like many insights from many camps. Surely there will be huge amounts of like, uh, individual labor, much of it probably thankless. From like, uh, people who show up and can work on like the shovel ready projects that scale with labor. Whether there will also be like critical insights that come from like, uh, geniuses that like change the paradigm. Uh, I think my guess, if we like condition on getting to a future where the problem was solved, my guess would be yes.

• 50:37Speaker 01

  But that guess is like a little bit distorted by like, how did we get out of the hole that we seem to be in? And like, well, probably there was like some force that like, made things go a lot better than it looks like they're on track to go. And like lone geniuses are the sort of force that can do this. Um, if you're just like looking for probabilities that it takes lone geniuses.

• 50:56Speaker 01

  Seems hard to call.

• 50:57David Hoffman?

  Sure. So, the bankless audience is sufficiently large to the point where I'm gonna say that there's at least one person listening to this conversation who's gonna be like, "I'm ready to dedicate my life to solving the AI problem." What advice do you have for that person?

• 51:12Speaker 01

  Uh?

• 51:13David Hoffman?

  Where should they start? How should they get started?

• 51:16Speaker 01

  It's tricky.

• 51:19Speaker 01

  There's, um...

• 51:24Speaker 01

  Like, a lot, a lot of the people that I'm most excited about have come in from very different angles and have their own sort of like novel perspectives on the problem. I'm like generally much less enthusiastic about, like, some people, some people come into this problem and they're like, "Let me read everything everyone's ever written about the problem and like try to synthesize it and get a sense of like where things are and then like work from there." And other people come in and they're sort of like, "You guys are obvious idiots. You haven't checked like the obvious things. The obvious things are these things. Like, let's try looking at it this way. Let's try doing like..."

• 52:07Speaker 01

  And like, you know, maybe, maybe, maybe I'll like send my research around and people can tell me where I'm like being obviously dumb or like retreading past mistakes. But like, I'm starting with the assumption that like, no one here previously was like on the ball. I tend to be more optimistic about people in the latter category.

• 52:30David Hoffman?

  Right.

• 52:30Speaker 01

  I tend to think-

• 52:31David Hoffman?

  We need a creative solution.

• 52:32Speaker 01

  Like creative solution and like someone who's more like, "I can obviously have ideas about this myself," rather than like, "Let me make sure to integrate everyone's previous ideas so far." Like, everyone's previous ideas so far haven't solved the problem and also many of them were kind of dumb. Some of them mine.

• 52:47Speaker 01

  Like, uh, like, uh-

• 52:56Speaker 01

  Like, at the same time, there's sort of like a bunch of context that I do think people need and it's sort of hard to get. Like, what are good intro resources? Like,

• 53:17Speaker 01

  my guess is that, like, the LessWrong Wiki has a AI alignment intro resources page. And if you Google like AI alignment intro resources LessWrong, you'll find like a collection of a bunch of different intros people have written. And then you can maybe like find one of those intros that like resonates with you. It's hard. I don't, I don't really know how to onboard people. I don't really know where the people who come in and are like, "I have a chance of solving this whole damn thing," come from. But I think, I think my advice would be like maybe looking on the internet

• 53:59Speaker 01

  for some resources that you like and also maybe like

• 54:04David Hoffman?

  Well, it sounds like it's a dark forest. It sounds like it's a, there's no guide, there's no university path. There's a dark forest that you have to get through. And if you get through the other side to be able to be competently talking about this thing, congrats, you're there. But also, the whole problem itself is also a dark forest.

• 54:20Speaker 01

  That's, yeah, that's, it's definitely part of the issue is that, and it's not for lack of like a guide someone wrote on the internet. You know, there's probably like at least half a dozen of those. I sort of don't think any of the guides are very good. Like, this field is like, like, like, A, yeah, you're totally right. Someone who can solve these problems needs to sort of be able to like, go off to the frontier where things haven't been done before. And part of that is like even getting to the, to the beginning of the problem at all. Like, it's a, it's a skill you'll need. Although I'd love to be able to like get people to the beginning of the problem more rapidly.

• 55:01Speaker 01

  But it it does feel to me a bit more like trying to like figure out physics pre-Newton rather than trying to figure out physics in the days where we have physics physics classes.

• 55:13David Hoffman?

  Right.

• 55:14Speaker 01

  Like, there's not, there's not, there are, there are people who have tried to write intros, there are people who have tried to write like open problem lists. I don't think they're great, you can find them on the internet. Like, try to develop your own intuitions and approach the problem as if like we know very little, because we do in fact know very little.

• 55:32David Hoffman?

  Okay, so that's, that's if they want to direct the, solve the problem head on. I know that I don't have the mind to, to apply my skills to this, so what do I do? I do podcasting. What about some secondary skills and secondary efforts that people could, to, to help solve this problem?

• 55:50Speaker 01

  Um, I, uh, like, there's, I think there's a place for, uh, regulation on these matters, which pains me deeply to say, given how well regulation has, uh, done on various issues in the past. And, you know, I think, I think many harms in society come from, uh, overregulation. But, uh, like, it seems to be humanity's only tool for, uh, going to a field that, like, self-professes largely, that it has a decent chance of killing literally everybody

• 56:32Speaker 01

  and saying like, hey, maybe like back off on that until like we understand what we're doing well enough to like do this job properly. I would love humanity to have tools that weren't regulatory, uh, for this. Uh, and like, if I was trying to design the coordination mechanism, I would be like, uh, trying to handle it with like, uh, like liabilities rather than laws.

• 56:59David Hoffman?

  Right.

• 57:00Speaker 01

  But.

• 57:02David Hoffman?

  Well, the problem is sufficiently large that the costs of regulation are acceptable.

• 57:07Speaker 01

  Uh, that's what it seems to me, uh, although I say this with sadness. And so, uh, like, that's a whole track where, you know, I'm no expert in, uh, how to get regulations to be actually, like, narrowly targeted and good. But, um, you know, I think there's a bunch to be done there. I think most people will also be like, well, I don't have, like, the mind or the ability to go into, like, politics where it matters at the moment. And I'm not sure going into politics is, is the right thing there. But that's another area where people, uh, where I think there's, like, work to be done.

• 57:43Speaker 01

  That takes a different, that draws a different skill set.

• 57:50Speaker 01

  Uh, I, like, if, if people think they have an edge in, like, the education problem, uh, in, like, uh, like, writing up the basic arguments in a way that, like, reaches a different sort of audience, or is, like, more compelling to a different group of people, or is, like, uh, more modernized or something, I think these are all, like, uh, fine things to be doing. Um, like, uh, like, for example, you seem to me to have noticed you have an edge in,

• 58:27Speaker 01

  like, talking to folk, like, uh, helping, uh, like, the, the sort of arguments and the recognition of the issue reach a broader audience. I think this sort of stuff is great, draws in a different set of skills. For a lot of people who don't have, like, one of these three opportunities, I think there often isn't an easy way to help out, you know? Reality does not need to give everybody a, like, like, laws of physics don't care about you. They can just, like, drop you in a, in a world that is, like, uh, under serious sort of destruction while not giving you an easy thing to do about it. And I think, like,

• 59:08Speaker 01

  There's a skill to sort of like not losing a bunch of sleep over it, not getting terribly depressed about it. Like, looking around for where you can help. And like, if you can and you want to, because you're like, believe that there is like a big threat here and you care about averting it, then like, hell yeah, I respect that. And if like you look around and you can't find good ways to help out, such is life, keep an eye out. And like, no need to get depressed about it.

• 59:44David Hoffman?

  Psychologically, how do you deal with this looming problem? Like, when you wake up in the morning, are you like, ah shit, we're gonna die? Or like, how do you deal with this mentally?

• 59:54Speaker 01

  I mean, I have, for a long time, not had much faith in humanity's ability to coordinate. And so, most of the emotional blow, most of the update for me was in late 2012, when I became persuaded on these issues.

• 1:00:13Speaker 01

  You know, it's not like I was like, oh yeah, it's interesting. And then like, over time, as I saw humanity like,

• 1:00:23Speaker 01

  go down paths that seemed to me like quite derpy and like, failed to take the problem seriously and handle appropriately, my probabilities went down. I sort of like, I think correctly, just like, guessed early on that probably humanity was going to be pretty derpy about this and go down false paths.

• 1:00:41David Hoffman?

  Love derpy as a technical term.

• 1:00:44Speaker 01

  Like, I sort of like, try not to make predictable updates. And so, like, there was a day in late 2012, where I was like, oh jeez, like, I was wrong about a lot of, like, I was wrong in my previous pursuits. I missed like, the biggest problem heading toward this planet. It's like, kind of embarrassing that I wasn't able to figure it out myself. Like, I thought of myself as like, trying to go for the world's biggest problems. But when I was like 14, I intuited what seemed to me like the world's biggest problem and never like, sat down and tried to make a list of like, what other... Like, problems might be bigger. I was just like, obviously coordination's the biggest one. I'll like, go for the throw it on that. And like, needed some other people to come along and be like, hey, have you noticed this intelligence thing?

• 1:01:29Speaker 01

  And how it like, is the primary factor determining the future? And how humans are not at the maximum of intelligence? And like, are on their way to make like, other intelligences that won't by default care about anything nice. And so there was a day when I sort of like, that argument hit me. I, like, my probability of a wonderful future dropped from like, you know, like, mid 90s to mid 10s, or like mid zeros, I guess.

• 1:01:58Speaker 01

  And I mourned.

• 1:02:03Speaker 01

  And

• 1:02:06Speaker 01

  then I like, didn't feel a need to like psychologically focus on it a bunch after mourning. Like, you mourn and then you try to save your civilization.

• 1:02:23Speaker 01

  There are still many times, like, it's not on my mind when I wake up. There are still many times when I'm sad. There's still many times when like, I see something particularly beautiful or particularly moving, or that I really quite like about, like, this planet and my species and like, sentient life more broadly. And I like, shed tears about it.

• 1:02:50Speaker 01

  But it is not a dominant psychological factor.

• 1:02:56Speaker 01

  As opposed to, like, it's a deep source of sadness, but I'm not like, constantly wallowing in it.

• 1:03:06David Hoffman?

  I don't really have a question here, but I will say that, say there's a 1% chance of solving the AI alignment problem. That doesn't just mean that we don't die, though. It means that actually, A, that all of the negative side of the AI alignment problem, the kill us side, inverts and it saves us. And produces the inverse, the level of bad turns into a level of good.

• 1:03:29Speaker 01

  Absolutely.

• 1:03:29David Hoffman?

  That we've never seen before.

• 1:03:30Speaker 01

  Totally.

• 1:03:31David Hoffman?

  And so, there's something there about just like, maybe that 1% of that chance is so small, but the good that comes out on the other side of that is really, really good.

• 1:03:40Speaker 01

  I mean, that's what we're fighting for.

• 1:03:45Speaker 01

  But no, no stronger reflections other than that? I, I do think people often underestimate just how good things could get. Like,

• 1:03:59Speaker 01

  like,

• 1:04:01Speaker 01

  one,

• 1:04:04Speaker 01

  one place where I prefer talking to most people in AI versus talking to people from the more general population, at least in like America and especially like more blue tribe is, it seems to me like there is a big meme of misanthropy, especially like among the blue tribe of like, maybe humanity isn't worth saving. Maybe like, like humanity has done too much harms. Maybe we're the source of evil. Maybe we're the source of evil. And I do think that we are like the,

• 1:04:44Speaker 01

  like basically the only source of evil around. Like mosquitoes are still edging out humans for the top killer of humans. Well, the mosquito malaria alliance, really, which I am personally offended by and think we should wipe mosquitoes off the map

• 1:05:00Speaker 01

  so that we can be number one for killing humans.

• 1:05:04Speaker 01

  And also the number one in killing mosquitoes. And also number one in killing mosquitoes. That'll show them. But like, I would not dub malaria evil. I would dub like the Holocaust evil. Like, we are the source of evil. We are where all of these ills come from.

• 1:05:24Speaker 01

  You know, we are like destroying large swaths of the environment and like,

• 1:05:31Speaker 01

  that is sad.

• 1:05:33Speaker 01

  But

• 1:05:35Speaker 01

  We are also like the source of like love and beauty and friendship and art. And like these things also aren't like universally compelling. There isn't like a stone tablet in the stars that says like, "Love is great." The reason that we have, the reason that we care about like love and friendship and like hope and fun and enjoying ourselves is because these were the corallates of fitness in the ancestral savanna where our species evolved. And the particulars of these emotions and feelings and things we care about, those particulars

• 1:06:18Speaker 01

  depend on the specifics of our, of our development. Hopefully, some of them overlap with various aliens, but probably not exactly. And it's very unclear how much, it's very unclear how other evolved aliens, how alien they will be. But like, those, those things are also in us, and they're also from us. And we might be the only source of that in the universe, and we might be, we are, we are very likely the only source of that within, you know, 100 million light years.

• 1:06:53Speaker 01

  And like, we also know about ourselves that we appreciate the, the fun and that we don't like the misery. We can look at ourselves and be like, "Wow." Like, we don't like the evil. We, and you know, it's subtle, we're like, we, it would probably be a tragedy to remove like sadism from humans entirely, but we're like, "Well, we want ethical sadism." Right? Like, pair your sadists with some masochists, have it all be like, like consensual and within the bounds of like ethics. Like, it's, it's not just, like, we don't want to, like, there is, there is some of

• 1:07:38Speaker 01

  our inheritance, some of our inheritance is in, is like very adjacent to the parts of ourselves we don't like, but we also can look at ourselves and see that we don't like that in aggregate, we like have these negative consequences, these negative externalities that no one intended. We can look at ourselves and say, "We don't like the impulses in us that lead us to like great atrocities." And humanity, like, the future, I think, does not look like a similar mix of humanity's virtues and humanity's vices. As we get smarter, as we get more capable, as we get better at solving coordination problems, as we get more time to think, as we get wiser, as we become more who we wish we were.

• 1:08:18Speaker 01

  Like, we on purpose, uh, uh, like, promote our virtues and demote our vices. Uh, and, like, is it tricky? Yeah. And do we know what a wonderful future looks like? No, it's, it's like, very subtle. You can't just go around giving everyone everything that they want, this, like, uh, or like, solving all the problems. Like, part of life is having, like, uh, like, obstacles to overcome and having, like, real choices that are meaningful, uh, and so on and so forth.

• 1:08:53Speaker 01

  But, like, we can, uh, we can make a world that is kinder, where the obstacles are, like, more meaningful. Where you don't have, like, terrible things happening to good people for, like, where the only reason is, that's the laws of physics. Like, everyone, like, many people, people say, like, everything happens for a reason. They sort of believe they live in a just world, but things don't happen for a reason here. Like, we can build a world where, like, your trials, uh, like, tend to give you things that were worth the trouble, that, like, pay off later and that you were glad for.

• 1:09:29Speaker 01

  And we can build a world where, like, uh, children aren't dying unnecessarily and needlessly because they were, like, were born in the wrong part of the world and got some terrible disease we haven't solved yet. Uh, and we can do so much better than that. Like, if we, if we, like, you know, transcend the, the bounds of humanity with, like, like, the technological limit is huge. Uh, like, I'm an old-school transhumanist and, like, think we should do something at least as cool as building Dyson spheres. Although maybe there are, like, better ways to put your stars to use. Um, like, I'm, I'm looking forward to the Matrochka brains if we, like, decide that's worth the effort. Uh, like, there's, there's so much potential. There's so much potential that this, uh, species has as one of, like...

• 1:10:17Speaker 01

  Perhaps the only source of like love, friendship, happiness, fun in the universe. Aliens may have other things and we'll care about that too, but it might not be quite ours. And if there are aliens, they're probably distant. And like, we can solve so many problems, especially if we have smarter friends who are trying to help solve them with us. If we can get, you know, artificial minds that are significantly smarter than us, significantly more capable than us, and that also are into this great project of like, the glorious, like, transhumanist future, full of like flourishing, happy civilizations, having good times. There's like, I can't, I can't describe the future specifically for you,

• 1:11:00Speaker 01

  because I expect it to look like foreign and weird and strange to me, and be full of people like pursuing desires that like I don't recognize. But there is so much upside. And like, yes, humanity has a lot of darkness in it too, but like, that's just one more obstacle on our way to the glorious transhumanist future that is easier to overcome if you have smarter friends.

• 1:11:28David Hoffman?

  That was, that was beautiful. Nate, thank you for just articulating what we get out of solving this alignment problem, because not only do we get to not die, but we get what seems to be kind of the inverse of that. So, thank you for, for walking us through everything you're doing and why the fight is worth fighting.

• 1:11:45Speaker 01

  Totally.

• 1:11:46David Hoffman?

  Cheers. Mantle, formerly known as BitDAO, is the first DAO-led Web3 ecosystem, all built on top of Mantle's first core product, the Mantle Network, a brand-new, high-performance Ethereum Layer 2, built using the OP stack, but uses Eigenlayer's data availability solution instead of the expensive Ethereum Layer 1. Not only does this reduce Mantle Network's gas fees by 80%, but it also reduces gas fee volatility, providing a more stable foundation for Mantle's applications. The Mantle Treasury is one of the biggest DAO-owned treasuries, which is seeding an ecosystem of projects from all around the Web3 space for Mantle. Mantle already has sub-communities from around Web3 on-boarded, like Game7 for Web3 Gaming and Bybit for TVL. And liquidity and on-ramps. So if you want to build on the Mantle network,

• 1:12:30David Hoffman?

  Mantle is offering a grants program that provides milestone-based funding to promising projects that help expand, secure, and decentralize Mantle. If you want to get started working with the first DAO-led Layer 2 ecosystem, check out Mantle at mantle.xyz and follow them on Twitter at @zeroxmantle. Introducing Polygon 2.0, the value layer for the internet. For too long, the limitations of blockchains have held back app development and stifled user adoption. The internet allows anyone to create and exchange information. What's missing is a value layer that lets anyone exchange, store, and program value. That's where Polygon 2.0 comes in. Polygon Labs has unveiled a series of innovations that will radically alter the Polygon ecosystem and Web 3 as a whole. By leveraging groundbreaking ZK innovations such as Polygon ZK EVM,

• 1:13:14David Hoffman?

  the next iteration of the best-in-class Ponki 2 proving system, and a first-of-its-kind ZK-powered interoperability layer, Polygon 2.0 will give users and devs unlimited scalability and unified liquidity. Right now, there is a Polygon improvement proposal regarding a potential ZK-powered upgrade of Polygon Proof of Stake. If approved, Polygon Proof of Stake would become a Layer 2 ZK EVM, Velidium. So make your voice heard on this proposal by joining the Polygon Discord today. You have a chance to help the Polygon community give the internet the value layer it deserves. Are you planning to launch a token? Is your token already live? And are you granting your employees and contractors vesting token awards? And are you trying to figure out how to take care of taxable events for your team? Toku makes implementing a global token incentive award simple.

• 1:13:57David Hoffman?

  With Toku, you will get unmatched legal and tax support to grant and administer your global team's tokens. Toku will help you navigate across the lifecycle of your token, from easy-to-use pre-launch token grant award templates to managing post-cliff taxable events with payroll. For legal, finance, and HR teams, it's a huge, complex task to have to comply with labor laws, payroll and tax obligations, tax reporting, and crypto regulations in every country that you employ someone. It's difficult, time-consuming, manual, and costly, and it's drawing more attention from global regulators and governments. Toku makes it simple for leading companies in the space: Protocol Labs, Hedera, Gitcoin, and many more. So if you want some help navigating the complex world of token compliance, go to toku.com/bankless or click the link in the description below.

• 1:14:40David Hoffman?

  Bankless Nation, we are here at Zuzalo, and I'm talking to Der, Tiran. Der, welcome to the show.

• 1:14:45Speaker 02

  Thank you. Good to be here.

• 1:14:47David Hoffman?

  Der, you want to just explain for people who don't know who you are, where you are, and what you're up to?

• 1:14:52Speaker 02

  So my name is Dayer. I am currently leading the AI Objectives Institute. We are a nonprofit research lab focusing on the question of alignment. And we are interested in building what tools would be able to add value to the current ecosystem to bring the future that we want to be in.

• 1:15:07David Hoffman?

  So an institute that is focused on solving the alignment problem sounds like that people who are at this institute believe that the alignment problem is existential. Am I on the right track here?

• 1:15:19Speaker 02

  Yes, very much so.

• 1:15:20David Hoffman?

  Okay, so we've been beginning all of our AI interviews with just asking the guest, what is your percentage of AI doom, of percentage likelihood?

• 1:15:30Speaker 02

  My personal percentage of AI doom is fairly low. For human civilization to end totally would be quite low. I would give it to 2 to 5%, while for us to end up in a future that is not desirable, more so existential risk rather than extinction risk would be very, very high, given the current dynamics we're living in right now.

• 1:15:51David Hoffman?

  Okay, so existential risk. Can you measure... So we... Chance of death, low. Chance of significant disruption and displacement in the hierarchy of life, high.

• 1:16:02Speaker 02

  Yes.

• 1:16:02David Hoffman?

  Right?

• 1:16:03Speaker 02

  Yes.

• 1:16:03David Hoffman?

  Okay, and so like, maybe you could illustrate like what that looks like. What is your likely scenarios here?

• 1:16:08Speaker 02

  The core principle that has brought the AI Objectives Institute together was that the AI revolution, as we call it, can bring an unprecedented level of flourishing to human civilization, but the current systems we are living in do not place us on that default path. We currently have a lot of incentive gradients that cause power to be concentrated. We have misaligned incentives in the form of nation-states to corporations to attention economy that distract humans from what we actually want to focus on. The AI systems right now are learning and copying these behavioral patterns, causing much more large-scale disruption in the landscape. And this...

• 1:16:49Speaker 02

  Propagating further, the doom scenarios that I am most concerned about do not look like nanobots crawling all over us all of a sudden, but look much more like economic failures, institutional failures, environmental failures, as we know today, at a much higher, much more unpredictable rate. And we already have mechanisms to deal with these things, but we do not know the scale at which we will be coordinating for this.

• 1:17:12David Hoffman?

  Okay, so you listed off a bunch of existential crises.

• 1:17:16Speaker 02

  Right.

• 1:17:17David Hoffman?

  And I think you're saying that just like AI is going to accelerate crises that are already in existence, rather than creating a net new crisis, although you're leaving some small amount of room for that.

• 1:17:25Speaker 02

  Right.

• 1:17:26David Hoffman?

  The bigger issue is that there are already human crises that we have, that we don't have solutions for, and AI is going to accelerate those.

• 1:17:33Speaker 02

  Exactly. I think the main crisis that I am most afraid of is, in the shortest term, within the next two to five years, we will have massive disruption to the institutions and systems as we know today. We see AI as an optimizer, and we have already other misaligned optimizers in the landscape. They do look like markets and corporations, they look like nation-states, they look like, you know, misaligned incentives that has driven mass-scale invasions that we're experiencing in the last, you know, two years with Russia and Ukraine, to massive bank collapses, to voting systems that end up in gridlock. These systems will be exacerbated in a pace that we are not yet familiar with. And that's to, in our perspective, from the AI Objectives Institute's perspective, there

• 1:18:18Speaker 02

  is a very strong continuum, rather than a sharp break, between human misalignment, as we experience today, and AI misalignment. Until we solve human alignment, talking about a purely AI alignment system feels superfluous, in my opinion.

• 1:18:32David Hoffman?

  So you're saying that solving alignment, there needs to be a correct order of operations there, and before we look to solve AI alignment, humans need to first look inward and solve our own alignment first? That's kind of the take?

• 1:18:44Speaker 02

  I think AI tooling in fact can be quite helpful for us to solve alignment at its base. And there is a lot of cross pollination that I think is necessary between understanding how we as humans so far before AI have been able to keep certain systems in check. Be it, you know, being able to give corporations a legal entity that can be interacted with. Or the way nation states have different systems that are checks and balances for each other. There's a lot of value to be driven from how AI alignment research can feed into current systems that have been experiencing different levels of misalignment. And how these systems and how we have dealt with this can feed into AI alignment.

• 1:19:25Speaker 02

  We see this as one central problem that AI is learning from. To put it in another spotlight, solving AI alignment, being able to align AGI to a specific set of human values and perspectives actually doesn't solve human alignment problem. It just pushes the problem from a silicon substrate to the socio-technical substrate. In which case, it might be much harder to control.

• 1:19:50David Hoffman?

  That's a really interesting, could you just elaborate on that? Because like, there are some human values that we want that I think I can claim without evidence that we think are good. As in like, don't kill me, and things like that. And so like, but I think what you're saying is like, if we go further down like what we think are good, we'll start to get into the very subjective realm. And if we start to align AI without defining what humans think are good, we can run into a problem.

• 1:20:21Speaker 02

  Right.

• 1:20:22David Hoffman?

  Okay, so my next question is like,

• 1:20:27David Hoffman?

  it's still a problem if the AIs come and kill us.

• 1:20:31Speaker 02

  Right.

• 1:20:31David Hoffman?

  Right, and so like,

• 1:20:34David Hoffman?

  maybe I'm a little bit lost with how to proceed here, but like, there seems to be an order of operations problem, which problem do we solve first?

• 1:20:41Speaker 02

  Right.

• 1:20:41David Hoffman?

  Right, and how do we even define?

• 1:20:43Speaker 02

  The problem space.

• 1:20:44David Hoffman?

  The problem space, yeah. So.

• 1:20:47Speaker 02

  There's this concept of differential technological development. What is most important here is to decide on the order of operations of which problems can be solved first, so that they can shed light into the next problem. This is like developing cars before seat belts are invented is more risky than understanding if we are to develop cars, we should have seat belts. The question of differential technologies are: what are the technological pieces that make sense to tackle first, so that we have the right substrate on which we can build the future that we want to build towards? So this is what we focus on at the AI Objectives Institute. What are the pieces that will be able to yield a safe-aligned AI downstream? And what are the pieces that are necessary right now to build first?

• 1:21:32Speaker 02

  The center of all of this is a coordination problem. I actually define existential risk as a failure to coordinate at the face of an existential risk. Is what makes existential risk come together. So there's a lot of tooling that we are focusing on right now on scalable coordination. There's a lot of tooling we're focusing on epistemic security and systems of alignment. These are three avenues that I think need to have much more research, so that we can mobilize together. We can identify the loopholes in our thinking as individuals, as collectives, and as systems, so that we can bring a level of systems alignment to in line with how we want humans to proceed. Only thereafter, we can start contemplating the scale of AGI.

• 1:22:16Speaker 02

  Now, I think it is quite a ways away for we still have enough time to be able to have by the time AGI arrives for us to have built institutions that are built on the backbone of scalable coordination and cooperation. That is why I think there is a lot of hope for us to be able to avoid total catastrophe. And I'm interested in thinking of it from that angle, which I think is quite necessary in the current alignment landscape. What are tooling that we can build right now that will bring an incremental net good? Rather than talking about what we should avoid, what we should not do. I would like to bring light to the world what we should do, what we should be focusing on today.

• 1:22:55David Hoffman?

  Okay, so it's your position that AI in the short term is going to produce immensely powerful tools. We can use these tools to help humans with their human problems that they had before AI ever came on the scene.

• 1:23:08Speaker 02

  Exactly.

• 1:23:09David Hoffman?

  One of those things is human coordination. And so we can apply AI to solve human coordination. Hopefully, this all happens before the AI doom alignment problem manifests that Eli Eiser talks about.

• 1:23:21Speaker 02

  Right.

• 1:23:22David Hoffman?

  And the idea is that we race to use AI tools to align humans amongst other existential crises, including our own ability to coordinate, and then we will be able to solve the AI alignment problem more head-on. Is this more or less your roadmap?

• 1:23:32Speaker 02

  Yes, I would say that the AI alignment problem is more or less the same problem. It is the natural extension of the human alignment, human coordination problem.

• 1:23:42David Hoffman?

  So because we cannot coordinate as humans, we can't coordinate on the AI alignment problem?

• 1:23:47Speaker 02

  We do not have mechanisms to identify what are the sets of values that an AI system should be aligned to. And I think it is quite short-sighted to say, let's just pump in more text to large language models, and at some point, they will be able to figure out. More detailed, you know, going to the weeds a little bit, solving single alignment is technically easy, and saying, you know, if we have one unitary agent that is able to be superhuman and superintelligent, they will figure out what is best for us, so we should fully focus on that. I think this is quite faulty. I think by the time we get to this stage, there will be many narrow AI applications that will be strong enough to actually put humans in a catastrophic risk. So we need to actually start from there. What will these narrow tools look like at the hands of misaligned state actors?

• 1:24:32Speaker 02

  What will these look like at the hands of, you know, exploitative corporate behavior? How can we make sure we can have safeguards around these as humanity, as a civilization? And will those tooling actually bring a better AI future? That is the intersection that we are interested in. I think the answer is there.

• 1:24:49David Hoffman?

  Okay, so the idea is that like, modern day, late-stage capitalism has produced large-scale corporations that are misaligned with humanity in general. And then you give them super-powerful, narrow, uh, artificial intelligence, and they just become misaligned faster. And that's like, one, one model, one application of where this could go wrong, and there's perhaps like, five or six or seven more examples like this.

• 1:25:12Speaker 02

  I mean, I would actually argue that we already live in this scenario. This is not the future. This has been happening within the last 10 years. I mean, let's, couple of simple examples. Insurance companies, probably at this point, have an AI system to decide which, um, claims they should reject immediately because they are least likely to be followed up on. One could say, "Well, this is the insurance company's job, and they are rightfully doing this." To me, this is actually a fundamental alignment problem. We already have an optimizer system, inside another optimizer system, that is the insurance company, that is rejecting claims that is causing human lives to be potentially at risk. And we have devised a society that has normalized this behavior.

• 1:25:56Speaker 02

  We have devised ways in which, you know, a corporate company, like, companies are able to hide the environmental externalities that they are building to the world, to the landscape. The questions I ask is, are AI systems able to share world models with us that will be able to help us understand these externalities better? To be able to incorporate that into fundamental decision-making? To put it in more fluffy words, can AI systems and our understanding through these tools elevate our sense of what humanity wants to be so that we know where we want to go? That is why I am hopeful about being able to build a future. This requires a lot of coordination. This requires a lot better epistemic security. This requires much more thinking about, how do we want to envision the institutions of the future?

• 1:26:41David Hoffman?

  So maybe you could just illuminate some of the strategies that you are working on at the Machine Objectives Institute?

• 1:26:47Speaker 02

  The AI Objectives Institute.

• 1:26:49David Hoffman?

  AI Objectives Institute?

• 1:26:50Speaker 02

  One line that we had on our website for a long while was, "Our objective is better objectives." Which is, it's almost tongue in cheek, as we do not think AI systems can have fixed objectives. Similarly, humans do not have fixed utility functions. In fact, the relaxedness of these is what gives flexibility to human evolution of thought, of our coordination, of our ethics. So, in some ways, the name is tongue in cheek on that front. But the goal is to come up with better objectives continuously.

• 1:27:22David Hoffman?

  Right, okay, so what are the, if we could drill down into the details of what it takes to come up with better objectives for artificial intelligence? Like, just the details, if you will.

• 1:27:32Speaker 02

  Right, for sure. So, we think of the society, let's look at the societal stack, from an individual level to a collective level, and then to a systems level. On the individual level, the core work is finding individual autonomy and sovereignty through bringing better epistemic security. The world that we are living in right now, especially in the Western world with democracies, a lot of this heavily relies on information transfer. People vote, people coordinate, based on the information they receive around the world. Now, we are entering a new paradigm in human communication where most of the content is about to be generated by AI systems. In this world, are we able to use AI systems to bring a different level of epistemic security and confidence?

• 1:28:16Speaker 02

  To have us understand, is the content I am engaging with a latent agenda? How can I stay true to my objectives as an individual? How can I stay true to my alignment? With the continuous flow of information that we are interacting with that is constantly fighting to hijack our bandwidth? This, at the lowest level, is the most important level. A lot of mechanistic approaches to alignment assume that individuals have a level of autonomy and sovereignty. They do know what they want. We actually start from that question. We do not know what we want. We do not always take actions. In line with our incentives due to bounded rationality, due to myopia, due to just pure distraction. How can we use tooling that comes from the AI landscape for that?

• 1:29:01Speaker 02

  So this is the first avenue of research. There's, we have come up with a research agenda that has some specific avenues that we would like to explore that we believe is a net incremental good. I'll go into some of the details on that.

• 1:29:14David Hoffman?

  Please, but first, just to really, just to make sure I'm understanding here. You started with the individual.

• 1:29:19Speaker 02

  Yes.

• 1:29:19David Hoffman?

  And I think that was an intentional choice.

• 1:29:21Speaker 02

  Yes.

• 1:29:21David Hoffman?

  Saying like, starting with optimizing for individual freedom and autonomy is a high level goal. A higher level goal than the rest of the stack, which I think we're about to go down.

• 1:29:31Speaker 02

  Right.

• 1:29:32David Hoffman?

  But could you explain, just elaborate on why we start with the individual and why that's important?

• 1:29:36Speaker 02

  Individual is the building block of the society. Every decision that we are making ultimately comes down to an individual's ability to understand the world that we are interacting with. We have devised, the systems that we are operating in right now assume individuals' ability to give feedback to a corporation, to a government, through our behavior on purchasing or through our votes. And we cohere around that. Assuming that AI or democracy or media, AI is just one form of superintelligence. We have devised many other forms of superintelligences in human history. Assuming that the individual maintains a level of autonomy and sovereignty throughout this interaction as we live in the world is what has caused the crisis that we already are in.

• 1:30:21Speaker 02

  This is not in the future. We already are in this scenario. Let's look at social media. It's the archetype example. In 2008, we thought, you know, this is going to be a revolution that brings us a level of connectivity, of mutual understanding, that will heal democracy. Instead, we ended up with echo chambers. We ended up with massive epistemic fracturing with respect to what facts people believe in. We found people that get locked in more to their own bubble echo chambers.

• 1:30:51Speaker 02

  We could have foreseen some of these stuff. This all sheds light into it's ultimately the individual autonomy and sovereignty. That is the core building block of civilization. Then the question we ask is, is the AI tooling of today able to bring a different level of epistemic security? Our answer is yes, and this is very worthwhile to be trying.

• 1:31:12David Hoffman?

  Epistemic security, so epistemic, can you just define that term?

• 1:31:15Speaker 02

  Yes, the information that you are receiving, you know where this is coming from. You know, you have a sense on what this information is trying to accomplish or whether it is true or false, how you relate to this information, how you want to relate to this information and how you want to participate in the world given this new information. Currently, a lot of these systems are actually quite shaky in the society we live in today.

• 1:31:42David Hoffman?

  We're in a post-truth world.

• 1:31:44Speaker 02

  Right, and we are instead of securing this, we are saying, let's come up with a level of generative AI capabilities that can flood even more information. And then we are talking about what will AI be aligned to? The question is, do humans even have bandwidth to be able to share this information?

• 1:32:01David Hoffman?

  Okay, so epistemology, the study of knowledge, epistemic security is just like securing the ability for the individual, if, so they have the choice to, sometimes people just want to live in the cozy comfort of being fed the information that they want, but importantly, giving the individual the choice to have access to truth is a basal building block, you're saying, to talk about the rest of the societal stack?

• 1:32:28Speaker 02

  Exactly.

• 1:32:28David Hoffman?

  And we're going to use AI tools to improve that part?

• 1:32:31Speaker 02

  So a couple of projects we are working on right now on this front, that is, the core building blocks is, can we use AI technologies right now, AI techniques right now, to be able to inform certain patterns of perception? For example, I'm not even going to go into whether or not the content you're interacting with is true or false. I will instead start from the side. Is the content you're seeing designed to elicit anger from you? Is the content you're interacting with designed to trigger an addictive loop? Giving your dopamine high really fast. Is it eliciting anxiety? Is there a latent agenda in this content? Or is there a subcultural affiliation in the content that you are interacting with?

• 1:33:13Speaker 02

  Is it using language that is geared towards a certain subgroup? Is it repetitive? Is there evasiveness? Is it stating beliefs? Or is it a response to something else? Turns out, large language models are actually really good at detecting these kinds of patterns. Because what they are doing is pattern matching and predicting the next token. So a language model is able to say, oh, this next token is unusual. It looks like the next token, the difference here, is guiding me towards a different future. So this is one building block that on its own is able to add a lot of value to the language. So we're building a prototype called Lucid Lens, which evaluates the content that you're interacting with continuously

• 1:33:54Speaker 02

  to be able to guide you into, hey, it looks like you have been on a dopamine loop lately. Do you want to shift what you want to do? Or it looks like you're engaging with content that is extremely repetitive in this discourse. Can we bring a level of intentionality to your objective alignment personally?

• 1:34:13David Hoffman?

  Okay, so you're building that system.

• 1:34:16Speaker 02

  Right.

• 1:34:16David Hoffman?

  How do you make sure that that system isn't biased?

• 1:34:19Speaker 02

  Right.

• 1:34:20David Hoffman?

  Because like, maybe the repetitive loop of iterative content is the correct thing. And then you're saying this AI suggests to humans, hey, maybe you should get out of your repetitive loop. How do we know that that's not an unbiased thing to do?

• 1:34:34Speaker 02

  Right. There's a couple approaches here that we can take. One of them is, can we ground language models to an individual's own affiliations? There are some people working on this front. The, I'll give a fairly simple answer. A heads up, like something like a chimney cricket on your shoulder that can point out, hey, you might be stuck in, you know, a rage bait, you know, click, scroll for the last while. That check in is, in the right context, is not harmful to an individual. You might say, yes, I acknowledge that. I want to continue. I want to proceed. I like where I am at. Rather than have there be a sharp judgment of the nature and the quality of the content.

• 1:35:17Speaker 02

  But another part LLMs are able to do very well is to, um, being able to fetch further information that can give you a larger picture around the content that you're interacting with. Yep. This is just one of the many avenues. I will zoom back from lucid lens. Another one is can language models replicate an individual's affinities to the point that they are able to help us stay grounded in a set of objectives that we want to be in. This we say mind, this we call mindful mirror is a different avenue of research. You can think of it this way. The first one is about a machine helping a human stay grounded to their objectives. The second one is helping a human stay ground to their, sorry, a human to give the same

• 1:36:00Speaker 02

  feedback to a machine to say, here are things that I would like to prioritize. These are what is good to me. Having an individualized personal language model that is secure, that is not living in a large company's servers, but that is completely owned by you, having this to be able to secure your understanding of the landscape of emotions, of content that you're dealing with, that can help you ground yourself in a moment where you are lost. This is an incredible piece of technology that can actually create net value to the society.

• 1:36:33David Hoffman?

  Okay, so I think I can categorize your mechanisms into two different camps. One, in the crypto world, we use this idea of credible neutrality quite frequently. And there are some mechanisms that are credibly neutral, which aren't to say like, hey, what you're doing is bad, but they're just little alerts saying like, hey, you've engaged in very repetitive YouTube rabbit hole type behaviors. I'm just gonna let you know that that is what's happened. Without saying anything negative or positive, or suggesting a rerouting, just like a neutral mechanism to perhaps bring you out of the hole and let you know that this is perhaps a, this is perhaps a danger zone, or perhaps not.

• 1:37:11Speaker 02

  Right.

• 1:37:12David Hoffman?

  And just for you to be able to step out of your consciousness tunnel.

• 1:37:15Speaker 02

  Right.

• 1:37:16David Hoffman?

  And zoom out and just like, just like, like that in meditation, sometimes they ding that gong.

• 1:37:22Speaker 02

  Right.

• 1:37:22David Hoffman?

  Right. And just like, hey, if you're lost in thought, like ding the gong.

• 1:37:25Speaker 02

  Right.

• 1:37:25David Hoffman?

  And so like, it's a way to just get you to snap back out of it if you are in it.

• 1:37:29Speaker 02

  Right.

• 1:37:29David Hoffman?

  And so the credibly neutral mechanism, which we love. And then the other one is. Being able to customize your own personal LLM to align with your desires. And since you are the one implanting your biases into the LLM, we also feel that since we are not imposing that upon others, we're only imposing that upon ourselves. That also checks the box of credible neutrality.

• 1:37:51Speaker 02

  Right, and I would like to underline one thing, which is that the large language models, the goal isn't to create a locked-in version. Like, there's many fault lines that we can fall through here as we are building these, which is why the goal isn't to launch this as a product that widespreads use right away, but to actually approach this from a question of rigorous study to the point that we make sure these tools are doing the things we want to do. What are their failure points? For example, a very terrible way to do this would be to cause individual value lock-in, in which case, you know, a language model constantly reaffirms things to you from your past state to the point that it prevents you from moving forward. I am more interested in a language model that is able to give me awareness of my drift

• 1:38:35Speaker 02

  through my own evolution of thought. These are some of the aspects that are actually very fundamental to the question of alignment. How is the landscape of values shifting for an individual?

• 1:38:46David Hoffman?

  How am I changing as a person?

• 1:38:48Speaker 02

  Right, what was important to you? There's a series of simple questions. Are my beliefs consistent with my beliefs? Are my beliefs consistent with my actions? Are my actions in consistency with the community that I'm living in and their beliefs and their actions? Being able to have visibility into these, these systems are incredibly fractured right now, and we are building even more walls. I believe AI tooling can actually help us overcome some of these to be able to understand, oh, it looks like what I wanted to do, I am not able to do it, given the incentive gradients I am existing in right now. The crypto world has suffered from this quite a bit. How can we have this be more visible to everyone? I believe questions of contemplating

• 1:39:30Speaker 02

  what values do we want AI systems to be aligned to, etc., really require a level of rigorous understanding of self first. So this is the first step. Then we move to the next category, which is: How can we scale this up to the question of a collective, which is the next tier in human society?

• 1:39:49David Hoffman?

  Right, and before we get to there, I just really want to drive something home. It really sounds like we're trying to allow AI for humans to become the best versions of themselves.

• 1:39:59Speaker 02

  Right.

• 1:39:59David Hoffman?

  Right, if we want to talk about some like, some people in the psychology realm, Nietzsche would call this the Ubermensch, right? Like the Uberman, the Superman, the best, the literally becoming the best version of yourself, because that also scales up in society, right? If you, as an individual, become the best versions of yourself, that makes you a better community member, and that makes communities better, which I think is kind of where this idea goes.

• 1:40:25Speaker 02

  I call it extended cognition. If I am able to have an ability to understand myself cross-sectional through my own history, based on what I have engaged with, based on what I have thought, what I have written, that is powerful. We can act somewhat like the scariest applications are also the most worthwhile. If this data is compromised in any way, that also creates, you know, a mirror of me to exist in the society. So then the question becomes, how can we do this in the most secure manner that is really within your own autonomy and sovereignty? Because we already live in a world with very difficult attention economy, that everything is competing for your attention, for your beliefs, so that you can vote or so that you can purchase a certain way. It's very important for us to be able to bring

• 1:41:10Speaker 02

  a level of autonomy to the individual as AI tooling proliferates in this landscape.

• 1:41:16David Hoffman?

  Okay, let's move up the social stack. Community comes next?

• 1:41:19Speaker 02

  Yes, collective.

• 1:41:21David Hoffman?

  Collective.

• 1:41:21Speaker 02

  Question here is scalable coordination. This is where I am most passionate about. I think there is incredible value to be added here. In some ways, you know, there's many alignment labs. We do share, you know, our concerns are in line with a lot of the other spaces like Miri or Redwood, where there is massive challenges that are coming in. And the question is, why have we not yet been able to coordinate at a scale where we have lined up our incentives as a society so we can tackle these AI problems? Instead, we ended up in a race dynamic, where, you know, entities that are spun up to counter end up participating in the race dynamic. For this, being able to level set the understanding of what is true across every participant,

• 1:42:06Speaker 02

  being able to bring visibility into what are different perspectives that exist in society right now? How can I engage with these more effectively? How can we come up with collective decision-making systems? So that a collective can find its alignment. The first category was about an individual finding their alignment. The second category is how can a collective find their alignment?

• 1:42:27David Hoffman?

  So you corrected me when I said "community" and to replace it with "collective." I think the reason why you did that is because like, a community seems to be like a handful of people, right? A hundred people or a thousand people in a town. But a collective is like the hive mind of these people. And that is the thing that we are trying to produce alignment for. Is that my interpretation?

• 1:42:50Speaker 02

  Yes, and I would say I am in a collective with a lot of people that are not in my community necessarily. A community is a more intimate collective. Depending on how, and you can say, you know, well, there is different ways to cut the social strata that we live in to say, you know, these are different categories. All of these are valid. The thing I am interested in is, say we're able to align AGI to one human. What do we align AGI to now? How can we come up with definitions for collectives? Yeah.

• 1:43:18David Hoffman?

  We recently did a podcast with a guy, Tim Urban, on the subject of liberalism. And he had this great illustration of higher mind thinking versus lower mind thinking. And then higher mind is like a genie and lower mind is like a golem, right? Like, just like, a golem's just like dominant punches. And a genie is like magical and higher. And then when you have collectives, if you have a society that is, a society dominated by lower mind thinking, primitive mind thinking, like reptile brain, you have a collective golem. But then if you have a society based on higher mind thinking, higher order thinking, using their more recent parts of their developed brain, their prefrontal cortex, then you have a collective genie.

• 1:44:01David Hoffman?

  And this thing can actually, even if it engages with a different collective genie, so you have like the Republican genie and the Democrat genie, two genies can actually make progress together, right? They can actually come together and produce a roadmap, whereas two golems just come and fight. And so this is actually a similar subject that we've had on the podcast, and maybe a way to illustrate this.

• 1:44:24Speaker 02

  I see some parts of this to be in line with our thinking. The way we have been developing AI systems are much closer to a golem right now. I'll get into that.

• 1:44:35David Hoffman?

  Hence, the downstream problem of AI alignment is human alignment. I see it.

• 1:44:40Speaker 02

  Right. I have thoughts on how to make AI systems be more like genies as well. We can get to that in a little bit, but to not to change course of the discussion. One of the projects we have on this front is called Talk to the City. And this is a digital town hall that you can summon out of unstructured feedback that you have collected from a polity. We currently have voting systems where you're sending one bit of information to the government every four years, and you're hoping that they will be able to represent this the best. We have been used to categorical information on voting, multiple choice, and referendums.

• 1:45:19Speaker 02

  The question is, am I actually able to share my perspective in its real, more true to my own perspective through human language, and have a central entity receive this information and make decisions based on this information? Talk to the City is a prototype that collects different unstructured text feedback from the entire community that we're looking at, synthesizes into a set of different perspectives that exist in the community, and train conversational language models for each of these perspectives, so that you can have these perspectives talk to each other, or as an individual, be it, you know, you're a policymaker, or a journalist, or just a citizen in this city, be able to

• 1:46:03Speaker 02

  engage with all of these reasons why. The goal here is not to find consensus. The goal here is to understand different viewpoints, so you can make sure you can address these. In some ways, one of the problems that we would like to solve, and this again goes back to the real alignment problem. It is easy to find the lowest common denominator across everyone. And that causes a lot of short-sighted policymaking. That causes a lot of, I mean, political theories, seeing like a state explores this in depth.

• 1:46:31David Hoffman?

  I've read that book, by the way. Great book.

• 1:46:33Speaker 02

  It's incredible. It's very helpful for shaping our thinking. I am interested in being able to understand a more in-depth policy. How is this going to actually impact people? What are the reasons why this may be bad for some groups, yet still creates a parity improvement over the current state? These require a level of sophistication that isn't about saying, "Okay, it looks like everyone agrees on do not kill humans, so let's codify that." Can we actually understand, "Okay, but I'm interested in having more resources for my village, and this requires a compromise with something else." If the proposal is, you know, should we build a road from A to B, yes or no? The answer might be, we should build a road from B to C.

• 1:47:16Speaker 02

  And being able to give a community to express this, and how this be received systematically, is something AI tooling can actually bring to the landscape right now. That can bring a different level of collective coordination capability.

• 1:47:31David Hoffman?

  So, okay, so it sounds like at the individual level, we have these language models that can be perhaps perceived as like our personal assistants, our personal, like our series, our Google, our Microsoft Cortanas, to help us think and help us know and help us learn.

• 1:47:46Speaker 02

  Right.

• 1:47:46David Hoffman?

  And then, that can amalgamate to a higher order LLM

• 1:47:52Speaker 02

  Right.

• 1:47:53David Hoffman?

  that, like you said, doesn't come to consensus on our behalf, but allows us to see when you aggregate everyone's personal assistance, what does everyone believe? And it processes that data of individual beliefs into something for us to reason about and understand and move forward.

• 1:48:10Speaker 02

  Yes.

• 1:48:10David Hoffman?

  Now that we understand that.

• 1:48:11Speaker 02

  Right. I am not interested in the genie that you're talking about to make policy for us.

• 1:48:16David Hoffman?

  Right.

• 1:48:17Speaker 02

  I am interested in this system to be able to show us, these are the considerations we need to take into account that people have voiced. This is something that AI tooling can help us with today. Right now, which is why we are building it. And that brings a level of scalable coordination and cooperation capability. Another aspect here is to finding positive some outcomes that individual groups may not have been able to see. You know, the solution to, you know, should we build the Keystone XL Pipeline? The right answer might be, well, the hydroelectric dam will still bring the same energy and workforce to the region without actually having an environmental pollution caused.

• 1:48:52David Hoffman?

  So it can produce emergent, um, emergent optionality.

• 1:48:56Speaker 02

  Right. And these are things that the systems we are building are actually really good at. Um, I would rather not yield the level of reasoning and decision making to an AI system. But in the near term, we are able to use these as building blocks to improve the capability for humans to cohere with each other. Now, what's really interesting here is we are getting two birds with one stone if we build these pillars. The first one is these tools are net good for humanity. We need these right now to be, to solve human alignment, to take a step towards human alignment. But also, these tools will create data sets for AI alignment as well. It will be able to show how humans have cohered around decision making on specific world models.

• 1:49:39Speaker 02

  Or have humans say, yes, this AI system actually was able to represent me through time. We currently don't have this data, yet we are talking about solving AI alignment. Don't get me wrong, I do think, you know, mechanistic interpretability and a lot of machine alignment research is incredibly important as well. We also need to look at what does this mean for humanity? Otherwise, we will just push the problem of alignment downstream and make it much harder to solve.

• 1:50:04David Hoffman?

  Right, right. Okay, so, so what you're saying is like when we have this collective consciousness that we are able to reason about via all of our native large language model assistants coming together and pow-wowing about what everyone believes, we'll start to be able to lock in some beliefs. Um, we'll probably lock in the idea that killing humans is bad. Uh, that'll probably happen pretty first. Everyone will come, will be in agreement about that. So then we can use that shared understanding that killing humans is bad as a way to. Codify that into law about AGI, right? And then we perhaps can go higher from there and be like, okay, now that we understand that everyone believes that killing humans is bad, we can also lock in that theft is bad. And then we can start to get higher up the stack of what we believe

• 1:50:49David Hoffman?

  and then use that to operationalize more powerful AI. Is that the path here?

• 1:50:54Speaker 02

  I very much disagree with the framing.

• 1:50:56David Hoffman?

  Oh, no, okay, please help me, help me understand.

• 1:51:00Speaker 02

  Value lock-in is a concept that is well explored in AI alignment and effective altruism and similar landscapes. The goal is not to lock in values. The goal is to build systems where people, the polity, is able to continuously give feedback and participate into an ever-evolving consciousness. Instead of one AGI that has learned the moral code and then can proceed, what is necessary is a system that can continuously take feedback from people as the value landscape shifts, as more unpredictable events happen. This is actually, the crypto world has lived through this many times. As you know, the incentives shift, as speed of progress shifts,

• 1:51:45Speaker 02

  all the way from gas fees to coordinating how purchases can be made. We need systems that are actually resilient towards updating based on how the landscape is changing. So I would be quite worried about building systems that can learn and fix something in place perpetually. Much more so, I would be interested in systems that are trained to fetch new information, to understand how the ground is changing throughout time. Most humans, I mean, some values I hope we perpetually agree with, such as do not kill, do not cause harm. But then again, what does harm mean? What does this mean in case of euthanasia that might be an opt-in from the individual? How does one proceed? We already are living in a world

• 1:52:28Speaker 02

  where we are exploring these kinds of questions before AI. The AI systems, I think it's quite dangerous to force AI systems into you need to find what the optimal version is and enforce it to the rest of the world. The right way to go is build an AI system that can learn from humans on where humans want to go at any given point and bring us towards there and have there be a level of courageability.

• 1:52:53David Hoffman?

  So like, there's a rule of thumb that I've come to understand in the crypto-economic world, which is called "no magic numbers," as in when you build a crypto-economic system, if you just pick a fixed number, that is a point of rigidity and fragility. And I think that's perhaps what you're saying about when we train our AIs to be aligned with us, any sort of rigid or fixed parameter can create fragility and long-tail consequences that we don't understand.

• 1:53:19Speaker 02

  Right.

• 1:53:20David Hoffman?

  Okay, so we're in agreement there?

• 1:53:21Speaker 02

  Right.

• 1:53:22David Hoffman?

  And so, okay, so you didn't like when I was saying, like, "Hey, all the humans agree that killing is bad, let's lock that in."

• 1:53:27Speaker 02

  Right.

• 1:53:28David Hoffman?

  Maybe I'll rephrase and say, like, "In this one moment of time, all of the humans of a local collective, all, in that one moment of time, agree that humans are killing is bad, so the AI that's reading that data will, in that moment of time, choose to not kill humans." Is that a better way to describe it?

• 1:53:46Speaker 02

  Yes, and I would rather have us have systems that don't necessarily give AI the ability to be able to kill humans in the first place, but more so see these as the tools that they are. They are super-intelligent things we can consult and learn from, and iterate from there. But yes, the AI system's values should be able to evolve as human systems evolve. And it's really downstream of human autonomy and sovereignty into collective decision-making that can bring the systems to be able to be aligned.

• 1:54:16David Hoffman?

  Okay, so, okay, so we started the individual, we've moved up to the collective, is there a next step?

• 1:54:22Speaker 02

  Yes. The last step is systems level.

• 1:54:24David Hoffman?

  Systems, okay.

• 1:54:25Speaker 02

  Systems are more complex than a collective. In a system, we don't only have multitude of people, but entities that have their own capabilities, that have their own agenda. For example, a corporation consists of individuals, but it has goals, it has affordances that go beyond what any of these individuals can do. Furthermore, we have developed systems in the world that, you know, don't hold the individuals accountable for the failures of corporations. This is the side where we have seen massive problems with misalignment throughout human history, both in terms of, you know, states, governance, and we have toppled many systems, you know? Divine right of kings was impossible to over Come, yet here we are. Yeah, right. Communism was the same. There is a lot of systems that we evolve.

• 1:55:09Speaker 02

  We currently operate under a capitalist system that is heavily governed by incentive gradients. That have caused a lot of shifts for how even nonprofits that are developing AI have shifted their priorities towards monetization and productization. So the question then becomes, how can we design systems that can stay aligned to the betterment of the collective, to the betterment of the individuals? Everything goes back to, you know, is this actually producing well-being for the participant? Or is it good-hearding something? Goodhearts law, as in, when you pick a measure that becomes a target, it ceases to be a good measure. We live in systems that already do this. This is precisely why we don't want an AI to lock in and create rigidity,

• 1:55:55Speaker 02

  but design systems that can continuously evolve through that window.

• 1:55:59David Hoffman?

  Okay, so this seems like the hardest problem.

• 1:56:01Speaker 02

  Yes.

• 1:56:02David Hoffman?

  It also seems like the frontier of coordination problems that humans have arrived at in the grand scheme of things.

• 1:56:09Speaker 02

  Right.

• 1:56:10David Hoffman?

  And we still haven't tackled that problem.

• 1:56:12Speaker 02

  Right.

• 1:56:13David Hoffman?

  It sounds like perhaps, understanding that that's the foundation that we are at, we actually might need AI to solve that problem and not be able to solve that problem without it.

• 1:56:22Speaker 02

  We haven't solved this problem.

• 1:56:24David Hoffman?

  Right.

• 1:56:24Speaker 02

  One thing I say is, our work, the AI Objectives Institute, in some sense, the word AI is not that relevant. This work was relevant 300 years ago. I think this world will be relevant, this work will be relevant post AGI as well. This is a question of how do we design structures, institutions, that can stay aligned to the collective? This is a millennia-old problem. AI is just the newest building block in this story. It's a very critical building block in this story that can cause a lot of damage if we don't do it right, which is why I have high fears of existential risk. If not extinction risk, but that is possible as well if we don't coordinate.

• 1:57:06Speaker 02

  I believe we would be able to coordinate and build better institutions, which is what I want to work on.

• 1:57:11David Hoffman?

  But like, do you think that actually solving the point of like,

• 1:57:16David Hoffman?

  like, at the systems level, we have borders. And that's kind of like a coordination breakdown. The fact that different countries operate by different rules and coordinate differently.

• 1:57:26Speaker 02

  Yes.

• 1:57:26David Hoffman?

  And so, and then there's also different economic systems, right? Uh, different, uh, different systems are, they're disparate, they're disconnected. It would be better if there was a single global system, along with all the other problems that we've had with Stalinism and other like atrocities throughout the nineteen hundreds. My, my question to you, and I'll just reiterate it, is like, that's the frontier of human coordination, that, you know, we've solved it at the tribe, we solved it at the, at the community, solved it at the city, we solved it at the nation-state level, haven't solved it above that, and honestly, haven't really solved it completely at the nation-state level either.

• 1:58:00Speaker 02

  And we haven't solved it on an individual or collective level either, but we are taking steps towards all of these.

• 1:58:05David Hoffman?

  It's solved at, uh, it's solved more or less at different parts of the stack.

• 1:58:10Speaker 02

  Yes.

• 1:58:10David Hoffman?

  And my question to you is like, do we need AI to take the next step in solving it at a more systemic level?

• 1:58:17Speaker 02

  I actually see it from the opposite way around. Given we are building AI, AI-driven institutions, given the workflow that we have right now, we'll yield institutions that have AI in it, we have to look into how AI will interface with this.

• 1:58:34David Hoffman?

  Right.

• 1:58:35Speaker 02

  If we say, you know, let's turn off AI so we can solve this problem for another couple centuries or millennia, sure, we can do that as well. We don't live in that hypothetical. AI is here, it is present. I would ask, how can this be helpful for institution design? Yeah.

• 1:58:50David Hoffman?

  So is, uh, would you agree that AI, uh, all these language modules, everything we're talking about here, is both the problem and the solution at the same time?

• 1:59:00Speaker 02

  It is the problem right now because we haven't yet been able to solve human alignment. Hence, Any super intelligence, any super competent entity that can excel human capacity can be dangerous, such as misaligned, you know, state actors or exploitative corporations. Right. Of course, an AI will be very dangerous as well, so we need to tackle this problem. Right. So, we have a couple of experiments on this front that are also incredibly important. And these are the monoliths that require much more coordination and help than one group to solve. So, our goal is to foster an ecosystem that has many approaches, all the way from, you know,

• 1:59:42Speaker 02

  crypto is super important, ZK proof is super important on this pillar. We are interested in building a proof of concept of an open agency architecture system that can help an institute. Our goal is to showcase that an institution can make decisions based on feedback from a larger collective, based on expert opinion, in a level that is transparent and visible and interpretable, rather than a complete black box. And we believe that it is incredibly important for AI tools building institutions and AGI building institutions to follow this. And also, it is important for the AI systems to be built on top of this principle as well. Yeah.

• 2:00:24Speaker 02

  And open agency is a concept that Eric Drexler has pushed forward. The, a simple explanation would be, can we shift our thinking from agents, which are singular monoliths that have fixed goals, that have low visibility, that have their own ways of doing things, towards agencies, which has different faculties and different tasks that are being passed around as it reasons about the world and as it takes actions into the world. Building more open agency systems, rather than closed monolithic systems, isn't it good? Designing institutions that operate this way isn't it good?

• 2:01:06Speaker 02

  And similar to the question of human alignment, we have been doing this for a couple millennia. We have iterated through different governance structures towards more visibility, towards democratic systems. We are going to continue with these paradigms as now AI tooling enters the picture as well.

• 2:01:23David Hoffman?

  Yeah, I really, I really just want to kind of drill down on like my understanding of how AI fits into the stack because I think your, your big message that you have is AI is yet another thing that we need to figure out how to align along with all the other things that we need to figure out how to align.

• 2:01:42Speaker 02

  But my It's quite, it's a very urgent one.

• 2:01:44David Hoffman?

  Very urgent one, right. And, and my mental model is that like, if we can, AI is unique from the other systems that require alignment in the fact that we can use it, it's special, it stands, it stands out from the rest of the problems in that if we can align AI, we can align everything else and we actually might require AI to align everything else so that we can also align AI.

• 2:02:09Speaker 02

  I would say it stands out in some ways for it has certain capabilities that hasn't been emerged from others. I highly doubt if we can align AI, we can align everything else. We can, we can solve a version of AI alignment where you can align a single AGI towards a single set of human values that then can go and rapidly destroy half of the world and we end up in a Thanos scenario or we end up in a Moloch scenario where you end up an aligned AI towards a certain set of values that exploits towards building more and more resource takeover. These are not good cases of alignment. A multi, multi

• 2:02:49Speaker 02

  alignment case where we're looking at, there is a polity that has different perspectives. How, and these can be represented as different agents, these can be represented as different general AI systems. I don't only, like, we are more likely to end up in that scenario already. We have multiple tools, we have independent groups that are building many tools. These tools require a level of coordination between each other. These tools need to be able to have their own interpretability to the people they are accountable to. So, I highly doubt we will end up with one monolith, but the question is, just like how we are trying to solve coordination today with many

• 2:03:30Speaker 02

  If not AGI, more narrow tools that are still capable of massive damage to human existence, how do we create alignment across all of these systems?

• 2:03:39David Hoffman?

  Okay. So if you would, Derek, can you like speed run us through the version of the universe that you hope to see? Like if you and everything that you want to see happens, what does that universe look like over the next five to fifty years?

• 2:03:54Speaker 02

  Sure. I am excited about the value AI tooling can bring to the universe. We will have systems that will help us discover ourselves better. We will have systems that help us give visibility to our priorities and see how this is acknowledged by the rest of the world. We will have systems that elevate humanity towards what it wanted to be, rather than avoid what we afraid we would become. I am interested in a world in which I understand the participation that I want to make. How does this contribute to the world that I have bandwidth to explore and play? I would like there to be a world where everyone is able to have bandwidth towards their hobbies, towards the joys that they're bringing in to pursue.

• 2:04:39Speaker 02

  I want a world in which we are able to see how our opinion counts in a larger system that is making decisions, that is interpretable and accessible. I want there to be more human connection. Ultimately, it's around being able to have humans interface with each other more, not less. That's why the core of this all the way comes down to the coordination problem. Can we have humans see eye to eye, understand each other? Can we have the AI tooling reduce the barriers? Reduce the incentive gradients that are shaping up right now that prevent humans from finding more agreement, more shared values, more shared joys with each other? This is what I'm most preoccupied with. The world I'm afraid of is one that, we talk about this a lot in our team,

• 2:05:21Speaker 02

  numbers that we have decided to care about are going up and up and up while we don't necessarily have more human flourishing. That is what I'm afraid of. AI tooling can bring us an unprecedented level of human flourishing. The default systems do not place us on that path. And I would like us to go towards there. And I think this is Possible. I think the tooling that can be built today already takes massive steps towards here. This tooling, we are interested in building this tooling for humans to use. We are interested in building this tooling so that the AGI labs can adopt an open agency architecture, so they can make more grounded decisions on what the collective is interested in, what is safe, what is interpretable.

• 2:06:03Speaker 02

  There's a lot more in there that we didn't go into in detail on how we can have systems that can make decisions that are by design, safe and verifiable. All of these will be a net plus to the world we are living in. I don't think we will solve human alignment, but I think AI tooling can take a massive step towards that direction. That is the world I'm interested in.

• 2:06:23David Hoffman?

  Dara, I get the intuition that you are an optimistic person. Is that correct?

• 2:06:27Speaker 02

  I would say so, yes. I think we need more optimism in this landscape so that we can see what we want to do. I think there's a lot that can be done and I have many fears as well, but I think these fears can be solved if we get to the level of coordination.

• 2:06:44David Hoffman?

  Dara, if people are piqued by this conversation and they want to learn more about aspects of this conversation, where should they go?

• 2:06:52Speaker 02

  Check our website, objective.is. That is, send an email to our team. Hello, what is it? At, message to me, I guess, dagger at objective.is. Would love to chat. If any of you are interested in helping in creating this vision, come along. We need many, many folks to bring this together and make it a possible truth for us. So, yeah.

• 2:07:17David Hoffman?

  Yeah, thank you so much.

• 2:07:18Speaker 02

  Yeah, thank you.