LIMITLESS - The Intelligence Curse: AI Makes Us All Obsolete | Luke Drago & Rudolf

Welcome to Limitless. Today we’re joined by Luke Drago and Rudolf, authors of the powerful essay series "The Intelligence Curse." Together, we explore a future where artificial general intelligence (AGI) threatens to upend the economic and social contracts that underpin modern civilization. Will AI empower us or make us obsolete? We unpack how labor-replacing AI could dismantle the very incentives that once gave rise to liberal democracies, social mobility, and human-centered innovation—and what it might take to build a future worth living in. ------ 💫 LIMITLESS | SUBSCRIBE & FOLLOW https://pod.link/1813210890 https://www.youtube.com/@Limitless-FT https://x.com/LimitlessFT ------ BANKLESS SPONSOR TOOLS: 🪙FRAX | SELF SUFFICIENT DeFi https://bankless.cc/Frax 🦄UNISWAP | SWAP ON UNICHAIN https://bankless.cc/unichain 🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle 🌐SELF | PROVE YOUR SELF https://bankless.cc/Self 🟠HEMI | BTC & ETH, ONE NETWORK https://bankless.cc/hemi ------ TIMESTAMPS 0:00 What is the Intelligence Curse 4:29 Resource Curse 8:20 Pyramid Replacement 18:19 Institutional Pushback 21:25 Capital, AGI & Human Ambition 32:00 Liberalism Falls Apart? 36:30 Powerful Actors 41:15 Rentier States 46:19 Human Labor in a AGI World 52:46 Nation States 57:37 Shaping the Social Contract 1:06:23 AI Snake Oil? 1:07:41 Balance of Power 1:08:51 Breaking the Intelligence Curse 1:16:45 Vitalik’s Defensive Accelerationism 1:18:16 Diffusion 1:19:58 Open-Source AI 1:22:06 Democratization 1:24:06 Who Wins? 1:26:43 Action Items 1:29:17 The Positive Scenario 1:30:21 Closing ------ RESOURCES Luke Drago https://x.com/luke_drago_ Rudolf Laine https://x.com/LRudL_ Time Op-ed https://time.com/7289692/when-ai-replaces-workers/ Intelligence Curse https://intelligence-curse.ai/ Contact Form https://docs.google.com/forms/d/e/1FAIpQLSft2iBV9z1AYsM3TcDnh8z3juc2k4yD0TQTZ91oy37S-KlSSQ/viewform AI Snake Oil - Arvind Narayanan https://press.princeton.edu/books/hardcover/9780691249131/ai-snake-oil Vitalik’s Defensive Accelerationism https://vitalik.eth.limo/general/2023/11/27/techno_optimism.html ------ Not financial or tax advice. See our investment disclosures here: https://www.bankless.com/disclosures⁠

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Ryan Sean Adams?

  Hey Bankless listeners, this is Ryan here. You guys are in for a treat today. So the episode you're about to hear was pulled from our new Limitless podcast feed. That's a podcast feed that I occasionally host. I'm a guest host on that, along with David and also our new Limitless co-host, Josh. So as you're about to hear in this episode, the intelligence curse is not just an AI topic, it's a crypto topic. This is about AI people discovering that without decentralization, the future of AI is kind of bleak. These aren't just technologies, these are political systems. Now, moving forward, episodes like this will be on the Limitless feed, so you won't see them here anymore. So what you need to do is stop what you're doing and go subscribe to Limitless.

• 0:47Ryan Sean Adams?

  Apple, Spotify, YouTube, wherever you get podcasts, there's a link in the show notes for all of these. Limitless explores the frontier of AI, the same way Bankless explores the frontier of crypto. We believe these are the two frontiers for wealth generation and freedom this decade. Bankless and Limitless, AI and crypto, this is how you level up on both. Enjoy the episode.

• 1:10David Hoffman?

  In the Wild West of DeFi, stability and innovation are everything, which is why you should check out Frax Finance. The protocol revolutionizing stablecoins, DeFi, and Rollouts. The core of Frax Finance is Frax USD, which is backed by BlackRock's institutional Bittl Fund. Frax designed Frax USD for besting class yields across DeFi, T-bills, and carry trade returns all in one. Just head to frax.com, then stake it to earn some of the best yields in DeFi. Want even more? Bridge your Frax USD over to the Fraxtill Layer 2 for the same yield plus Fraxtill points and explore Fraxtill's diverse Layer 2 ecosystem with protocols like Curve, Convex, and more, all rewarding early adopters. Frax isn't just a protocol, it's a digital nation powered by the FXX token and governed by its global community.

• 1:52David Hoffman?

  Acquire FXX through frax.com or your go-to Dex, stake it, and help shape Frax Nation's future. Ready to join the forefront of DeFi? Visit frax.com now to start earning with Frax USD and staked Frax USD. And for Bankless listeners, you can use frax.com/r/bankless when bridging to Fraxtill for exclusive Confusive Fraxel perks and boosted rewards. Uniswap is your gateway to a more efficient DeFi experience. With Uniswap swapping and bridging across 13 chains is simple, fast, and cost-effective, helping you move value wherever, whenever. Thanks to deep liquidity on the Uniswap protocol, you'll enjoy a minimal price impact on every trade. And now, Uniswap V4 takes it even further. Swappers benefit from gas savings on multi-hop swaps and ETH trading pairs, while liquidity providers can create new pools at 99% lower costs.

• 2:37David Hoffman?

  The best part: you don't have to do anything extra. Each trade is automatically routed through Uniswap X, V2, V3, and V4, so you get the most efficient swap without even thinking about it. Whether you're swapping, sending, on-ramping, off-ramping, or bridging, Uniswap's web app and wallet gives you the tools to unlock DeFi's full potential on Ethereum, base, Arbitrum, Unichain, and more. Use Uniswap's web app and wallet for a more efficient way to use DeFi. Imagine verifying yourself without handing over personal data. No hacked databases, no unnecessary personal exposure for airdrops, and no AI bots ruining community governance. Meet SELF, the on-chain identity verification protocol built for privacy and control. SELF protocol uses zero-knowledge proofs to confirm your identity safely. Users prove key details like age or citizenship

• 3:20David Hoffman?

  without revealing sensitive personal information. SELF never stores your data, it only generates cryptographic proofs. Here's how it works in three steps. First, register and verify. Use the SELF app to scan your biometric passport's RFID chip. SELF verifies authenticity with zero-knowledge proofs. Each passport creates one unique identity. Second, you can share proofs privately. Third-party apps request identity proofs, like confirming you're over 18. You can also link proofs securely to public wallets for airdrops or governance participation. And then last, secure verification. Apps validate your proofs instantly on-chain, like on Cello, or off-chain. Audited by ZK Security, the SELF app is live on iOS and Play Store. Visit self.xyz and follow SELF Protocol on X.

• 4:03Speaker 02

  So the intelligence curse, pretty briefly defined, is the set of incentives that we might get when we unlock artificial general intelligence. And we're reusing the OpenAI definition here, which is the ability to automate most or all human labor. And in that world, we're really concerned that government- Powerful actors, corporations, won't have that incentive to care about regular people. That doesn't mean they guarantee you won't, but it means that strong economic incentive that we've had for basically all of social, all of human history, where powerful actors have needed regular people, and so there's an exchange of goods and benefits. If you sever that, you're relying a whole lot on goodwill. And we think that's not nearly as stable or strong of an arrangement, and it mirrors the kind of patterns you see in economies like those that are afflicted with the resource curse.

• 4:46Ryan Sean Adams?

  Are regular people just everybody? Just like, you know, you, me, white collar workers, you know, people in developing countries, people in very developed countries? Like, who are the, who are the regular people to which you speak of?

• 4:58Speaker 03

  Yeah, so I think we really do mean here everyone, in particular, maybe like everyone who does not have capital. That is, they're like very dependent on AI, that it can be physical capital or financial capital. And I think there's, people often talk about white collar workers. There's also people in developing countries, of course, who we shouldn't ignore their existence. So it really does mean everyone, where like right now the state is that everyone can contribute economically. States and companies have an incentive to care about everyone, but then if all of everyone's labor is replaced, then this is less true.

• 5:25Speaker 04

  So I'm really curious to ask you guys, why now? What did you see that sparked the interest to actually make this post? Because we read through it and it was very, very thoughtful, very pragmatic. But what was it that sparked this now versus a year ago or a year in the future? Was this, was this the exact right time? Or do you think this thesis kind of changes as we progress over the next few years?

• 5:43Speaker 02

  So I think for us, just for some history of how we got here. I think shortly after O three was announced and it looked like, oh, timelines are getting pretty short. AGI doesn't look, you know, 20 years away or 50 years away. It might look five years away or even sooner than that. I think we were having this series of conversations. We worked at the same building at the time, or the same office building. And I'd had this observation about the resource curse, and oh, this looks somewhat similar to those patterns. And Rudolph had separately been working on this draft of an essay, which is now in the full series called Capital AGI Human Ambition. We published earlier versions of those essays back in January that were, didn't propose any sort of like solution or way to try to, you know, stop this problem. And we spent the next three or four months of banging our head against the wall, trying to figure out, well, what can we do about this?

• 6:28Speaker 02

  What does it look like to actually solve this problem? A whole lot of the related essays in the piece that aren't squarely focused on the solution were essays that we were kind of writing and taking notes on as we got what we thought was closer and closer. And in the last couple of weeks, I think we just thought we had enough. And so we went ahead and hit the publish button. We are big fans of shipping in public.

• 6:47Speaker 03

  Yeah, I think also the mention of the timelines, I think there's also something about like when when when EGI is far away, it feels very much like a technical problem. And for a long time, a lot of the people who take AGI seriously have been thinking about it purely for the technical and thinking about the systems. And I think just as it draws nearer, you start realizing that, okay, it's not just going to be a technical thing. This is actually going to interact with the rest of society, with the real world. It's not like very concrete effects in that sense. Maybe we should actually think about those.

• 7:11Speaker 04

  It seems important. And it's funny, because at Limitless, we are very, we're very optimistic about the future. We're very excited about the pro tech, pro AI future. And then Ryan surfaced this, this document with me, your, your post. And, and I was like, I was reading through it and I was, it kind of hurt a little bit. I was like, this isn't the future that I'm super excited about. But it was very thoughtful and very pragmatic. And I wanted to ask you guys, why, why did you define this as the intelligent curse? Why is intelligence not a blessing?

• 7:36Speaker 02

  So I think the name just strictly comes from the comparison to the resource curse. So I don't think we, we don't root the entire analysis in the resource curse. It's a very helpful example to really conceptualize what is a similar environment that has similar incentives, what are the outcomes there. But the initial observation was this looks a whole lot like the resource curse in development economics. And so that's kind of where the name came from. There's this huge wealth of existing literature and debates. We didn't want to center the entire argument on that. But we did think that name seemed quite relevant here. And I think in particular, why it might be a curse as opposed to a blessing, it really depends on how it gets deployed, if it's a centralizing or decentralizing technology, if it accumulates power in the hands of the few, or distributes power out to many, many people. And so I think it can be a curse. And we set out the scenario for which it could be a curse, but we also offer the world in which it could be a blessing.

• 8:22Ryan Sean Adams?

  There's so much really to unpack here. And some of our history at Limitless is comes from also crypto, right? And I know the series of essays, you referenced Vitalik Biedren's work on decentralized or defensive accelerationism. And we've had him on the podcast actually talk about that. And that might be indeed one of the ways out. But when you start to talk about decentralization, that very much does seem like maybe our main defense against the centralizing effect of this. But I don't want to project this too far forward in the solution. And there's so much to unpack here, so much to go through. We'll do it kind of sequentially by, by route. But one thing I do want to get, since we've we've defined it, we haven't defined it yet, but we've we've dropped this phrase, the resource curse, several times so far.

• 9:07Ryan Sean Adams?

  Luke, could you just define what the resource curse actually is? I believe this pertains to countries and how endowed they are with maybe natural resources. Tell us about the the resource curse and why it's basically a meme for the name of this essay.

• 9:22Speaker 02

  Yeah, so I'll stress first that it's not the the sole piece of evidence we rest on. It's very much show an example or an analogy that we want to build around. But the resource curse, succinctly put, is the tendency for countries that have lots of natural resources to oftentimes, instead of having very rich or wealthy citizens, to have actually worse conditions. And there are a lot of different explanatory mechanisms for why that can happen. But one of them that I think is pretty prominent in the literature is that if you have oil in the ground, and all it takes for your state to get really wealthy is to get oil out of the ground, onto the roads and onto the ports, where your incentives are not to build this really complex economy, your incentives are to make as much money out of oil as you can. It doesn't require a whole lot of people to make money off of oil. It might require workers to actually extract the resource, to get it out to the ports and to sell it.

• 10:06Speaker 02

  But that's a whole lot less people involved in an economy than, let's say, a more developed, advanced economy like the United States, where there's lots of moving parts here. Now, there's a lot of different ways the resource curse ends up, but for a whole lot of countries, particularly those that don't have really strong institutions, the resource curse ends up in pretty terrible poverty. There are ways out that we think, we kind of talk later for the piece, as to, you know, what ways are out, what ways are potential, like analogies here that we could be looking to for solutions. But the core thing here is that you either want a diversified economy, or you want institutions that can withstand the curse.

• 10:37Ryan Sean Adams?

  Okay, and examples of that are countries in the Middle East, maybe that are oil-rich, and they really haven't developed their, I guess, civil liberties, or kind of the the labor economies of their citizens, right? Or maybe a country like Russia, which is kind of in the grip of authoritarian totalitarian powers, and it's, you know, they kind of devolved into a plutocracy. I suppose that's what you mean by the the resource curse. Now, we also have a counterexamples, maybe, like Norway is very well endowed. There's a lot of energy there. Canada might be another example. I mean, they seem to be doing fairly well with liberal democracy.

• 11:14Ryan Sean Adams?

  But the counterintuitive thing here, and why you're labeling this the intelligence curse is, you would think that more resources equals like better. More resources equals better for everybody. And it turns out that's actually not the case for resources, for nation states when it comes to natural resources. Sometimes more resources actually lead to an instead of structure that makes things worse for the population. And that could be the same with the intelligence curse.

• 11:43Speaker 02

  Yeah, that's what we're saying. And we also think that there's a lot of signs of hope there. We talk a lot about Norway. We talk a bit about Oman as well. It's two examples of states that broke the curse, what we can learn from those. But yeah, I mean, states like the Democratic Republic of Congo, for example, or Nigeria, that are just like really, they have tons of resources and yet their people are very poor. And the question is, well, what are the incentives that are creating this outcome?

• 12:04Ryan Sean Adams?

  Okay, so let's, now that we've got kind of the gist of it, let's flesh out this argument in a lot more detail. And you have basically a series of essays with different sections on this. But when I was kind of looking at the high level thesis, it feels like you're playing with a few premises, like maybe three in my mind. Like one is that AGI is the only game worth playing. There's a famous essay titled this as well. But basically, AGI accrues incredible capability and power. And as you said, this could be like on the near term horizon. We're talking about years, maybe five years, for instance. So that's like the first premise you sort of have to believe. The second is that AI will replace humans for valuable economic labor. And we're going to flesh that out in a second.

• 12:45Ryan Sean Adams?

  And as a result of that second premise, the third kind of, I guess, idea here is that powerful actors, these would be like nation states and companies. They no longer have an incentive to care about the regular people, as you said. Why? Because the regular people used to be their economic engine and their labor. But now with AGI, the regular people aren't providing utility. So do we need these welfare states? Do we need these social structures? Do we need civil liberties? Okay, so that's the base idea we're going to flesh out. And it begins here, which is this concept. Of pyramid replacement. I want you to sharpen this mental model. So AIs, this idea that AIs will replace humans for all valuable labor.

• 13:27Ryan Sean Adams?

  And I'm showing on the screen a picture of a corporation, I think. This is a typical company. You know, companies are arranged in hierarchies. At the base of the pyramid, you have your entry level employees. In the very top, you have the executives, you have the C suite. So can you describe what this pyramid actually is in the typical corporation? And what you see AIs doing to this pyramid?

• 13:50Speaker 03

  Yeah, so basically, so there is currently this hierarchical structure in companies. And it's actually not from first principles obvious which end of the pyramid AI will start automating first. But like empirically, it seems like AIs are getting good at tasks that have short time horizons, where the task is completed quickly and then you move on to the next thing pretty quickly. And getting better at longer time horizon tasks slower. And there's also the social fact that the C suite is less likely to unemploy themselves than to unemploy other people. And like these people who are supposed to unemployed are the entry level employees, because you don't need to fire anyone, you just stop hiring. And that's why we think the like first step in automation, something that might already be happening in software companies, is that the entry level employees, instead of hiring more and more than the company, you just instead of giving the senior developers at a software company,

• 14:35Speaker 03

  an entry level intern or something, you just give that senior developer cursor. And they code with cursor with some other AI code and tool. And they don't need the entry level employees anymore.

• 14:44Speaker 04

  So Ryan, if you don't mind scrolling down just a little bit, what I loved about this, this section was kind of the visual that you guys created, which, which showed this pyramid and the pyramid is blue and that means it's all humans. And then as AI starts to roll out, it starts to absorb the entry level employees. And then as it goes to junior and as it goes to middle management, it slowly absorbs the bottom layers until eventually we're just left with the C suite on top and then nothing. And then everything gets absorbed to AI.

• 15:08Ryan Sean Adams?

  So it's just literally like one big AI like red block, right? The pyramid becomes just like this AI Borg machine or something.

• 15:16Speaker 02

  It's so long pyramid, it's a square.

• 15:18Ryan Sean Adams?

  Yeah, it's a square.

• 15:20Speaker 02

  I like that. I might steal that. But this is a, this is one of the things we changed in the original essay. I think I still have like the rough draft published on my blog. But originally, it was just the pyramid kept getting smaller and nothing was replacing it. And you get to this last slide and it's just blank. Uh, and we had like an outline of a square for like, that looked like it was just part of the picture the whole time. And then at this point, it's just like, there's nothing. I think I wrote the, the org chart goes blank. And Rudolph had the idea of, maybe we should just like show people an entire automated company. Actually show, it's not just that the people are going away, but that AI is rapidly filling those functions. And so now you got the visual in front of you.

• 15:51Speaker 03

  And there's also something here where like, we don't want to imply that when the AI takes over, you know, in the future company, for every human employee, there will now be one AI agent that does, you know, like mashes one to one with each original human employee. I think the optimal way to structure AI is in companies where we look a bit different from the like current thing where you stack humans into a pyramid. And therefore the AI, you know, we represent with this like the square thing, it's sort of like, it's like blob of AI computes around your sort of like shrinking amount of humans that are like providing our direction.

• 16:19Speaker 04

  And this was, this was what I was curious to ask now is because currently in the world of AI, I feel like I am a leveraged human when I use it, where I am, I'm capable of X and then because of AI, I'm capable of Y. And I guess the question to you is, is will humans not just get better jobs? I guess if you could imagine stacking the pyramid on top of the AI, where now we have this foundation that provides a lot of leverage for entry level employees all the way up to CEOs. But the productive output that's unlocked as a result of that leverage creates new and interesting problems for them to tackle. So would that not be the case where we become hyper leveraged humans while, while removing some of the workforce, but not all of it? Yeah. Why can't this be a box with like a pointy hat, you know, pointy hat on top?

• 16:59Speaker 02

  Well, for what it's worth, if you go one up, you'll actually find that box with a nice pointy hat on top.

• 17:06Speaker 02

  I guess the real question is, the way that we currently structure are like major white collar companies. These like big mega conglomerates. It's a company of like a couple hundred thousand people, like 10,000 people. And if you look at one, the success stories and two, the existing statement CEOs are making. On the success story side, the cursor has demonstrated that you can be a multi-billion dollar company, getting tons of money with only a couple of people there. And if the general advice is you should only hire As many people as you need to run the organization. I'm not sure why Cursor would then hire 50,000 additional people. I'm not sure if that would actually buy them additional runway right now. But I think maybe what's more important anecdotally than Rudolph, I head off to the more systematic argument, but I think Duolingo CEO has now said they're an AI-first company and this means they're going to ask in every role they hire, every contract position that they have,

• 17:49Speaker 02

  whether or not they can automate this first. I think we have at the bottom some links to other companies who've also made similar statements here. I can't recall off the top of my head all of them, but I mean, the general ethos that we're hearing right now, as this is kicking off, is what we really want to be doing here is being more efficient, being leaner. Rudolph, I head off to you for the more systematic argument.

• 18:06Speaker 03

  Yeah, and I guess I think there is some hope that humans currently have this advantage in long horizon tasks. I think basically we know how to train AIs to do tasks where there's a large data set, or where we can build a digital environment, reinforcement training environment, where the correct behavior is rewarded. And this works for things like writing where there's a lot of data on it and you just train to write like the average and can person pretty well. And it works for things like master code where it's easy to verify whether something is correct. But then it's harder to train AIs to like be the CEO because the CEO interacts with the real world. They take a lot of actions. This is just like, we're currently less good at getting AIs to be good at this stuff. So I think the state where like the AI uplifts the humans, this will continue for a while and probably longer than some of the most aggressive AI projections estimates.

• 18:48Speaker 03

  And I think there is hope that we can extend this period during which humans are mostly just uplifted by the AI. And this will be like very good for just human agency and like ability of humans to affect change in the world. I think right now we are definitely in this regime. But then in the limits, there's no theoretical reason why the AI can just also get a bit at the long-term planning. That's what the AGL is trying to crack right now. And at some point, the like board will come in and the board will be like, look, you're the CEO, you have a nice job. But like, I'm sorry, but it looks like GPT-9 is, you know, starting to get better at making decisions than you are. And we're responsible to shareholders and I'm very sorry, you've done a good job, but now we're gonna have to lay you off.

• 19:22Ryan Sean Adams?

  Can we start there? Can we start at the top? That would be kind of nice for a change.

• 19:25Speaker 02

  You know, I've gotten a couple of those reactions. And I think, I think the way that we're most likely to be wrong on this model, and I want to be, you know, as like episcentically rigorous as I can be, is that the middle gets cut first. It could be the case that there are entry-level roles that we just really need. A whole lot of people to be doing the like the basic work and management becomes dramatically easier. I think the evidence really points towards the former, that we're getting this bottom-up pattern of automation as opposed to this middle out right now. And I think the reason for that is it's just quite simple. If it costs you $50,000 to hire a person to do something every year, but it would cost you $10,000 in compute to do the exact same task, it's really hard to justify the additional $40,000. And sure, like Mike's a great analyst and you go golfing with him on the weekend, but that's $40,000.

• 20:08Speaker 02

  And you can go golfing with Mike whether or not you work with him. And so I think a lot of companies, whether it's a downturn or whether they just want to save some money, they're faced with that question.

• 20:17Ryan Sean Adams?

  Luke, that was a spoken, like, a member of the C-suite. Let me tell you, that is...

• 20:22Speaker 02

  I actually can't play golf.

• 20:23Ryan Sean Adams?

  Sorry, Mike.

• 20:24Speaker 02

  I really can't play golf.

• 20:25Ryan Sean Adams?

  We can golf, but you can no longer work here. So you guys are kind of, I guess, you see it maybe emerging right now as sort of bottom up, where, you know, entry-level programmers are the first to go, or like support teams, customer support, something like that, kind of the first to go. And it works its way upward. But you're also agnostic in this model as to whether it's kind of middle out or even top down, or whether it's bottom up. Correct me if I'm wrong, but I believe this is particular to white-collar jobs, yes? So is that part of the thesis? That you kind of, the information, like, knowledge worker class is kind of going to be the first to go because our robotics technology hasn't quite caught up to our software LLMs?

• 21:05Speaker 03

  Yeah, I think that's the default. I think right now at least, LLMs are advancing faster than robotics. And this creates the interesting possibility, I think Carl Schulman talks about this idea that we might have a period where humans are valuable not for their brains, but for their hands. And maybe we get this, we have like a robotics...

• 21:20Ryan Sean Adams?

  That sounds worse.

• 21:22Speaker 03

  So, I want to make this concrete. Imagine your job is just you're like assembling widgets in a factory, but you have a year piece where the AI is giving you instructions and it gives you like motivational posts from time to time to keep you on task or something. And then, but you don't actually have to do any thinking because the AIs are better at all thinking. Like, maybe this is the future. However, don't worry. Maybe we also fix the robotics and we get robotics quickly and then you can't do the widgets either. You're just like fully unemployed. So there are many possibilities here.

• 21:47Ryan Sean Adams?

  Okay, so that's the concept of pyramid replacement. And let's do some pushback, though. Some objections to this. So, one is kind of the, I guess this is maybe the, at least I'm familiar with, the Tyler Cowen kind of pushback argument, where he's basically like... You know, there's diffusion barriers. And we've certainly seen this like, you know, kind of coming from crypto. So like crypto could replace the entire world's money system. But guess what? There's actually regulators who kind of don't want that to happen. Right? There's institutions, there's structures, there's all sorts of breaks in society, meatspace, government that just slow things down. It's kind of the the human piece of it. And so you might have this technology in a box, in a Geniuses in a data center, whatever.

• 22:30Ryan Sean Adams?

  But they might not diffuse through society because society has all of these breaks. And, you know, like big breaks in in meatspace. And so will that kind of slow this down? I mean, it feels like there's, we can adapt better. I mean, just the general idea is we can adapt better if this happens very slowly versus if this happens like in a period of like months to years. And what do you think about that diffusion argument?

• 22:53Speaker 02

  So one, I think a lot of our like solutions focus on bringing the breaking the intelligence curse section is really at its core, an argument to try to extend the augmentation window so that we get more time to adapt. And I think if you look at the way the pyramid replacement flows right now, we argue it happens pretty slowly. It's a it's a bottom up approach. I don't think we give an exact time horizon because it's really hard to predict. But I mean, I think if AGI hits in 2027, I think most people are still employed in 2028. The question really for you is how fast after that moment? And there are a couple of reasons why you should expect companies would want to speed up pretty quickly. Maybe, for example, they don't do the automation, but a competitor does, and they start moving faster. So now there's a competitive pressure to automate. And the same way that maybe a state doesn't want to acquire certain weapons capability. But of course, the other state has also acquired the capability.

• 23:36Speaker 02

  And now you're in a race to to kind of get to the top here. Maybe it's the case there's an economic downturn and there's forces like cost cutting everywhere. And you do the layoffs and discover that you actually are like at equal productivity or maybe even faster when you try to automate that away. So there are a lot of diffusion barriers. We do not think that this six months after AGI, everyone's unemployed. But it's also important to note that diffusion barriers also have acceleratory pressures that are pushing against them. If you have this kind of technology and there's strong reasons to adopt, if investors are hyping it up, if people are seeing it work in the real world, it really is only a matter of time before critical mass starts to emerge and the way that we work is fundamentally changed.

• 24:10Ryan Sean Adams?

  Okay, and there's been other points to talk about, you know, sort of AI being very jagged now, right, where, you know, like it, some things you look at its output and you're like, oh, my God, you are so dumb, like, I could do this. And other things, you're like, wow, this is incredible. And so this could happen in a jagged way, I guess. I feel like we've established then that there is the possibility, if we get this kind of like acceleration towards some sort of an AGI, that AIs have the capability to replace the corporate human pyramid. And I guess this is the capability to, I mean, corporations, companies are the economic engine of like, basically, all societies, right? So that's effectively what we're doing is we're replacing the economic engine of these societies. Let's, let's move to kind

• 24:52Ryan Sean Adams?

  of the, the second essay and the second piece of this intelligence curse, where we start to talk about capital. All right. So now we've got a world where AIs have maybe started to erode, replace the human labor pyramids, our corporations, they're doing the work. And so I think you're making an argument here that the power, which was in the hands of labor, of course, capital always has power, but we have a large portion of labor in society, because humans are valuable, that has power, that would begin to shift. And this is almost like a startling revelation that the AIs might make non-human factors of production more important than the human ones, and particularly capital. Can you, can you develop some intuition for that for us?

• 25:37Speaker 03

  Yeah. So first, I think it's worth clarifying that capital, when economists talk about it, often means, you know, like, it means money, it means, but it also means stuff like physical factories. And therefore, like economists talk about factors of production, like land, labor, capital management, and like capital here is a bucket that includes like factories, GPUs, and also just like cash on hand. And I think-

• 25:56Ryan Sean Adams?

  Does it include, like, energy too, Rudolph?

• 25:58Speaker 03

  Yeah, I would, I was, I think economists would call energy a type of capital, because they say like, non-human factor of production is not land, which works a bit differently, and it's not management, which is, for our purposes, kind of a bit like labor, because both involve humans, kind of like. So, yeah. And then it's basically, the connotation here is just the point about, like, right now, the economy needs a huge amount of human input, and you add more human input on the margin- And the economy goes, you know, up and therefore the marginal unit of human labor is like compensated pretty highly. And like at least compared to historical like pre-incident here. And I think you can see this like historically, so like before the industrial revolution, where the human factors like human capital and sort of like education skills, stuff like this was less important. Because there was less like real technology, less like complicated processes.

• 26:41Speaker 03

  Then also the like amount of power that human labor had was lower. And then, yeah, so this is sort of like general arguments. And then in this essay in particular, we talked a lot about just this like point that you can just start substituting capital for labor more effectively than you can right now. And it's actually like, I think right now, for instance, if you're a like if you're trying to hire talented people, that's actually like a big bottleneck on your ability to like convert money into results in the real world. And then like this will go away if you can just like buy, use money to buy credits from OpenAI to spend on tokens that is like replaces the talent. And it's like, it's sort of like, right now, there is like a lot of complexity and like free friction to converting money into real world results. But this will go down a lot once the real world results you acquire this by just spending money on the AIs.

• 27:24Speaker 04

  Tokens become your workforce essentially.

• 27:27Speaker 03

  Yeah.

• 27:28Speaker 04

  This was an important element that I didn't really realize until after reading this is that there is this this difference in capital between general capital and and human capital, the actual labor workforce. And tokenizing the labor workforce seems a little scary. So I'm curious to get your takes on kind of the the system, the way this kind of rolls out over time in either maybe best case to worst case scenarios is what happens as humans get replaced by tokens and as as we kind of reduce our workforce incrementally. Does that happen quickly? Does that happen slowly? And what are like the second order effects downstream of that?

• 28:02Speaker 03

  Yeah, so let me say a bit about the second order effects here. So I think one of these is just that. So the thing I already mentioned that, for instance, if you have a bunch of money, but you want results in the real world, you're still bottlenecked on you've identified talent, you need to hire talent. There's a lot of friction here. Another is that a lot of social mobility today is based on like you are a talented human and you don't have capital, but you can go out in the world and do something. And people with capital have to pay attention to you. You're like nimble startup founder or something. And as you like VCs, you have a lot of capital, just, you know, need you, they think they will give you money, stuff like this.

• 28:35Ryan Sean Adams?

  I mean, another word for that is the American dream, right?

• 28:37Speaker 02

  Yeah, it's what we're all told, right? If you, I, you know, I live in London now, but I grew up in the States and, uh, you know, we're all told from a very young age, you know, if you work really hard, if you do well in school, if you go to the right college, you will have the, a shot at the American dream. And the American dream, it looks like accruing enough capital to be able to own things, or accruing enough capital to be able to make it and have a nice life and fundamentally change your social position. And I think a lot of the argument here is that provided that capital can be totally substituted for labor here, because you can just sub an AI, your ability to walk that way up the social hierarchy just gets a whole lot harder and maybe gets eliminated.

• 29:12Speaker 03

  And then like as a society, I think a lot of social progress and like change depends on this thing that someone who is not currently incentivized to care about the current status quo comes from the outside and shifts things. And then if you lose this ability to have social mobility, it's not as bad for individuals as makes society more static as a whole.

• 29:29Ryan Sean Adams?

  Okay, so there's this idea if capital becomes a substitute, like a general substitute for labor, which you can imagine if pyramid replacement is true. Basically, what pyramid replacement means is like, instead of paying the humans a labor force, I can just like pay the open API, like AI APIs and do this through tokens and pay the geniuses in the data center. And that's my labor's force. And so I can just take my capital, which is like my assets, right? My money. And instead of, you know, like putting my money into the, the slot machine of, of human labor, I just put that into the slot machine of AI. And what you're saying is this kind of like destroys social mobility?

• 30:14Ryan Sean Adams?

  You talk about, Joshua's just asking about like, kind of, you know, the, the best to worst case scenario. And I think maybe the, the, these essays are like really focusing on the curse side of things and maybe less on the blessing side. So one could imagine some blessing, but you talk about like, I mean, I guess one of the worst case scenarios, but in this, in a way, this is maybe one of the, the better worst case scenarios is this permanent caste system, where we're all kind of like locked into the capital ledger. That we're born into. And so like, maybe if you're born into a nation that has really like, I guess, embraced AI and like your, I don't know, like your father worked at OpenAI or was in

• 30:54Ryan Sean Adams?

  the industry was early, right? And like, really was hooked up to this spigot of capital. Like, that's your caste. You're kind of locked in. It almost sounds sort of futile in that way. I mean, not having lived in sort of a strict caste society and certainly embracing the idea of meritocracy, like, maybe that all fades away is what you're saying. We're like permanently casted into these kind of capital ledgers.

• 31:18Speaker 03

  Yeah, and I think it's worth noting that social mobility before the Industrial Revolution was very low. And I think social mobility depends on this thing of like, human talent matters, and also the economy is growing and stuff like this. And then, you know, before the Industrial Revolution, if you were rich, probably at some point in the past, your ancestors did something cool and the king gave them a bunch of land and made them aristocrats or something. But then, like, then you get the Industrial Revolution, human talent really matters. It's like, social mobility is possible by going out and inventing things, pushing science, pushing industry, stuff like this. But then if, you know, maybe we'll keep having technological progress, maybe the like, amount of abundance in society will go up. But even then, you've like, lost this element of new people can enter the elite, if AI is a substitute for elite human capital.

• 31:59Ryan Sean Adams?

  Okay, but that's, that's the thing that's counterintuitive, or like, what I'm wondering about the argument. So we got the Industrial Revolution, which is sort of machines replacing some human physical labor. And you're saying that was actually good for the humans, basically. Why does that not follow if we get an intelligence revolution that is not just like, good for the humans?

• 32:19Speaker 03

  Well, I think the most important differentiating factor for humans as a species is our brains, and that when freed up from physical labor, we're like, we're better than some animals at physical tasks, we're worse than other animals at physical tasks. And having the thumb is a pretty great advantage at using tools. But at the end of the day, it seems like the single best advantage that people have is that they can think up new things and execute on them. And so, post-industrial societies get these really complex information economies that spend a whole lot of time, both, I mean, producing lots of physical abundance in the real world, and with those resources, using our brains and our heads to come up with even more abundance and more ideas.

• 32:50Speaker 02

  And you can see this not just in like existing human, like existing economies versus old economies. You can see this today between economies and those kind of more resource curse afflicted states. Social mobility is lower because non-human factors of production mean that your ability to have some huge idea and make an outsized impact is also quite limited. Capital begets capital and your ability, and this is true in every society, but your ability to have outlier talent succeed is less if you don't need outlier talent in the first place to make money.

• 33:17Speaker 03

  And maybe also to add on this like a quick econ thing. It's like if what matters is like whether it's a substitute or complement for human labor in a sense of like what like the thing that sets wages basically like when you add one additional marginal units of labor, like how much other returns? And if you have like, you know, pre-industrial revolution, additional unit of labor, you have another additional peasant farmer, it's not very much. First industrial revolution, you have an additional unit of labor, but then they they command a lot of machines, they command a lot of capital, they actually have like boost the economy a lot, they get high wages. But then if like all the labor is done by AIs, you've got this like total like substitution of humans, then additional unit of labor output does not change, human wages are very low.

• 33:53Ryan Sean Adams?

  In this model, who are you saying owns the capital, right? So your capital is basically sort of a, it's property rights, you're, somebody's got to own it. In this model, do the humans still own it? Does it kind of consolidate to the the tech companies? Or do the AIs own it? Like, how sci-fi are you getting in this?

• 34:10Speaker 02

  So we we call for a ban of AI ownership of property and being CEO. So we're willing to, we do call for it. We do, we do. But I I don't know how likely that scenario is, but I don't want to preclude it. But I do, we went ahead and said, well, that's a pretty, it's a pretty cheap ban to do, right? It's not that hard to ban it right now. Maybe it's way harder in the future when we've already delegated lots of authority. I think existing law today, at least in most countries, probably does prefer this outcome anyways. But it's worth getting that explicit. But even if it's people, it really matters on how many there are. I think most people in in modern economies don't own a whole lot. That doesn't mean that's necessarily a bad thing today, because of course your labor is a very powerful thing to trade. And in many cases, you might own way more than you did in in previous societies. But it's not the same

• 34:53Speaker 02

  as being able to own like, you know, the kind of capital you might need to command many, many, many AI agents that are replacing lots and lots of labor. And I think, Rudolph, you've thought a lot about the kind of ways this create a more static dynamic where, as you mentioned earlier, like this could lock people into the existing positions pre-intelligence explosion.

• 35:08Ryan Sean Adams?

  Okay, so this idea that that capital can now buy labor, so human labor is no longer necessary, there's sort of another implication here, which is that classical liberalism starts to fall apart. And so, you know, post feudal societies, I think we've just like in post enlightenment, we have generally experienced, not in all places, not in all countries, not in all regions, of course, but we've generally, it's generally led to better human outcomes, right? You know, quality of life, life expectancy in general, wealth, freedoms, the whole concept of just like humanism. We've ended terrible practices for humans like chattel slavery,

• 35:49Ryan Sean Adams?

  at least in like most places. So it's, we kind of pat ourselves on the back and we think like, oh, wow, we've really advanced. We've just like gotten some better moral software and we've kind of like clearly evolved. I think that what you're arguing, though, is that there's a just like a more utilitarian perspective on this, which is like, maybe you could sharpen this argument for me, but it's like, nation states gave labor these rights, citizens these rights, because they were so damn useful. And it was just, they gave the citizens these rights because they needed to attract the labor pools and the brains to develop their economies. And if humans become less necessary for, say, nation

• 36:34Ryan Sean Adams?

  states, we've already demonstrated how they may be less necessary for corporations, then that entire, I guess, social construct starts to fade out. Can you sharpen that intuition for us?

• 36:46Speaker 03

  I think it's definitely true that there's a lot of institutional inertia in the sense of, right now, if you live in a society that really values humans and cares about humans and like politically might be willing to like introduce UBI or stuff like this, or like universal basic compute or whatever, then, you know, there's a strong chance that like this society has a lot of inertia in this direction. But then, like societies don't exist in a vacuum, they compete with each other. There's a sense which like, for instance, a lot of all the countries in Europe, like Britain was doing the most to sort of like be compatible with industrialization. And therefore, as a result, you know, they were doing a lot, they were quite political advanced for their time. They had quite a lot of freedoms. They were like good at encouraging industry, stuff like this. And as a result, like Britain becomes the preeminent power. And there's a thing of like, you know, there's a lot of societies, a lot of countries in the world, they're in competition with each other. So it's not just sufficient that like, if one society makes

• 37:31Speaker 03

  this choice, it's sort of like, they can continue on their own. It also matters like, which strategy wins overall in the world.

• 37:37Speaker 02

  It's not clear to me if this dynamic is bottom up or top down. Is it the states gave this right, knowing it would attract better competition, or that workers or people who own capital had more power than a state who were able to demand these? I think about like the Magna Carta in Britain, for example, the foundational document for modern, for the concept of like modern democracy, where landed gentry, people with lots of property, had powers that, you know, weren't necessarily as powerful as the king, but, you know, were in many cases like the fact, they controlled the factories of production that created wealth for that king. And so this put them in a position where they can make a whole lot of demands upon a king. You see the evolution of British democracies, that it first starts with this like landed gentry class. And, you know, here in America, voting rights begin, you know, the idea of like a self determined government doesn't start with everyone being

• 38:22Speaker 02

  involved. It starts with these like diffused property owning men, who because of that position, had some sort of diffuse self relying power. There's this Charlie Munger quote that I think was at the top of the original intelligence curse on my blog, which is just show me the incentives and I'll show you the outcome. And I don't think it's the case that cultural evolution plays no role here. I think it's quite important. But it's also worth asking, like, what is the role that economic incentives play on cultural evolution? And how strong are those incentives? And so I think in the limit, these incentives are probably the dominating force here.

• 38:51Ryan Sean Adams?

  There's this quote, I think in this part of the essay, where you say this, "The classical liberals today can credibly claim that the arc of history really does bend towards freedom and plenty for all, not out of benevolence of the state, but because of the incentives of capitalism geopolitics. But after labor replacing AI, this will no longer be true." Wow. So potentially classical liberalism on the line here. Let's get to the next essay. So we've talked about capital and its importance, how that could be kind of the dominant feature of a post AGI type society. So

• 39:28Ryan Sean Adams?

  Let's draw some more implications for what that means for, I guess, the nation-states relationship with its citizens. So this is the heart of maybe the intelligence curse. This is where the curse starts to come down on us even stronger. And the summary is this: "With AGI, powerful actors will lose their incentive to invest in regular people, just as resource-rich states today neglect their citizens because their wealth comes from natural resources rather than taxing human labor." And this is the intelligence curse. Why do powerful actors like nation-states invest in their people today?

• 40:06Speaker 02

  Well, think about it. If I'm a government right now, and I want to make a lot of money, and I don't have -- and maybe I want to make money for a variety of reasons. Maybe I'm altruistic, and I want to provide better care for my citizens. Maybe I'm self-interested, and I want my state to do well. There are a host of reasons why you might want this, because money gets you power. But in order to get that right now, you can do a couple of things. One, you can try to find some sort of resource, but maybe you don't have it. Two, you can offer -- you can try to create the -- increase your return on an investment. And given that most economies right now really flow through people, developed economies, diverse economies flow through people, and their labor and their work, you can up that return by doing a couple of things. You can really increase the quality of education. You can build infrastructure like roads and public transportation, which helps get investment

• 40:51Speaker 02

  flowing into areas. You can build these really reliable governance systems to encourage investment. You can foster competitive markets. You can support small business formation. You can do all of these things that make it more likely that your population produces meaningful economic results, and then you can tax them heavier. I think right now in the United States, taxes on income are something like 50%, whereas taxes on corporations are something like 12. So a share of total tax revenue, 50% derives from income taxes, whereas 12% or 13% derives from corporate taxes for the government. And so in that world, of course, you want to make sure that people are making more money, because if they make more money, well, then you accrue more tax revenue, and you can do more things with that tax revenue. It just so happens that these investments are the kind of things that we associate

• 41:36Speaker 02

  with a better quality of life, and they also give you better bargaining power. But the other thing, of course, and we wouldn't get into this a bit later, is maybe you impact the ability of that government to maintain or retain power, maybe either through democratic means like voting, or through credible threats to an autocratic regime.

• 41:52Ryan Sean Adams?

  Okay.

• 41:52Speaker 02

  So you might want to buy those results.

• 41:54Ryan Sean Adams?

  Yeah, so you're basically you're saying like kind of the public goods, the public infrastructure, take for example, like just like free public education taught citizens. This is not the government doing kind of like quote unquote, the right thing for its citizens. This is not governments, nation states of the world, just being altruistic and kind of like nice and kind. They're doing it for a reason, for an economic outcome, which is if we have an educated population, then we have greater ability to produce GDP and like government benefits from being able to tax that GDP and that's how it makes like revenue itself. And like, this is basically the reason, the incentive for governments to invest in education, or for that matter, any kind of like good thing for its people.

• 42:38Ryan Sean Adams?

  Does that analysis miss something? It feels like, it feels very mechanistic and bleak. Maybe that is the way the world works, but like, you know, is there not like some sense of morality in like doing the right thing here?

• 42:50Speaker 03

  So I'll mention one thing here, which is that this analysis doesn't require anyone consciously to be the cynical in government. So for instance, you can imagine in the year 1500, like in Europe under the feudal limits, like maybe there were people who are really altruistic and you want to like, you know, give people free healthcare and stuff like this, but then they invaded by the neighboring country who like invested in one of their military and stuff like this. Whereas then today, it's like the people who were altruistic in like 1800, who were like classical liberals and like promoting like British and US like political progress. Like, these are the people in the future because their values at that point in time were associated with winning strategy.

• 43:24Speaker 02

  Well, ask yourself too, just if you hear, I'll tell you nothing about about these two politicians, but a politician who wants to increase spending on education and a politician who wants to decrease spending on education, gut check, which one feels more altruistic to you?

• 43:36Ryan Sean Adams?

  Increase, right? Yeah.

• 43:39Speaker 02

  And I think it just, it is the case here that the incentives are aligned here. The thing that sounds really good and is altruistic also is the thing that returns the money to the state. Now, in the world where the state has to cut that education spending because they're in a fiscal crisis, or because making that spending choice would be super irresponsible, maybe there are some sort of like, as Rudolph said, there's like an invasion coming, they've got out military spending. Doesn't make that other politician any less altruistic in theory. It just means that other politician has really strong countervailing incentives to do something else.

• 44:05Ryan Sean Adams?

  Okay, so it's not altruism, it's just a happy coincidence of incentive alignment. That's why we get beneficial things. It's interesting, going back to the idea of the resource curse, maybe we could talk a little bit more about that because we do have real world examples, experiments at play, where there are very resource-rich countries, to the extent that they can generate economic returns and taxes, revenue essentially, to pay for the government based on resources rather than human capital, rather than human labor. And this is the resource curse that we were referring to at the outset of this episode. And we actually, there's a name for these, which are rentier states, I believe. Okay, so tell us again about the rentier states.

• 44:47Ryan Sean Adams?

  Let's get into some more detail here, because this is essentially the experiment at play. Again, if your nation is resource-rich, how does it treat its citizens? What are the experiments that we've seen right now in the real world?

• 44:58Speaker 02

  So we can look at the raw incentives. There's a very interesting counterargument that I think we actually incorporate, which is that many states have such good institutions that they aren't exposed to these incentives that can avoid these incentives. But at the raw incentive level, I think we talk about like the Democratic Republic of Congo, for example, extremely complicated history, but with literally trillions of dollars in minerals below their feet and hundreds of billions of dollars in, I believe, total revenue from those minerals, and yet their people subsist on a couple of dollars a day. And if the state has the kind of resources that could enable relatively widespread wealth, the question is, why don't they do it? And oftentimes, in these states, you can imagine, I think it's very easy for us to imagine in the West, a state far away, that is exposed to high levels of corruption or exposed to high levels of inequality because the leaders choose themselves over their people.

• 45:43Speaker 02

  It just sounds quite foreign to us here in the West. But of course, this is what we see time and time again in many resource curse-inflicted states, is that the leaders get rich, or the people who control the mining rights get rich, the oligarchs get rich, but many regular people don't see the benefits of economic activity because there's no reason to invest directly in them to reap that reward.

• 46:02Ryan Sean Adams?

  Is there an explanation of counterexamples of resource-rich nations that are actually like protecting their citizens fairly well? So like Norway comes to mind, again, maybe like Canada is sort of comes to mind.

• 46:15Speaker 02

  UAE comes to mind as well.

• 46:17Ryan Sean Adams?

  Yeah, how do you explain that?

• 46:19Speaker 02

  So one, I think the key thing here is like how much of one, like when we talk about rentier states, we're talking about states that have like a very large portion of their revenue coming from natural resources. I think that's it's a dominating force in their economy. There's, I think we cited in the piece, but there's these two really interesting examples that the authors give of Norway and Oman. It's counter examples of states that don't fall into the resource curse. In Norway's case, by the time Norway discovers oil, they have this really efficient, really anti-corrupt, fantastic democracy where voting incentives kind of override raw capital incentives, where the bureaucracy understands how to do things in really complex ways. And this means that voters have like very real power and the benefits get dispersed in different ways. The voters can out the vote capital incentives. In states like Oman, Oman in particular had a pretty credible

• 47:02Speaker 02

  threat of some sort of uprising or revolution. I can't recall the exact example from the paper. I know we cited it directly. And as a result of this, the state capitulates. And I think the kind of analogy that I've used here before is that if you are the person that owns the rents, you'd really like to keep all the rents to yourself. You'd also like to keep your head meaningfully attached to your body. And so if capitulating on that, if, if, if, if paying out people, if doling out the rent money keeps your population in check, you're much more likely to want to, to do this because otherwise that credible threat could be quite real. Now, what we say is twofold. One, we think that advanced AI makes it way easier for the state to know everything and suppress things like revolution. And so we think the revolutionary arguments and autocracies is kind of off the table.

• 47:45Speaker 02

  We find ourselves in being the contrarian position of defending democracy pretty strongly here, which is I think become kind of contrarian to some tech circles in recent years, but we actually think it just is.

• 47:53Ryan Sean Adams?

  It's not contrarian around here. We're big defenders of decentralization democracy.

• 47:56Speaker 02

  I appreciate this. But we want, you know, we, in the Norway example, the question you have to ask is, do you think most states have the kind of like robust and rigorous democracies? Norway, one. And two, do you think the intelligence curse for placing all labor might be stronger? In our case, we think it probably could be stronger, but it really shows us an interesting way out.

• 48:15Ryan Sean Adams?

  That's interesting. So you're basically saying it's just like Norway had the democratic decentralized institutions strong enough to kind of withstand some of the pressures of the resource curse. But the open question is like, how many other countries have that? And when you get into the intelligence curse, no matter what your institutions, your democratic institutions are, will that be enough to withstand the tidal wave of this intelligence curse? And that's kind of an open question, but it does provide maybe a sliver of hope, which is that with robust institutions, maybe we can start to like, maybe we can withstand the coming tidal wave. I don't want to get into solutions too soon because we're still on the problem, but I'll just earmark that right now.

• 48:57David Hoffman?

  Imagine if your checking account and DeFi wallet finally spoke the same language. That's Mantle Banking, an all-in-one fiat and crypto account. It lets you save, spend, and invest all from one dashboard. Swipe for coffee, stake ME through Yield, or even use virtual cards for payments through Apple Pay. So it feels Web 2 simple, yet stays Web 3 sovereign. For allocators, meet Mantle Index 4, the S&P 500 of crypto. A tokenized, institutional-grade fund seeded with $400 million from the Mantle Treasury and balanced across Bitcoin, Ether, Sol, and Yield-enhanced stables. One asset, broad exposure, pure DeFi composability. The momentum is real. ME faults, FBTC bridges, and a $2.4 billion community treasury are all powering the next phase of on-chain finance. Mantle brings real-world access, yield, and utility to digital assets. Ready for the next era of

• 49:41David Hoffman?

  on-chain finance that actually belongs in 2025? Explore Mantle at mantle.xyz or follow mantleofficial. Mantle, bridging TradFi and DeFi so you don't have to. Have you ever imagined Bitcoin and Ethereum truly working as one, unlocking the full potential of Bitcoin, DeFi, and more? Meet Hemi, a groundbreaking modular network designed precisely for that vision. Co-founded by early Bitcoin core dev Jeff Garzik. Unlike other Layer 2s that treat Bitcoin and Ethereum as separate silos, Hemi connects these giants into a single, powerful super network. With Hemi, users gain unprecedented asset portability and possibilities, combining Bitcoin's security and value with Ethereum's versatility. Hemi's unique innovation, the Hemi virtual machine, integrates a fully indexed Bitcoin node directly into an EVM, enabling dApps that seamlessly

• 50:25David Hoffman?

  Interact with both networks and with Hemi's proof of proof or POP consensus, users benefit from truly decentralized censorship resistant Bitcoin level security. Since its recent mainnet launch, Hemi has rapidly ascended the ranks as one of the top Bitcoin chains. With a thriving global community and robust ecosystem support, Hemi isn't just building a network, it's shaping the future of Web 3, DeFi and beyond. Visit hemi.xyz slash banklist to learn more, discover ways to interact, participate in the leaderboard program and be part of the community that's uniting Bitcoin and Ethereum.

• 50:54Speaker 04

  Yeah, I think what I want to know as a layman listening to this, who is just on the street and now understanding that he will soon not be able to trade his labor for capital. I'm curious.

• 51:04Ryan Sean Adams?

  You're out of a job.

• 51:06Speaker 04

  So.

• 51:07Ryan Sean Adams?

  It's very last few podcasts.

• 51:10Speaker 04

  So I'm saying I'm listening to this, I'm like, hmm, okay, well, I can't do this, I can't do this. I am no longer able to trade my labor for capital. What does that look like for the average person? Are they collecting government welfare? Is there universal basic income? What, how does my life, how am I able to accrue capital if I am just one of those elementary workers in the workforce?

• 51:26Speaker 02

  Well, I guess the question is how much do you want to get into our solution section right now?

• 51:30Speaker 04

  So perhaps we'll hold that for a second because there is, there is another element that I was, I was also interested in talking about, which is just the, the human element. As a human, I like human interaction. I like going to hang out with friends. I like buying homemade things from people. I like meeting the artists that create the art that's on my walls. I really enjoy that connection. And when we introduce this, this AGI element, this, this artificial intelligence force, it feels very inhuman and artificial and it feels very sterile in that way. And when I think as someone who is experiencing the human experience, I'm really curious, what element does the human nature have on the way this all plays out?

• 52:05Speaker 03

  So I think it is definitely true that there's a lot of things where humans have a preference for interacting with humans. And I think this will continue and I think there will be a lot of like social facing jobs where the humans have a very high bar for replacing that role with a human, with an AI. And I think this does provide a sort of like buffer where I think there will be some jobs that last quite long. Either for like, maybe like a, like teacher or maybe if there are for like interfacing with customers, even if you're like a salesperson who's like already kind of on personal relationships, like humans might prefer that for quite a while. There's some things about like, how charismatic are the AIs? Like, how do they get at like, I'm hijacking human social instincts, stuff like this? But there's also a question around then like, uh, like, so the humans currently have money.

• 52:48Speaker 03

  And there's like some capital flowing around the human economy. But there will also be these like, the AIs will be increasingly doing stuff. And it might be like increasingly to like, the like money flows towards the AI part of the economy. And in particular, like, I think, like, how are the humans earning the money with which they pay each other for the human services? When at least some of that human money also has to be spent on like, doing the AI stuff that probably like, keeps them alive, keeps them fed, stuff like this.

• 53:10Speaker 04

  Is it all for the workforce or are there, like, how far does this go? So I guess if currently a human thing that I would really be really excited about is, is to teach my kids something or to be a father to my children. And how far does that, does that go? Does it get into the household? Does it kind of remove the need for humans through the entire process?

• 53:29Speaker 02

  So there's a really good paper that walks you some of the cultural elements here. It's called gradual disempowerment. We know the authors quite well, came around the same time that ours did. We focused less on this initial cultural element, most because we were trying to isolate what we think is like a really critical variable here on the economic side. But I'll tell you, I had an interesting interaction a couple of days ago with someone who was telling me that they talk to ChatGPT constantly. And that they think their dad talks to ChatGPT more than he talks to his kids, that it was like two or three hours a day, maybe more. And so I think the capacity for machines to alter our relationship with each other seems quite high. I don't have the exact quote that Mark Zuckerberg said in the podcast recently. So I hopefully am not butchering this too bad, but it was something along the lines of, you know, the average human has, you know, four or five friends.

• 54:14Speaker 02

  The average American is four or five friends, but they have capacity for 15. We can substitute a lot of that with machines. For me, I'm not excited about this vision. This is really not exciting to me at all. I really value the real world and the people that I get to interact with. And maybe this is something I don't want to impose and say that I get to make a choice, that nobody ever gets to go down that rabbit hole. But it's certainly not the technology that I'm excited about building.

• 54:35Ryan Sean Adams?

  Okay, that's really fascinating because when you start getting the family and you talk about just like being a parent or something, or just like being a father, and can an AI really do that better? But then you get into scenarios where, I mean, a lot of people grow up without their father, right? Just maybe by route of an early death, or just like something else. And is an AI maybe providing some parenthood there? And I guess what you guys are saying, though, is you're acknowledging that AI cannot replace all of our labor, because we still might want to go to an arts and crafts fair and purchase a piece of artwork for cultural reasons from a real human artist that we just resonate with and identify with. And that's still going to be a market and economy. What you're

• 55:20Ryan Sean Adams?

  saying is like, over time, that could become a smaller and smaller and smaller portion of the economy. And even the humans' purchasing power in this world could actually decrease. Because where is their wealth to go purchase the artwork actually coming from? And so you could imagine just that economy, that human-to-human kind of economy, where only humans can provide this, that just gets smaller and smaller over time. It's kind of a niche. And so the humans are maybe, I guess, disempowered, even though these economies still exist. Yeah.

• 55:49Speaker 03

  Or even it could be that the human wages stay roughly constant. Everyone has vaguely social, pro-social jobs. The money flowing into the human part of the economy comes from something, something in government, something, something existing, human wealth. And those are like, human wages are what they are today, but then also humans just don't really have political power anymore, because states worry about real things like energy and GPUs and military competition. And all of these fields are done by AI. And I think the human role has become a bit peripheral and not anymore tied to the real power that exists in the world. And I think I'm a bit worried about that, even if humans have their wage level at what it is right now.

• 56:26Speaker 02

  Another way to think about it, too, is I hear a lot that people will always want human teachers, right? Because there's this human interaction that you have with a teacher, and it's really hard to replace. A relevant question, though, is what will be the demand for schools? What is the incentive for states to fund mass public education in a world where they aren't receiving a return there? That doesn't mean it isn't going to happen, but you You should look at the underlying economic incentives. And it could be the case, as you describe, where like many, many fields are automated. And so the money flowing in this human economy is just increasingly limited or, you know, dwindles over time. I think there are a lot of ways which you can reach a pretty bad outcome through different mechanisms here. And a lot of our solutions base is focused on trying to keep humans meaningfully economically involved in many different ways, while also strengthening democratic incentives and democratic structures so that they can override capital incentives when they need to.

• 57:11Ryan Sean Adams?

  Well, just if we could stretch this a little farther and kind of like imagine a world here. So like, what, how, how do future nation states actually like make money in an AI dominated economy? Like, how do they tax, like, obviously now our tax mechanisms are just income tax, capital gains tax, consumption type tax, excise tax, increasingly tariffs, that's fun. So, but the nation state is really going to have to reorient around AI labors. And that, that, that's another interesting question. It's like, maybe actually the nation state is not the one in charge. I mean, we're in a world of, of nation states, but that is kind of a post feudal model that kind of arose on the back really of the last major technological change, which was the industrial revolution. Maybe we're going to reorganize.

• 57:56Ryan Sean Adams?

  Balaji Srivastava has this concept of the network state. And you sort of wonder if maybe some of these AI labs could be in a position to accrue such power that they actually become the dominant force, some kind of like, you know, open AI network state, complete with like a flag and Sam Altman as the, the president. I mean, like, who knows, right? What, how do you guys see this playing out?

• 58:18Speaker 03

  Yeah, I think there's definitely this question over like, do nation states continue as the main form of political organization? Or like main form of organization of power in the world? And I think there's something where like, so one, I think you should have some prior that these things are pretty sticky. Sort of like even the like, the like Catholic Church, you know, they were extremely powerful. They run, they run Europe for a few centuries in the past, but I'm like, you know, they don't want to run Europe anymore. But they're actually making a lot of commentary about AI recently. They're like, this stuff decays quite slowly.

• 58:46Speaker 02

  I think, I think this is, Rudolph has been subject to me spending the last couple of days really nerding out about this. I just, I literally just had a.

• 58:53Ryan Sean Adams?

  About the Catholic Church and AI?

• 58:54Speaker 02

  Yeah, so I have a reading list that I'm compiling right now. Because there's the most recent Pope, this is a slight sidetracked, most recent Pope. He's now said publicly, one of the reasons that he took Leo XIV is because Leo XIII had this very prescient encyclical called Rium Navarum on the Industrial Revolution in the 1890s, and he views AI as a similar style of societal reorganization. I actually, Francis had a whole lot of commentary here. I've got a reading list I'm working through right now. I just was, yesterday we were in Oxford and I was talking to a friar.

• 59:23Ryan Sean Adams?

  All right, Josh, Josh, new guest request. We've got to get the Pope on the list. Ask his thoughts on AI. All right, we'll do this in the Vatican. Sounds good, okay. Oh, fascinating. So we don't really know what the organizing political structure might be in this new world, but we could imagine it changes. But you're also saying that, hey, the nation state is pretty sticky. The Catholic Church is still doing like big things. Maybe it'll fade somewhat, probably won't go away. But that's kind of a TBD, like we don't know yet.

• 59:49Speaker 03

  Yeah, and also like if something, if we get, you know, AGI lab, network states, like the same incentives kind of apply to them by default. And also they aren't by default democracies unless they become democracies.

• 1:00:00Speaker 02

  Yeah, a core observation here is that AI can be both destabilizing and centralizing. And this seems kind of counterintuitive, but it could be the case that there's lots of very quick disruption and the winners of that disruption that can very quickly accumulate power and capital. I don't, I'm not saying that is certain, but that is, that is one scenario you could see here is that it can both destabilize a whole lot of things and then centralize power among the winners.

• 1:00:20Ryan Sean Adams?

  Yeah, the centralization of power seems to be a massive theme for you guys. Like where I'm getting out of this is definitely some worry about AI. I wouldn't call it doomerism, right? It's not, there's some, like, look, there could be a scenario where AI comes, kills humanity. I think you just like, you can see that point. That's not really the focus. The focus is more this attractor basin towards authoritarian totalitarianism, right? Which could be possible. I mean, this is even Daniel Schmoktenberger's work. I don't know if you guys have looked him up, but he talks about just with all of these tech revolutions, what we could see is this attractor basin towards like total society control to actually keep our tech in place. There's one more concept, though, we got to go through before we actually get to the solution.

• 1:01:01Speaker 02

  Let me say something about the power concentration thing, where I think like one, like-

• 1:01:05Speaker 03

  Thing that I think people, so throughout history, you've had really terrible tyrants and dictators, really terrible centralization of power. But like, all of them have fundamentally been limited by the fact that they, whoever the dictator is, they still need, they're not infinitely competent, they can't think incredibly fast, they still need a lot of other people to do things for them, they somehow need to get buy-in of like, a big bureaucracy, and then also like, populations they rule over. And like, fundamentally, their power like, is still rooted in humans. If you're a dictator, you're constantly paranoid about everyone else like overthrowing you.

• 1:01:31Ryan Sean Adams?

  Yeah, you know what, they also get strokes, they also, their life expectancy is only about 80 years.

• 1:01:36Speaker 03

  Even though they could pay for such good healthcare, exactly. But then like, once you don't need the bureaucracy of humans working for you, once you don't need the human military, you just have AI bureaucracy, you have like, AI military, you don't need the population to run your economy, like, constraints on how total totalitarian it can get, get a lot worse.

• 1:01:54Ryan Sean Adams?

  Indeed, they do. Okay, there is a way out, guys, all right?

• 1:01:58Speaker 03

  There is, yes.

• 1:01:59Ryan Sean Adams?

  For limitless listeners, if you're in despair now, never fear, we've got some solutions for you. But one more concept to cover, so this is, I think, the last essay before you kind of like, conclude all of the things and give some of your recommendations for the way out, which is this idea of the social contract, okay? An essay titled "Shaping the Social Contract", and what you're saying is, the intelligence curse is breaking the social contract. And I really like this diagram that you sort of show, which is just like, this nice equilibrium, balance of power. You've got like, three boxes here. You've got powerful actors, so these would be corporations, nation-states, you know, the big powerful networks, you've got the people, and then you've got the rules, okay? And so, there's dependency, there's lines of dependency between the powerful actors, the people, and

• 1:02:43Ryan Sean Adams?

  the rules. So, the powerful actors, they need the people for value, we've already established that, they need labor, right? And so, that's like, people, plus one for the people. The people can displace the powerful actors, we've seen that throughout history, French Revolution, just like American Revolution, right? If the powerful actors get too totalitarian, we stage revolts, right? The people are strong. And what we've done is, we've created these social contracts, basically, rules for society. And so, these rules are moral codes, but just like, more- I guess, yeah, like in more detail, it's kind of our legal system. It's the Constitution of the US. It's the Magna Carta. So there's this, and the people can influence the rules. The powerful actors have to, they're constrained by the rules.

• 1:03:27Ryan Sean Adams?

  We get the balance of power, separation of church and state, three co-equal branches, all of these things, right? It's like all very nice. And that's our current setup. That's the status quo. What you're saying is this whole AGI thing kind of disrupts the social contract because it, it means the people can't displace powerful actors, as you were just saying, Rudolph. It means the powerful actors, so the nation states, don't need the people for value. They can just pay, you know, for tokens in the AI geniuses in a data center. And then the powerful actors have the ability to influence the rules. The whole social contract is messed up. Did I, like, flesh this idea out a little bit more? Is this kind of what you're saying?

• 1:04:09Speaker 02

  So I'll zoom in on just a single interaction here, which I think helps articulate this. And I know you're listening, base, so let's zoom in on software engineer at Google. And let's say it's 2021, which I think, if I'm correct here, is like, that's the big year where like, it was, everyone is getting paid crazy amounts of money. You are negotiating with Google in your contract and you have something that they want. In this case, you have, like, you're really good at what you do. They want to hire you. Well, because of this, you get to extract a whole lot of concessions. You're a competitive on the marketplace. You get to ask for more RSUs. You get to ask for more stock. You get to ask for more money. You also get things like the free cafe on campus because they've got to attract you somehow. Or I think it's like 16 or 17 restaurants in Mountain Dew on their case. It's absolutely crazy. It's cool campus. You get a lot of these benefits because of that exchange.

• 1:04:54Speaker 02

  And of course, Google gets something out of you too, because they might pay you $400,000. But as long as they've done their, you know, vetting here, they're going to make a whole lot more than $400,000 from your labor. But everybody wins in this, in this relationship. Now imagine that Google is able to replace your labor with a machine that can code way better than you. This really disrupts the relationship, right? Because let's say, you know, in this case, it can create value for Google at a cheaper cost than you. It costs, I don't know, like $100,000 a year, or $150,000, $200,000. It's in the price range right there where it's really economically sensible for Google to cut you out of the process, but difficult for you to then go like trade, you know, 10 trillion clones of yourself and go compete with Google.

• 1:05:34Speaker 02

  And in the limit, this creates a world where powerful actors get more and more entrenched as capital substitutes for labor more and more perfectly. Your ability to displace them goes down while simultaneously, your ability to bargain with them also decreases because you don't have anything that they need. This might create a situation in which powerful actors get to set the rules and you are constrained by them. And it's very difficult for you to alter that relationship.

• 1:05:56Ryan Sean Adams?

  That follows through to the government too, right? And it's it's basically social contract with its citizens when like they don't need the citizens very much anymore. I I guess my my question here or a bit of pushback is, you know how we call them a social contract, right? And that's because it's sort of it's enforced socially. Yeah, there's power of the state, there's military, there's kind of like monopoly on violence types of things. But over time, human societies have been able to like construct their own social contracts. Like what is something like the constitution? That's just like a set of laws and legal codes and ideas that we all agree on in this in this nation called the United States of America, right? Like we put that in place.

• 1:06:39Ryan Sean Adams?

  It's you've all Noah Harari calls these kind of like myths, right? They're they're just like these shared beliefs that power so much of of human society. So my question is like, OK, if we get to kind of choose social contracts, why don't we just pick one that doesn't screw over all the humans? That doesn't screw over citizens and the labor. Like we put these things together, they're just shared myths, they're socially enforced. Why don't we pick one that's good? And by the way, if this AGI thing comes true, won't we have abundance too? Won't we have like basically 10% GDP a year? Won't we have like like fantastic wealth? At least somebody's making the wealth. And so this abundance, shouldn't this relieve the competitive pressures?

• 1:07:20Ryan Sean Adams?

  We don't have to think about, you know, the the basics of food and shelter because it's all provided for us. And so we're not in this competitive game anymore. We can just write, think about what makes society happy and pick a social contract that enforces that.

• 1:07:34Speaker 03

  I guess maybe one historical example here is like the British Empire tries to enforce a social contract on the US or like before it was the US and then the Americans were like, okay, actually, we don't think this is fine. And it was like a reality check the Brits and it turned out the Brits did not have the ability to enforce that. The institutions real powers against them. And then the Americans wrote their own social contract, which became the constitution. I think there's definitely a lot of power in like culture, institutional inertia, just like the beliefs that people have, which are like myths in the Harari sense, to like steer things and keep things on track. But then like over a long enough time scale, or like enough stuff happening in the world that like checks that like shakes like, okay, is there something behind this? Like if if someone tries to change that, either like, you know, on a bottom up way, because you know, there's some like social media movement or like from a top down way, if the like

• 1:08:18Speaker 03

  leader of a country decides to do something, like are those reality checks like, are does the like, does the economic structure and political structure like push back against that successfully? Or is this sort of like, oh, like you can actually shift it. And then if you can shift it, probably over time, it drifts kind of in the direction of the incentives over time.

• 1:08:34Ryan Sean Adams?

  How about this abundance idea, though? Going going back to that, right? So like where we have abundance, AIs are creating all of these things. Won't that relieve competitive pressure for us? Like, can't we get a utopia out of that?

• 1:08:47Speaker 02

  So I think at a core, you should be really concerned about any arrangement where the long run arrangement has you with very little actual power. And so I think it could be there's lots of abundance, but you aren't creating any of it. You aren't involved with the creation of any of it. And so your material power here is entirely political. This is just way less stable. Another thing to think about here is that I think we talked about this in the essay, that I think it's not really clear that competitive pressures or human greed have this intrinsic stopping point. I think to paint an additional example, though, here, it could be the case that the worst outcome is that we have abundance, but you don't have any say in what happens afterwards. And so your needs are met, but your political reality is quite constrained. I think about a state like China, which has like been able to simultaneously lift a whole lot of people out of poverty. And one

• 1:09:30Speaker 02

  of them, the Chinese miracle is the thing that happened, not hundreds of millions of people get lifted up out of poverty under Deng Xiaoping. But simultaneously, I wouldn't say that this has resulted in like crazy political freedoms for people in China. It could be that your material conditions Improve and yet simultaneously your power is unaffected. This has been quite a, you know, Herculean effort by the Chinese state to keep this equilibrium going. And the Chinese state is in many ways like responsive because it's afraid of losing legitimacy or really afraid of like revolutions. It has a zero tolerance policy on protest. But that is one outcome of, we just happen to think that you should be deeply concerned about scenarios in which you don't have the material power to guarantee abundance for yourself. And if you're written out of economic, social contracts, you are at this point at the mercy of the political one.

• 1:10:13Speaker 02

  We think the political one is better than nothing. We advocate really hard for strengthening that political contract so we can get to that outcome. But we don't think in the limit, it's what I, the only thing I want to be relying on. I really want to make sure that I have some real stake in the game here.

• 1:10:25Ryan Sean Adams?

  One last objection to all of this, which is basically Professor Arvind, he said, he wrote the intelligence curse. I don't know if you're familiar with him, but he has kind of this riff on.

• 1:10:34Speaker 02

  He wrote AI snake oil.

• 1:10:36Ryan Sean Adams?

  AI snake oil.

• 1:10:37Speaker 02

  I was gonna say congratulations.

• 1:10:39Ryan Sean Adams?

  Yeah, he plagiarized your work. So I just want to let you know right now.

• 1:10:43Speaker 02

  AI plagiarized. Yeah, I'd be frightened to find out. He wrote.

• 1:10:47Ryan Sean Adams?

  Sorry, Arvind.

• 1:10:47Speaker 02

  Ooh, that's, that's good to know. I didn't realize that title was taken.

• 1:10:51Ryan Sean Adams?

  AI snake oil is, he has this riff where he talks about, basically he's kind of downplays AGI. He basically thinks that AI is kind of like more akin to regular tech. And it's like, one of his riffs is there's a difference between AI capability and power. So there's, there's capability, right? All of this knowledge, intelligence inside of a data center. But then that's different than power. It's kind of constrained. Like maybe that idea of, you guys said earlier, part of the solution is not giving AIs the ability to accrue their own wealth, right? It's like wealth would be a vector for power. We don't necessarily have to give AIs wealth and power. And so capability and power could be somewhat isolated.

• 1:11:34Ryan Sean Adams?

  Like maybe this whole thing is, is just a question of like, who gets the power? How does that idea, the difference between AI capability and power kind of like factor into this whole analysis?

• 1:11:47Speaker 02

  If I'm understanding you correctly, you're saying that it could be the case, we don't delegate this power to AI systems and then it's relegated in the hands of people. Is that right?

• 1:11:54Ryan Sean Adams?

  Exactly. There's always humans in the loop, you know, like they can't get their own bank accounts or something. They can't accrue capital. We always have kind of a check on them. We don't have to give them the, the keys to the car.

• 1:12:05Speaker 02

  Well, I think nothing that we've argued is contingent on AI having this power in a self-directed way. One of the biggest oppressors of people in human history is other people. Totalitarian states require a whole lot of people doing that oppressing. And it could be the case that we've actually done is we've just expanded the power differential. We've made it like that that some people are far more powerful than others. This is already true today, but in the era of liberal capitalism and liberal democracies, your power as an individual, as a unit of society, has just really never been greater. And what we're saying here is it could be the case that for a couple of people, because they have existing access to capital and convert this directly into results, this could be a world where they have just such dramatic outlier ability to shape the world that their ability to materially impact your environment is really, really high and your ability to resist that is even lower than usual.

• 1:12:48Ryan Sean Adams?

  Okay, I feel like we flushed out the the intelligence curse to a sufficient degree. Let's talk about the the solution. Let's talk about how to break out of this intelligence curse. You've got three words here. You've got avert, you've got diffuse, and you've got democratize. Where do you want to take this? You want to you want to start with avert? How do we get out of this curse?

• 1:13:10Speaker 02

  Let's work it backwards. I think it's the initial affirmation, yeah.

• 1:13:13Ryan Sean Adams?

  Okay, start with democratize then. So this is, like, what's the idea here? That we're distributing the power to all of the people? We're just like not concentrating this in the hands of AI labs and AI models themselves? How do you think about the democratize word?

• 1:13:30Speaker 02

  So I think the way we'll flow this, if this makes sense, is I want to walk through really quickly just the observations backward because we started with democratize as the observation. And then I think we can kick it off with avert after that. And then what I mean by this, I just want to walk through the whole argument chain real quickly off the kind of initial observation that we have on how, why we need each step here.

• 1:13:46Speaker 03

  Yeah, so I guess the flow here is basically, as we mentioned, you know, like Norway, for example, solved the resource curse. They just had good institutions and therefore they can just all go to polls and like vote for, you know, everyone's well off and then they distribute the oil wealth between the people and everything is great. And so it's like great if we can get to the point where we have this very like democratic thing, a lot of people have power, they can affect the decisions that are made, we get like broad distribution of the benefits of AI, stuff like this, like, and there's various ways, we list some ways in which like For coordination and like various other things can help with this in our last section here. But yeah, this is basically great. There's various ways you can build tech to make this easier. And then kind of like the point we're making is that to be in the state where you can democratize

• 1:14:29Speaker 03

  and like have that be a stable equilibrium, often what matters is that you have you that like, you know, you get political power often when you have the economic power. So then this brings us like the idea that you you you need you need diffusion as well. If you want to like diffuse the benefits of AI to people such that then sort of like everyone gains in power, gains in capabilities, like continues having some stake in the economy and some like ownership stake over it. And therefore, like this sort of like makes the step about this makes the step about a democratization more stable because then it is actually an incentive of the powerful actors of people of everyone is to keep the like a democracy in place. So then we've gone from democratize to distribute to to diffuse. And then so there's this worry that some of those people have is if you diffuse AI too much, if you give everyone the AI,

• 1:15:11Speaker 03

  you know, it's like giving out this powerful technology that people use to do things like create bioweapons or like the all sorts of Nazi cyber attacks or whatever. Or like maybe the AI like takes over because it's misaligned and it is like very bad for everyone. And therefore, in order to make the diffusion step safe, in order to like prove that, you want to like avert the various catastrophes that could happen from widespread AI. And we're especially excited to hear about like how do we need the world against things like bio attacks, against cyber attacks, and also just making sure that we don't mess up on the alignment program.

• 1:15:39Speaker 02

  So from that, we've we're backwards here, right? So democratization is clearly a way out because democratic incentives can beat capital incentives. You can ensure all the things you want out of that. But we've noticed this pattern where the economic your economic power correlates with democracy. And it's oftentimes the engine of that. So we want to diffuse. But also, we want to make sure that diffusion happens the way it doesn't create the kind of catastrophes that either would just be bad in and of themselves or could give like license for states or other actors to freely powerfully centralize. So we have this avert section. So we kick things off with the avert and this backwards chain. And we've realized that in order to get to the democratization, there are some steps we're gonna have to take first.

• 1:16:14Ryan Sean Adams?

  Okay, so democratize is all about power diffusion to the people so that the people can hold the institutions in check, but it's a political type of thing, right?

• 1:16:28Speaker 02

  Yes.

• 1:16:28Ryan Sean Adams?

  And we have had democratic protocols in the past, right? And we have them right now, one person, one vote. We'll come back to that because I want to get into some tangible examples, but that is about distribution of power, I guess, and the humans having this power and retaining this power. And you're saying one way in order to do that is that other D-word, which is diffuse. And I think diffuse means give everybody access to AI tools. It can't just be a small percentage, maybe. Maybe you could sharpen the intuition there, but diffusion is about the distribution, the tools in the hands of everybody. And then avert is just like making sure that we don't completely go off the rails,

• 1:17:09Ryan Sean Adams?

  we have a misaligned AI or some sort of bioweapon. And also, I love that you say this because this is super important, a lot of people miss this, avert without requiring centralizing control. Because the attractor basin, when you start to clamp down and you avert and you sign letters like Pause AI, or you like Nick Bolstrom proposed kind of a high-tech panopticon where the government has to surveil everybody to make sure they're not doing a bioweapon with their LLM at home, right? Then we get this attractor basin of like totalitarian, authoritarian regimes that we then can't get out of. So you're saying avert this bad outcomes without requiring centralized control.

• 1:17:53Speaker 02

  Exactly. That's a logic chain we flow through. And the reason why we work through avert, diffuse, democratize, and the peace as opposed to the logic chain where you go backwards is because we think it's going to be really hard to diffuse unless you avert and really hard to democratize unless you diffuse. So this is kind of like the logic chain works backwards and then we present it forwards, if that makes sense.

• 1:18:09Ryan Sean Adams?

  Okay, it does. All right. Can we get into some real world examples? So avert.

• 1:18:14Speaker 02

  Let's yeah, let's start with averts. Let's kick it off.

• 1:18:16Ryan Sean Adams?

  Okay.

• 1:18:17Speaker 02

  I think the core observation here is that actually AI can do bad stuff. And this is like sometimes unpopular to say in like it's funny. I think we were we've got a position where saying unpopular truths to lots of different people. Certain truths are more popular to their communities than others. And I think it is the case that AI can make it a whole lot easier for a lot of people to do bad things. It can also make it a whole lot easier for systems themselves to, you know, lose control of them and take actions on on their own. And so our observation here is pretty simple. It'd be really bad if that's the end state. If AI is something that is bad for us and not good for us. And secondly, that historically, these kind of potential bad outcomes are the really powerful forces that justify centralization. You can see this here where just a host of tragedies.

• 1:18:59Speaker 02

  I think a lot about like on the September 11 attacks and how as a result of 9/11, the governments took very broad power grabs. The Patriot USA Patriot Act, which is actually a fun fact, an acronym. It was a response a couple of months later that resulted in what I would argue was a pretty significant restriction of civil liberties for Americans.

• 1:19:15Ryan Sean Adams?

  Have the cosign on that. Yep.

• 1:19:17Speaker 02

  Yeah, it gave the government warrant warrantless wiretapping capacities. Section 702 in particular has been quite controversial for a host of reasons, and I won't take a side on that argument. But the point is that it rapidly expanded government power. And government power, once distributed is very hard or once unlocked, is very hard to get back. The other observation that's important here, though, is that if AGI could in fact do a whole lot of economic tasks, you're not just centralizing a technology. This isn't just like giving only nukes to the government, which is a pretty common sense argument. You are also centralizing into a couple of points of failures, the development of a technology that might run your entire economy. In this case, it kind of looks like centralizing the means of production to the hands of a single, a single or a couple of actors.

• 1:19:57Ryan Sean Adams?

  That was another essay somebody wrote a while ago. Yeah.

• 1:19:59Speaker 02

  Yeah, I think we saw it. I think I think we don't cite that one. We do cite state revolution as an example of like it is, you know, we don't think that the idea of the transition state where a couple of people have all of the power and also all of the economic power is a good one. That's a state where you don't have very much power. And historically, your Stalin risks are pretty high here. Your risks of, you know, drawing the wrong leader out and putting them in the apparatus that you've built are pretty high.

• 1:20:21Ryan Sean Adams?

  Your P. Stalin, I guess, goes.

• 1:20:23Speaker 02

  In another essay, we called it P. Stalin specifically. Yeah. It's not a first one. It's not, it's in a, you have the piece on tacit knowledge, and we did, in fact, call it P. Stalin.

• 1:20:31Ryan Sean Adams?

  OK, OK, OK. Those are the goals. So how do we get there? I think one thing you cite, which is near and dear to our hearts, is Vitalik Guterin's defensive accelerationism. Maybe you could flush that out as like, you know, a part of the solution here.

• 1:20:45Speaker 03

  Yeah, I guess the basic idea of differential technology development or differential acceleration, whatever is called this month, is that like, you know, we can push, we can choose which order technologies arrive in to some extent. We can push the technologies we like and that help us guard against risks and like help humans, and we can like, you know, then hopefully get these technologies before we get to the bad worrying technologies. And for instance, we probably make sure that by the time like, you know, chapter B, they can do a cyber attack for you, that's where you're getting to the point where like our cyber defenses are good. And at the point where like the AIs can design bioweapons, that we've actually like hauled in the world against bioweapons system to an extent. And like, so this is true in the averse section is also true in like diffuse and democratize. I think the like core sort of spirit of most of our proposals is this thing of like, let's please build a technology that enable good things before we get to the threats.

• 1:21:29Speaker 03

  And actually, like by building technologies and like making them to come faster, we can avert a lot of these risks.

• 1:21:34Ryan Sean Adams?

  A lot of these things are defensive too, right? When you talk about biosecurity, it's not, it's more on the defensive, a low key of focus or cybersecurity is a kind of like defending from attackers, you know? Cryptography is sort of similar in that way, but we also need physical security. AI alignment, of course, the industry is like focused on that, but that's another element of the averting catastrophe here without centralizing. All right, let's get to diffuse. OK, so what does diffuse mean? To me, that's just like making sure that everybody, every human has AI superpowers. You know, so like the example that the tech CEOs give us is like, even Tim Cook doesn't have a better iPhone than you, kind of thing, right? We all have the equal access to iPhones, and that's great.

• 1:22:18Ryan Sean Adams?

  So does diffuse mean we all have equal access to these models and other people can't kind of like take them away? Is it like open source? What are the practical ways to diffuse this?

• 1:22:28Speaker 03

  Yeah, so I think basically the thing you want to do is help as many people as possible benefit economically from AI as quickly as possible such that by the time the really radical AI hits, people are like, they're like, first of all, there are more people who are like owners, there are more people who have like built companies, stuff like this, and then also you've like distributed technology more benefits, like everyone's gone to the AI power up. I like your phrase about, you know, everyone gets superpowers from AI. And then so in terms of like grand strategy here, we have this diagram at some point where you show that like you're these like two stages of diffusion where like first when AI is augmenting, you sort of like you want to, you want to like diffuse AI, which helps create decentralization. Like you diffuse AI and instead of like just

• 1:23:11Speaker 03

  the YGI labs have the AI, they use it to benefit themselves, everyone in society has access to AI. And what this means is you get to like decentralization because like the benefits of AI has been more widely spread. And then like the fact that you have decentralized the AI then helps you also diffuse the AI because then like once the humans are automated, they're automated, not by the big AI labs with their own AIs, and they still control the fruits of labor of the AIs that they own.

• 1:23:38Ryan Sean Adams?

  How supportive are you guys of open source then, models in this and like open source weights and just like all of that, that kind of movement? Is that, is that a key?

• 1:23:45Speaker 02

  Yeah, probably pretty supportive, especially in the world we've done a lot of the hard work here and on the hard work on, on, you know, proofing the world against the biggest disasters. Yes, exactly. And I think to kind of break this down concretely, this looks like two phases. There's this first phase where right now we're on this track where actually AI agency isn't that good, and yet everyone is investing more and more time to getting agency better. This opens up this interesting market opportunity where AI augmenting tools are both like under invested in and probably way better. Think about Cursor for a second. I know I keep coming back to Cursor. I love those guys. Cursor is not a tool that does all of the coding for you entirely. It is a tool where usually a software engineer who really understands what they're doing is like in the driver's seat. And it's enabled via the coding. It's enabled a lot of people who don't know exactly

• 1:24:30Speaker 02

  how to do it to still set the high level direction. But ultimately, you are in charge of what's happening. You are steering the ship. There's this huge market opportunity to build more tools. in that space right now and expand the window of AI human augmentation. We don't think this is like the long term permanent solution, but going ahead and starting in that direction now can both access like untapped economic incentive and really focus on what we can do today. What we're then excited about in the future is this a whole bunch of concepts here. But one of them is something like aligning the models directly to the user. Most people have some sort of like hidden knowledge that is very difficult to gather. And if you ultimately want like the single super intelligent singleton, you're gonna wanna have access to all that information. This gives you a wedge point wherein maybe it is the case that like you aren't the perfect data source because you are slow relative to your AIs in 2050.

• 1:25:15Speaker 02

  But it could be the case that there's this like AIs that are trained off of your tacit knowledge of your data. They understand you and could behave like you and can represent your taste in judgment faster. And these AIs are acting throughout the broader economy and they are interacting with other systems. Maybe the systems are smarter, but you have access to that information and behind that AI. And so this is a world where first we've extended the augmentative window. And second, we've aligned systems directly to the user such that even as the systems take off, they're still tied in a meaningful way to the user and therefore the user gets compensated in some way, shape or form by their economic activity.

• 1:25:46Ryan Sean Adams?

  Okay, that's cool. All right, let's talk about this last point then in more concrete terms. So democratize, right? So how do we do that? How do we, you know, let the humans still maintain some power, right? We're very used to like one person, one vote. I mean, are you talking about concepts like maybe you have an AI lawyer? Like you have the right to some sort of AI lawyer or data model to represent you. Is like, how do we really, you know, ensure that democracy and human ability political agency doesn't decrease in this world?

• 1:26:16Speaker 02

  Yeah, so I think so we take a very tech centric perspective here. This is not the essay in which we're gonna go out and propose how we solve everything in politics. But I think one thing that is underappreciated again is just like if you push forward technologies that make governance and like verification and coordination and trust easier. Then it becomes easier for society to like decide to do the bad things and to not do, sorry, to decide to do the good things and decide to avoid the bad things. So like where, so there are some ways in which like AI might help with this in particular, like the AI might help policymakers understand what voters think. You can imagine that, you know,

• 1:26:48Speaker 03

  Yeah, and then like in addition, I hope we understand what policymakers think. You know, the AI is going to advocate on your behalf, especially if you have a model that is aligned to you in particular. You can imagine that the AI is like, you know, you can have a provable guarantees that they're like, there's some like particular AI system that is making a judgment that is like more incorruptible than a human. You can imagine the AI like auditing information. There's this like fundamental difficulty with using humans to audit, that the humans have long-term memory. Whereas the AI is, you could just like, you know, use the AI. It's like process and context, and then the AI is deleted. But then it returns like yes or no, and like whether you're like a binding vice and protocol or like building a bioweapon. So you're like, you can like audit things without humans having knowledge about it afterwards.

• 1:27:27Speaker 03

  And like a bunch of ways like this where like technology gives you building blocks for like governments that might be and like more and more effective and more representative of the desires of the people. Then we can do right now is just like stacking humans into bureaucracies and having like laws about that.

• 1:27:40Ryan Sean Adams?

  I think these three words give us a good framework for for directionality, even if you can't solve everything in one essay, of course. It's like one last lens and filter for avert, diffuse and democratize. Let's say one society is kind of like chooses to do this and and puts these things in place in a more intentional way. Another society chooses not to, and there's this kind of like geopolitical race condition here, which is like we're in kind of a, you know, some sort of arms race for AI. Like, do you have, does your essay in The Intelligence Curse have anything to say about kind of that? Like, how do we, yeah, like one society chooses to go in the direction of trying to solve the intelligence curse, but another society races faster, fully embracing like the curse, they don't care.

• 1:28:23Ryan Sean Adams?

  They like, basically, do the authoritarian totalitarian societies win, you know, like no matter what? And so are we kind of like screwed, even if if we in the US or we in the the West kind of choose these ways out?

• 1:28:34Speaker 03

  So I think here is one of the things where the differential tech development approach is really powerful. It is fundamentally not about taking a sort of cuts to yourself and becoming less competitive and potentially being overrun by more safety oriented actors. It's about developing the technologies such as if they exist, doing the safe, good, pro-human thing is the winning strategy. And therefore it like shifts the equilibrium, it's not just about like, it's not reliant on coordination with other actors.

• 1:28:57Ryan Sean Adams?

  I like that. So we are doing the Munger thing of trying to get the incentives correct.

• 1:29:01Speaker 02

  Yeah. Exactly. Now, I will say incentives aren't everything. And I think we talk a little bit about some policies, especially in the democratized section. We talk about some of the more boring ones you hear all the time about this is like campaign finance reform and reforming anti-corruption laws and strengthening bureaucratic competence. And these all sound kind of boring today. But a really key thing is if you think incentives are about to get radically different and the self-interest of politicians might be much more powerful than it was in years previous, it is really important that the leaders that you're electing in the next couple of years are leaders that you would trust to make good decisions on your behalf in stressful situations. That integrity element that we've kind of lost in modern politics is much more important than ever because one of the most oftentimes, well, the one of the ways you can square like great man theory of history away with like a more incentives dominant view is that oftentimes

• 1:29:46Speaker 02

  the great men in history are those that take a decision that looks against the incentives and is ultimately the correct one. You really want to maximize your chance of grabbing one of those leaders when critical decisions come down because you could spend and you should spend as much effort as you can to get the incentives right. But you really also want to make sure the person that you have there is someone who at that critical moment might make a decision that is against those incentives if it matters, if it's important for your well-being. So there's that boring answer of you should vote for people who you actually trust, but you actually should vote for people who you actually trust.

• 1:30:15Ryan Sean Adams?

  Interesting. Unfortunately, it also feels like we're in a shortage of great men these days, at least in our politics. This has been very fascinating. I guess my question as we end this is what should listeners do with this information? Is there anything kind of actionable? I think it's a super valuable mental model and kind of like, hey, you might be out of a jaw. But what did people do, I guess personally with this information? What do you recommend listeners, you know, take action on?

• 1:30:39Speaker 02

  Yeah, I'd be just a big fan of, you know, go to the solution section of our essay series. We have a lot of specific tech ideas, you know, like read through this. If you're someone who might build something, you know, go and build something off this list or read this list, have your own idea for something in it like that is like helps these same goals and go build that out. And it's like, if we build the right technology that makes the equilibrium the good one, then... Yeah, there's this, there's this kind of, I hear this meme a lot in some of the more like air safety communities that, oh, if it's something that has like market value, the market will solve it. The market is made up of people. People are in the market. And they do things. And so you actually have, if you're going to do differential tech development, then some startup founders got to wake up and decide, okay, I'm going to go build this thing. And a VC has got to decide to back them. And we were not being, you know, we're not pontificating here.

• 1:31:23Speaker 02

  We can talk a bit more about it in the future, but the two of us are currently actively involved in taking a slice of this agenda and building this out ourselves. So we, we are going to go down this rabbit hole ourselves because, I don't know, it's really easy to point at a problem. It's pretty hard to build a solution. What we're much more excited about is building out the solution space. But I think there's stuff for people who aren't just in the tech community. There are policies that government should be thinking about enacting today. We call, for example, for like Operation Warp Speed for the DAX style technologies, the kind of things that could actually prevent major catastrophes and enable this like culture of innovation and democratization. Governments could be incentivizing that right now. I mean, Trump's first term did Operation Warp Speed. The bright people are in place to do something like that. Again, this massive moonshot. I think voters can start really thinking critically about what's going to happen in the next couple of years

• 1:32:08Speaker 02

  and electing politicians on those grounds. And if you're a student, I think about a blog post somewhere talking about like depending on who you are, what career decisions you might want to be making. Because again, if you think diffusion pressures are real, and it might take a while for this to accelerate, there are still some roles where it's pretty obvious that are going to go down first. Doing bigger, bolder things right now that actually require you to learn how to fail and be your own actor is really great preparation for a world where you might be able to command an army of agents or a lot of augmentative tools, even if the big companies aren't hiring junior analysts anymore. So there's a lot of things you can do right now today to both at a macro level, get us on the right path, or at a micro level, orient yourself for the coming wave.

• 1:32:44Ryan Sean Adams?

  That's great.

• 1:32:45Speaker 04

  So Luke and Rudolph, we spent the last 90 minutes kind of defining the intelligence curse, walking through it, providing solutions. I would love to end this on a positive note, on something a little optimistic. What happens if we solve the intelligence curse? I mean, what is the payoff? What do we get for solving this problem?

• 1:33:00Speaker 02

  We are talking about a, what could be the greatest revolution technologically of in human, at least at any time previously in history, and maybe like the final huge thing. The promise of that is honestly hard to fathom. It's things like curing diseases that we couldn't imagine of actual total abundance of unlocking crazy amounts of resources. Of really being able to provide what would have been just, you know, in this year and experienced only to the ultra elites, to everyone. That is a world that I want to be able to live in where we can do things like, you know, abolish poverty and abolish disease. And if we can get this right, the promise of artificial intelligence is that instead of having less agency and less control over your world, you get more with a whole lot less of the drawbacks.

• 1:33:45Speaker 02

  And I don't know, that's a vision I'm pretty excited about.

• 1:33:47Speaker 04

  That's a vision I can get very excited about, too. I want to thank both of you, Luke and Rudolph, for joining us today, walking us through everything. I know you guys mentioned you were working on some stuff. Where can people find you?

• 1:33:56Speaker 02

  So we've got a, I think, contact form or an email on intelligence-curse.ai. We're both also on Twitter. I think my handle is Luke underscore Drago underscore Rudolph. Yours is? Mine is currently at L R U D L underscore. Yeah, so if you want to reach out to us through the contact form there, or if you want to reach out to us on Twitter, we're both pretty active, unfortunately. We definitely tweet a little too much, and that's all right.

• 1:34:20Speaker 04

  Hey, for better or worse, but we very much appreciate you joining us today, walking us through this entire intelligence curse. I'm sure everyone listening now has a lot to chew on, a lot of interesting new questions to ask, a lot of new things to consider, whether it be the exciting case, the optimistic ending that we landed on, or any of those varied outcomes that we also discussed on the show. So Luke and Rudolph, thanks again. Thank you so much for joining us on the episode today.

• 1:34:39Speaker 02

  Thank you so much. And I guess the last thing I could say is I'm super excited, super optimistic and super excited where this could go. So I think even if we know about the problem, we also know how to solve, or we have a guess on how to solve it. And I think people should get more excited about jumping at that solution.

• 1:34:51Speaker 04

  Awesome.

• 1:34:52Speaker 02

  Yeah, please build a tech that will save the world.

• 1:34:54Speaker 04

  Absolutely, I love that. That's a really optimistic note to end it on. So thank you again, and I appreciate you guys taking the time.

• 1:35:00Speaker 02

  Take care. Thank you.