AGI is still 30 years away — Ege Erdil & Tamay Besiroglu

Ege Erdil https://www.lesswrong.com/users/ege-erdil and Tamay Besiroglu https://tamaybesiroglu.com/ have 2045+ timelines, think the whole "alignment" framing is wrong, don't think an intelligence explosion is plausible, but are convinced we'll see explosive economic growth (economy literally doubling every year or two). This discussion offers a totally different scenario than my recent interview with Scott and Daniel https://www.dwarkesh.com/p/scott-daniel. Ege and Tamay are the co-founders of Mechanize https://www.mechanize.work/ (disclosure - I’m an angel investor), a startup dedicated to fully automating work. Before founding Mechanize, Ege and Tamay worked on AI forecasts at Epoch AI https://epoch.ai/. Watch on Youtube https://youtu.be/WLBsUarvWTw; listen on Apple Podcasts https://podcasts.apple.com/us/podcast/agi-is-still-30-years-away-ege-erdil-tamay-besiroglu/id1516093381?i=1000703894255 or Spotify https://open.spotify.com/episode/68eeIiy3mT6PRlrTej9dtq?si=8bd51bdc846e47f6. ---------- Sponsors * WorkOS https://workos.com/ makes it easy to become enterprise-ready. With simple APIs for essential enterprise features like SSO and SCIM, WorkOS helps companies like Vercel, Plaid, and OpenAI meet the requirements of their biggest customers. To learn more about how they can help you do the same, visit workos.com https://workos.com/ * Scale’s https://scale.com/dwarkesh Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh https://scale.com/dwarkesh * Google's http://aistudio.google.com/ Gemini Pro 2.5 is the model we use the most at Dwarkesh Podcast: it helps us generate transcripts https://huggingface.co/spaces/dwarkesh/transcriber, identify interesting clips https://huggingface.co/spaces/dwarkesh/producer, and code up new tools. If you want to try it for yourself, it's now available in Preview with higher rate limits! Start building with it today at aistudio.google.com http://aistudio.google.com/. ---------- Timestamps (00:00:00) - AGI will take another 3 decades (00:22:27) - Even reasoning models lack animal intelligence (00:45:04) - Intelligence explosion (01:00:57) - Ege & Tamay’s story (01:06:24) - Explosive economic growth (01:33:00) - Will there be a separate AI economy? (01:47:08) - Can we predictably influence the future? (02:19:48) - Arms race dynamic (02:29:48) - Is superintelligence a real thing? (02:35:45) - Reasons not to expect explosive growth (02:49:00) - Fully automated firms (02:54:43) - Will central planning work after AGI? (02:58:20) - Career advice This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Today, I'm chatting with Tamay Besiroglu and Eggy Erdil. They were previously running Epoch AI and are now launching Mechanized, which is a company dedicated to automating all work. One of the interesting points you made recently, Tamay, is that the whole idea of the intelligence explosion is mistaken or misleading. Why don't you explain what you were talking about there?

• 0:22Speaker 01

  Yeah, I think it's not a very useful concept. It's kind of like calling the industrial revolution a horsepower explosion. Like, sure, during the industrial revolution, we saw this drastic acceleration in raw physical power. But there are many other things that were maybe equally important in explaining the acceleration of growth and technological change that we saw during the industrial revolution.

• 0:43Dwarkesh Patel?

  What is a way to characterize the broader set of things that the horsepower perspective would miss about the industrial revolution?

• 0:50Speaker 01

  So I think in the case of the industrial revolution, it was a bunch of these complementary changes to many different sectors in the economy. So you had agriculture, you had transportation, you had law and finance, you had urbanization and moving from rural areas into cities. There were just many different innovations that kind of, you know, happened simultaneously that gave rise to this change in the way of economically organizing our society. It wasn't just that we had more horsepower. That, I mean, that was part of it. But that's not the kind of central thing to focus on when thinking about the industrial revolution. And I think similarly, for the development of AI, sure, we'll get like a lot of very smart AI systems.

• 1:35Speaker 01

  But that will be one part among very many different moving parts that explain, you know, why we expect to get this transition and this acceleration in growth and technological change.

• 1:46Dwarkesh Patel?

  Yeah, I want to better understand how you think about that broader transformation. Before we do, the other really interesting part of your worldview is that you have longer timelines to get to AGI than most of the people in San Francisco who think about AI. When do you expect a drop in remote worker replacement? Yeah, maybe for me, that would be around like 2045 or... Wow, wait, and you?

• 2:11Speaker 01

  I'm a little bit more bullish. I mean, it depends what you mean by drop in remote worker and whether it's able to do like literally everything that can be done remotely or do most things.

• 2:20Dwarkesh Patel?

  I'm saying literally everything.

• 2:21Speaker 01

  For literally everything, yeah. Just shade, I guess, predictions by five years or like by 20% or something.

• 2:27Dwarkesh Patel?

  Why? Because we've seen so much progress over even the last few years. We've gone from chat GPT like two years ago to now we have models that can literally do reasoning, are better coders than me. And I I studied software engineering in college. I mean, I did become a podcaster. I'm not saying I'm like the best coder in the world. But if you made this much progress in the last two years, why would it take another 30 to get to full automation of human brains?

• 2:58Speaker 02

  Right.

• 2:58Dwarkesh Patel?

  Wait, I said that wrong. You know what I'm saying? Full automation of remote work.

• 3:01Speaker 02

  Yeah, yeah. So I think that a lot of people have this intuition that progress has been very fast. They like just look at trend lines and just like extrapolate like obviously it's gonna happen in like, I don't know, 2027 or 2030 or whatever. It's very bullish. And obviously, that's not a thing you can literally do. Like, there isn't like a trend. You can literally extrapolate of when do we get the full automation? Because if you look at the fraction of the economy that is actually automated, it's very like by AI, it's very small. So if you just extrapolate that trend, which is something, say, Robin Hanson likes to do, you're gonna say, well, it's gonna take centuries or something. Now, we don't agree with that view. But I think one way of thinking about this is like,

• 3:43Speaker 02

  how many like big things are there? How many core capabilities, components are there that AI systems need to be good at in order to have this very broad economic impact, maybe 10x acceleration in growth or something? How many things have you gotten like how over the past 10 years, 15 years? And we also have this compute-centric view.

• 4:05Speaker 01

  So just to double click on that, I mean, I think what I guess referring to is like, if you look at the past 10 years of AI progress, we've gone through about nine or 10 orders of magnitude of compute and we got various capabilities that were unlocked. So you had in the, you know, in the early period, people were kind of, you know, solving gameplay on on specific games, on very complex games. And you know, that happened 2015 to maybe 2020 and Go and chess and Dota and other games. And then, and then you had maybe, you know, sophisticated language capabilities that were unlocked with these search language models and maybe kind of advanced abstract

• 4:48Speaker 01

  reasoning and coding and maybe math that was maybe another big such capability that got unlocked. And so maybe there are a couple of these big unlocks that happened over the past 10 years. But it takes, you know, that that happened on the order of once every three years or so, or maybe one every three orders of magnitude of compute scaling. And so, and then you might ask the question, how many more such competencies might we need to unlock in order to be able to have a, an AI system that can match the capabilities of humans across the board, maybe specifically just on remote work tasks. And so then you might ask, well, maybe you need kind of coherence over very long horizons, or you need kind

• 5:32Speaker 01

  of agency and autonomy, or maybe you need multimodal, kind of full multimodal kind of understanding, just like a human would. And then you, you ask the question, okay, how long might that take? And, and so you can think about, well, just in terms of calendar years, you know, the, the previous unlocks took about, you know, you get one every three years or so. But of course, that previous period coincided with this rapid scale up of the amount of compute that we use for training. So we went through maybe nine or 10 orders of magnitude since AlexNet, you know, compared to the biggest models we have today. And, you know, we're, we're getting to a level where it's becoming harder and

• 6:16Speaker 01

  harder to scale up compute. And, you know, we've done some extrapolations and some analysis looking at specific constraints, like energy or GPU production. And based on that, it looks like we might have maybe three or four orders of magnitude of scaling left. And then you're really spending a pretty sizable fraction or a non trivial fraction of world output on just building up data centers, energy infrastructure, fabs.

• 6:39Dwarkesh Patel?

  Which is already like 2% of GDP, right?

• 6:41Speaker 01

  I mean, currently, it's less than 2%.

• 6:43Speaker 02

  Yeah, but also currently, most of it is actually not going through towards AI chips. Even most TSMC capacity currently is going towards mobile phone chips or something like that, right?

• 6:52Dwarkesh Patel?

  Even leading edge is good.

• 6:53Speaker 01

  Yeah, even leading edge is pretty small. But, but yeah, so that suggests that, you know, we might need a lot more compute scaling to get these additional capabilities to be unlocked. And then there's a question of do we really have that as a, you know, do we have that in us as an economy to be able to sustain that scaling?

• 7:14Dwarkesh Patel?

  And but it seems like you have this intuition that there's just a lot left to intelligence.

• 7:19Speaker 01

  Right.

• 7:19Dwarkesh Patel?

  When you play these models, it's like they're almost there. It's like you forget you're often talking to an AI.

• 7:25Speaker 02

  What do you mean they're almost there? Like, I don't know, like, I can't ask Claude to like pick up this cup and like put it over there.

• 7:31Dwarkesh Patel?

  Remote work, you know?

• 7:32Speaker 02

  Okay, but even for remote work, I can't ask Claude to like, I think the current computer use systems can't even like book a flight properly.

• 7:38Dwarkesh Patel?

  Right. How much of an update would it be if by the end of 2026, they could book a flight?

• 7:43Speaker 02

  I probably think by the end of this year, you're going to be able to do that. But that's like a very, very, like, nobody gets a job where they're paid to like book flights for like, like, that's not a task.

• 7:54Dwarkesh Patel?

  I think some people.

• 7:55Speaker 01

  I mean, if it's literally just book flight, job and without, you know.

• 8:00Speaker 02

  But I think that's an important point because a lot of people like look at jobs in the economy. And then they're like, oh, like that person, like their job is to just do X. But then, but then that's not true. Like, that's something they do in their job. But it's probably if you look at the fraction of their time on the job that they spend on doing that is a very small fraction of what they should do. It's just this popular conception people have, like, oh, travel agents, like, they just book hotels and flights, but that's not actually most of their job. So automating that actually wouldn't automate their job and it wouldn't have that much of an impact on the economy. So I think this is actually an important thing, that important worldview difference that separates us from people who are much more bullish because they think like jobs in the economy are much

• 8:42Speaker 02

  simpler in some sense, and they're going to take like much fewer competences to actually fully automate.

• 8:47Dwarkesh Patel?

  So our friend Leopold has this perspective of quote unquote unhobblings, where the way to characterize it might be like they're basically like baby AGIs already. And then there's because of the constraints we artificially impose upon them by, for example, only training them on text and not giving them the training data that is necessary for them to understand a Slack environment or Gmail environment. Or previously before inference time scaling, not giving them the chance to meditate upon what they're saying and really think it through. And not giving them the context about like what is actually involved in this job, only giving them this piecemeal, a couple of minutes worth of context in the prompt.

• 9:30Dwarkesh Patel?

  We're holding back what is fundamentally a little intelligence from being as productive as it could be, which implies that unhobblings just seem easier to solve for than entirely new capabilities of intelligence.

• 9:44Dwarkesh Patel?

  What do you make of that framework?

• 9:46Speaker 01

  I mean, I guess you could have made similar points five years ago and say, you know, you look at AlphaZero and there's this mini AGI there. And if only you unhobbled it by training it on text and giving it all your context and so on, like that just wouldn't really have worked. Like, I think you do really need to rethink how you train these models in order to get these capabilities.

• 10:09Dwarkesh Patel?

  But I think the surprising thing over the last few years has been that you can start off with this pre-trained corpus of the internet. And it just like, it's actually quite easy. Like, ChatGPT is an example of this unhobbling. Where 1% of additional compute spent on getting it to talk in a chatbot-like fashion with post-training is enough to make it competent, really competent at that capability. So why not think that agency? I mean, reasoning is another example where it seems like the amount of compute that is spent on RL right now in these models is a small fraction of total compute. Again, like reasoning seems like complicated and then you just like do 1% of compute and it gets you that. Why not think that computer use or long-term agency on computer use is a similar thing?

• 10:55Speaker 01

  So when you say reasoning is easy and it only took this much compute and it wasn't very much and maybe you look at the sheer number of tokens and it wasn't very much and so it looks easy, well, that's kind of true from our position today. But I think if you ask someone, build a reasoning model in 2015, then it would have looked insurmountable. You would have had to train a model on tens of thousands of GPUs. You would have had to solve that problem and each order of magnitude of scaling from where they were would pose new challenges that they would need to solve. You would need to produce kind of internet scale or tens of trillions of tokens of data in order to actually train a model that has the knowledge that you can then unlock and

• 11:40Speaker 01

  access by way of training it to be a reasoning model. You need to maybe make the model more efficient at doing inference and maybe distill it because if it's very slow, then you have a reasoning model that's not particularly useful. So you also need to make various innovations to get the model to be distilled so that you can train it more quickly because these rollouts take very long. It actually becomes a product that's valuable if it's a couple of tokens a second. As a reasoning model, that would have been very difficult to work with. So in some sense, it looks easy from our point of view, standing on this huge stack of technology that we've built up over the past five years or so. But at the time, it would have been very hard. And so my claim would be something like, I think the agency part might be easy in

• 12:25Speaker 01

  a similar sense, that in five years or three years' time or whatever, we will look at what unlocked agency and it'll look fairly simple. But the amount of work that, in terms of these complementary kind of innovations that enable the model to be able to learn how to become a competent agent, that might have just been very difficult and taken years of innovation and a bunch of improvements in hardware and scaling and various other things.

• 12:53Dwarkesh Patel?

  Yeah, I feel like what's dissimilar between 2015 and now, in 2015, if you were trying to solve reasoning, you just didn't have a base.

• 13:04Dwarkesh Patel?

  To start on, I don't know, maybe you try to like formal proof methods or something. But like there was, there was no leg to stand on where now you'd actually like, you have the thing, you have the pre-trained base model, you have these techniques of scaffolding, of post-trading, of RL. And so it seems like you are skeptical that, you, you think that those will look to the future as, say, AlphaGo looks to us now in terms of the basis of a broader intelligence. Why, yeah, and then I'm curious if you have intuitions on, why, why not think that like, like we, language models, as we have them now, are like, we got the, the big missing

• 13:48Dwarkesh Patel?

  piece right, and now we're just like plugging things on top of it.

• 13:50Speaker 02

  Well, I mean, I guess what is the reason for believing that? I mean, you could have looked at AlphaGo or Alpha, AlphaGo Zero, Alpha Zero, those seemed very impressive at the time, and you were just learning to play this game with no human knowledge, you're just learning to play it from scratch. And I think at the time it did impress a lot of people. And, but then people tried to apply it to math, they tried to apply it to other domains, and it didn't work very well, they weren't able to get like competent agents at math. So it's very possible that these models, at least the way we have them right now, you're going to try to do the same thing people did for reasoning, but for agency, it's not going to work very well, and then you're not going to.

• 14:30Dwarkesh Patel?

  You think it will, like you, you're saying at the end of 2026, we will have agentic computer use.

• 14:36Speaker 01

  I think I guess said you'd be able to book a flight, which is very different from having like full agentic computer use like a human.

• 14:44Dwarkesh Patel?

  Because like, the other things that you do on a computer is just made up of things like booking a flight.

• 14:50Speaker 02

  I mean, sure, but like, they are not, like, they are not disconnected tasks. That's like saying like, everything you do in the world is just like, you just move parts of your body, and then you like, move your mouth, your tongue, and then you like, roll your head, but like, that's, that's a very, like, yeah, like individually, those things are simple, but then how do you put them together, right?

• 15:09Dwarkesh Patel?

  Yeah, okay, so there's like two pieces of evidence that you can have, that are quite dissimilar. One, the, the meter eval, which we've been talking about privately, which shows that the task length over certain kinds of tasks.

• 15:24Dwarkesh Patel?

  I can already see you're getting ready. Has been double, AI's ability to do the kind of thing that it takes a human 10 minutes to do, or an hour to do, or four hours to do. The length of time for corresponding human tasks, it seems like these models seem to be doubling their task length every seven months. So the idea being that by like 2030, if you extrapolate this curve, they could be doing tasks that take humans one month to do, or one year to do. And then this like long-term coherency in executing our tasks is like fundamentally what intelligence is. So this curve suggests that like we're getting there. The other piece of evidence, I kind of feel like my own mind works this way of I get

• 16:07Dwarkesh Patel?

  distracted easily and it's like kind of hard to put, to keep a long-term plan in my head at the same time. And I'm like slightly better at it than these models. But they don't seem like that dissimilar to me. They're, I mean, even like, I would have guessed reasoning is just like a really complicated thing. And then it seems like, oh, it's just something like learning 10 tokens worth of MCTS of wait, let's go back, let's think about this another way. Like shade of thought alone just gets you this like boost. And so it just seems like intelligence is simpler than we thought. Maybe agency is also simpler in this way.

• 16:38Speaker 02

  Yeah, I mean, I would say that reasoning did seem, I mean, I think there's a reason to expect complex reasoning to not as be difficult, not be as difficult as people might have thought even in advance. Because a lot of the tasks that AI solved very early on were tasks of various kinds of complex reasoning. So it wasn't the kind of reasoning that goes into when a human solves a math problem. But if you look at the major AI milestones over, I don't know, since 1950.

• 17:07Dwarkesh Patel?

  Yeah.

• 17:08Speaker 02

  A lot of them are for complex reasoning. Like a chess is, you can say, a complex reasoning task. Go is, you could say, a complex reasoning task.

• 17:13Dwarkesh Patel?

  I think there are also examples of.

• 17:17Dwarkesh Patel?

  Long-term agency, like winning at StarCraft is an example of being agentic over a meaningful period of time.

• 17:24Speaker 02

  That's right. So the problem in that case is that it's a very specific, narrow environment. You can say that playing Go or playing chess, that also requires a certain amount of agency, and that's true. But it's not, like, it's a very narrow task. So that's like saying, if you construct a software system that is able to react to very specific, very particular kind of images, then you're, or very specific video feeds or whatever, then you're getting close to general sensor motor skill automation. But the general skill is something that's very different, and I think we're seeing that. We're like, we can't, like, we still are very far, it seems like, from an AI model

• 18:07Speaker 02

  that can take a generic game off Steam. Let's say you just download a game released this year, you don't know how to play this game, and then you just have to play it, right? And then, most games are actually not that difficult for a human, like.

• 18:21Dwarkesh Patel?

  I mean, what about Claude-based Pokemon? I don't think it was trained on Pokemon.

• 18:24Speaker 02

  Right, so that's an interesting example. First of all, I find the example very interesting because, yeah, it was not trained explicitly, like, it wasn't, they didn't do some RL on, like, playing Pokemon Red. But obviously, the model knows that it's supposed to play Pokemon Red, because there's tons of material about Pokemon Red on the internet. In fact, if you, if you were playing Pokemon Red, and you got stuck somewhere, you didn't know what to do, you could probably go to Claude and ask it, Claude, like, I'm stuck in Mount Moon, and like, what am I supposed to do? And then it could probably be able to give you a fairly decent answer. But that doesn't stop it from getting stuck in Mount Moon for 48 hours. So that's a very interesting thing, where it has explicit knowledge, but then, when it's actually playing the game, it, like, it doesn't behave in a way which reflects that it has that knowledge.

• 19:09Dwarkesh Patel?

  All it's gonna do is, like, plug, you know? Plug the explicit knowledge to its actions.

• 19:13Speaker 02

  Right, but is that, is that easy?

• 19:15Dwarkesh Patel?

  I just like, I'm not sure I understand why, like, okay. If you can leverage your knowledge from pre-training about these games in order to be somewhat competent at them.

• 19:26Speaker 02

  Yes.

• 19:26Dwarkesh Patel?

  I feel like that is some evidence of, okay, they're going to be using, they're going to be leveraging a different base of skills.

• 19:32Speaker 02

  Yes.

• 19:32Dwarkesh Patel?

  But that, with that same leverage, they're gonna have like a similar repertoire of abilities, right? It's like, if you've read everything about whatever skill that every human has ever seen.

• 19:43Speaker 02

  I mean, a lot of the skills that people have, that we don't have very good training data for them.

• 19:48Dwarkesh Patel?

  That's right, that's right. What would you want to see over the next few years that would make you think, oh, no, I'm actually wrong. And this was like the last unlock and it was like now just a matter of ironing out the kinks. And then we get the thing that will kick off the, dare I say, intelligence explosion.

• 20:04Speaker 01

  Yeah, so I, I think something that would reveal its ability to do very long context things. Use, you know, multimodal capabilities in a meaningful way and integrate that with reasoning and other types of systems. And also agency and being able to take action over a long horizon and accomplish some tasks that takes very long for humans to do. Not just in, you know, specific software environments, but just very broadly. Say, downloading an arbitrary game from Steam and, you know, something it's never seen before. It doesn't really have much training data. Maybe it was released after its training cutoff.

• 20:45Speaker 01

  And so there's no tutorials or maybe there's no earlier versions of the game that has been, you know, discussed on the internet. And then accomplishing that game and actually playing that game to the end and accomplishing these various milestones that, you know, are challenging for humans. Like, that would be a substantial update. I mean, there are other things that would update me too, like, you know, OpenAI making a lot more revenue than, than it's currently doing.

• 21:10Dwarkesh Patel?

  Is the 100 billion in revenue that would, in, according to their contract, mark them as AGI enough?

• 21:15Speaker 01

  I, I think that's, you know, not a huge update to me if that were to happen. So I think the update would come if it was in fact $500 billion in revenue or something like that. But then I would certainly update quite a lot. But 100 billion, that seems pretty kind of likely to me. Like not, I would assign that maybe on a 40% chance or something by the end.

• 21:37Dwarkesh Patel?

  Well, I mean, what is this like? If you've got a system that is, it just produces surplus terms worth 100 billion. And like, the difference between this and AlphaZero is AlphaZero is never going to make 100 billion dollars.

• 21:49Speaker 02

  I agree.

• 21:49Dwarkesh Patel?

  In the marketplace, right? So just the, what is intelligence? It's like you, something able to usefully accomplish its goals or your goals. 100, if people are willing to pay 100 billion dollars for it, that's pretty good evidence that it's like accomplishing some goals.

• 22:04Speaker 02

  Sure, I mean.

• 22:05Speaker 01

  People pay 100 billion dollars for all sorts of things, right? And like, that itself is not a very strong piece of evidence that it's going to be transformative, I think.

• 22:13Speaker 02

  People pay trillions of dollars for oil, but oil is not. Like, I don't know, it seems like a very basic point, but like, the fact that people pay you a lot of money for something doesn't mean it's going to transform the world economy. If only we manage to unhobble it, like, that's, that's a very different claim, right?

• 22:27Dwarkesh Patel?

  Look, a ton of B2B software companies start off by building self-serve, consumer-grade products. And that's fine at first. Eventually, though, you have to go after enterprise. The most successful and durable software companies of the last decade have all made this transition. But getting enterprise-ready is hard. Single sign-on, role-based access controls, and comprehensive audit logs are all actually quite complex and tedious to build. And they're ripe for bugs and annoying edge cases. These features take a ton of engineering time and capital, which you should be spending on the core product. For example, one of Slack's PMs said that they spent 30 million dollars building these features, and they were only half-done. That's where WorkOS comes in.

• 23:11Dwarkesh Patel?

  WorkOS has helped Vercel, Plaid, Vanta, OpenAI, and hundreds of others become enterprise-ready with APIs to integrate all of these common features. If you want to learn more, go to workos.com and tell them that I sent you. Okay, so then this brings us to the intelligence explosion. Because what people will say is, we don't need to automate literally everything that is needed for automating remote work, let alone all human labor in general. We just need to automate the things which are necessary to fully close the R&D cycle needed to make smarter intelligences. And if you do this, you get a very rapid intelligence explosion.

• 23:55Dwarkesh Patel?

  And the end product of that explosion is not only an AGI, but something that is superhuman potentially. These things are extremely good at coding. And they're good at the kinds of things that you would think, and reasoning. And it seems like the kinds of things that would be necessary to automate R and D at AI Labs. What do you make of that logic?

• 24:14Speaker 02

  I mean, I think if you look at their capability profile, it is like, if you compare it to a random job in the economy, I agree they are better at doing sort of coding tasks that will be involved in R and D compared to a random job in the economy. But in absolute terms, I don't think they're that good.

• 24:35Speaker 02

  I think they are good at things that maybe impress us about human coders. Like if you wanted to see, oh, like what is, what makes a person a really impressive coder? You might look at their competitive programming performance. I mean, in fact, companies often hire people based on, if they're relatively junior, based on their performance on these kinds of problems. But that is just impressive in the human distribution. So if you look in absolute terms at what are the skills you need to actually automate the process of being a researcher? Then what fraction of those skills do the AI systems actually have? Even in coding, like a lot of coding is you have a very large code base you have to work with. The instructions are very kind of vague.

• 25:17Speaker 02

  There isn't, for example, you mentioned a meter eval in which because they needed to make it an eval, all the tasks have to be kind of compact and closed and have clear evaluation metrics. Like here's a model, like get its loss on this, you know, dataset as low as possible, or whatever. Or like here's another model and it's like, its embedding matrix has been scrambled. Just fix it to recover like most of its original performance, et cetera. Those are not problems that you actually work on in AI R and D. They're like very artificial problems. Now, if a human was good at doing those problems, you would infer, I think logically, that that human is likely to actually be a good researcher.

• 26:00Speaker 02

  But if an AI is able to do them, like the AI lacks so many other. Competences that a human would have, not just a researcher, just an ordinary human, that we don't think about in the process of research. So our view would be automating research is, first of all, more difficult than people give it credit for. I think you need more skills to do it, and definitely more than models are displaying right now. And on top of that, even if you did automate the process of research, we think a lot of the software progress has been driven not by cognitive efforts, though that has played a part, but it has been driven by compute scaling. You just have more GPUs, you can do more experiments, you can grab more things, your experiments can be done at larger scales, and that is just a very important driver.

• 26:44Speaker 02

  Like, if you're just, if you're 10 years ago, 15 years ago, you're trying to figure out what software innovations are going to be important in 10 or 15 years, you would have had a very difficult time. In fact, you probably wouldn't even conceived of the right kind of innovations to be looking at, because you would be so far removed from the context of that time with much more abundant compute and all the things that people would have learned by that point. So these are two components of our view. Research is harder than people think and depends a lot on compute scaling.

• 27:16Dwarkesh Patel?

  Sorry, can you put a finer point on what is the kind of thing, what is an example of the kind of task which is very dissimilar from train a classifier or debug a classifier that is relevant to AI R and D?

• 27:30Speaker 01

  I think it's like, you know, examples might be introducing novel, having novel innovations that are very useful for unlocking innovations in the future. So that might be, you know, introducing some novel way of thinking about a problem. Or introducing, so maybe a good example might be in mathematics, where we have these reasoning models that are extremely good at solving math problems.

• 27:58Speaker 02

  Very short horizon.

• 27:59Speaker 01

  Yeah, sure. Maybe not extremely good, but certainly better than I can and better than maybe most undergrads can. And so, you know, they can do that very well, but they're not very good at coming up with novel conceptual schemes that are useful for making progress in mathematics. So, you know, it's able to solve these problems that you can kind of neatly excise out of some very messy contexts. And it's able to make a lot of progress there. But within some much messier contexts, it's kind of not very good at figuring out what directions are especially useful for, you know, you to build things or kind of make incremental progress on that enables you to have a big kind of innovation later down the line.

• 28:44Speaker 01

  So thinking about both this like larger context as well as maybe much longer horizon, kind of much fuzzier things that you're optimizing for, I think it's much worse at those types of things.

• 28:54Speaker 02

  Right, so I think one interesting thing is if you just look at these reasoning models, they know so much, especially the large ones, because, I mean, they know in literal terms more than any human does in some sense. And well, we have unlocked these reasoning capabilities on top of that knowledge. And I think that is actually what's enabling them to solve a lot of these problems. But if you actually look at the way they approach problems, they, like, the reason what they do looks impressive to us is because we have so much less knowledge. And the model is approaching the problems in a fundamentally different way compared to a human would. A human would have much more limited knowledge and they would usually have to be much more creative in solving problems because they have this lack of knowledge.

• 29:38Speaker 02

  While the model knows so much, like you'd ask it some obscure math question where you need like some of those specific theorem from 1850 or something, and then it would just like know that if it's like a large model. So that makes the difficulty profile very different. And if you look at the way they approach problems, the reasoning models, they are usually not. Creative. They are very effectively able to leverage the knowledge they have, which is extremely vast, and that makes them very effective in a bunch of ways. But you might ask the question, has a reasoning model ever come up with a math concept that even seems like slightly interesting to a human mathematician? And I've never seen that.

• 30:19Dwarkesh Patel?

  I mean, they've been around for all of six months.

• 30:21Speaker 02

  But that's a long time.

• 30:22Speaker 01

  Lots of people have been, I mean, that's a long time.

• 30:24Speaker 02

  But like, just think about the.

• 30:25Speaker 01

  One mathematician might have been able to like do a bunch of work over that time. And they have produced orders of magnitude fewer tokens on math.

• 30:34Speaker 02

  That's right, that's right. Yeah, and then, and then like I just want to emphasize it because like just think about the sheer scale of knowledge that these models have. Like, it's enormous from a human point of view. So it is actually like quite remarkable that like there is no interesting recombination, no interesting, oh, like this thing in this field looks kind of like this thing in this other field. There's no like innovation that comes out of that. And like, it doesn't have to be like a big math concept. It could be just like a small thing that maybe you could add to like a, I don't know, like Sunday magazine on math that people used to have. But there isn't even like an example of that.

• 31:08Speaker 01

  I think it's useful for us to explain like a very important framework for our thinking about what AI is good at and what AI is lagging in, which is this idea of kind of more of X paradox, that things that seem very hard for humans, AI systems tend to make much faster progress on. Whereas things that look a bunch easier for us, kind of AI systems that totally struggle are often totally incapable of doing, of like doing that thing. And so, you know, this, this kind of abstract reasoning, you know, playing chess, playing Go, playing Jeopardy, doing kind of advanced math and solving math problems.

• 31:48Speaker 02

  There are even stronger examples, like multiplying a hundred digit numbers in your head, which is, which is the one that's called solved first out of almost any other problem. Or like following like very complex sort of symbolic logic arguments, like deductive arguments. Which people actually, yeah, people actually struggle with that a lot. Like, how do premises logically follow from conclusions? Like, people have a very hard time with that, very easy for formal proof systems.

• 32:13Speaker 01

  An insight that is related and is quite important here is that the kind of very, the tasks that humans seem to struggle on and AI systems seem to make much faster progress on are things that have emerged fairly recently in evolutionary time. So, you know, advanced language use emerged in humans maybe a hundred thousand years ago, and certainly playing chess and Go and so on are very recent innovations. And so evolution has had much less time to optimize for them, in part because they're very new, but also in part because when they emerged, there was a lot less pressure because it conferred kind of small fitness gains to humans. And so, evolution didn't optimize

• 32:53Speaker 01

  for these things very strongly. And so it's not surprising that on these specific tasks that, you know, humans find very impressive when other humans are able to do it, that AI systems are able to make a lot of fast progress. In humans, these things are often very strongly correlated with other kind of competencies, like being good at just, you know, achieving your goals or being a good coder is often very strongly correlated with solving kind of coding problems, or being a good engineer is often correlated with solving competitive coding problems. But in AI systems, the correlation isn't quite as strong. And even, even within AI systems, it's the case that, you know, the

• 33:37Speaker 01

  strongest systems on competitive programming are not even the ones that are best at actually helping you code. So like, you know, O3 Mini's Hi seems to be maybe the best at solving competitive code problems, but, but it isn't the best at actually helping you write code.

• 33:54Speaker 02

  It doesn't get most of the enterprise revenue from places like Courser or whatever, like that's just Claude, right?

• 33:59Speaker 01

  Right, right, right. So, but an important insight here is that, you know, the things that we find very impressive when humans are able to do it, we should expect that AI systems are able to make a lot more progress on that. And but we shouldn't update too strongly about just their general competence or something. But because we should recognize that this is a very narrow subset of relevant tasks that that humans do in order to be a competent economically valuable agent.

• 34:26Dwarkesh Patel?

  Yeah. First of all, I actually just like really appreciate that there is an AI organization out there where because there's other people who take the compute perspective seriously, think empiric or try to think empirically about scaling laws and data and whatever. And it's striking how often that like taking that perspective seriously leads people to just be like, OK, 2027 AGI, which might be correct, but it is like just interesting to get like, no, we've also looked at the exact same arguments, the same papers, the same numbers, and like we've come to a totally different conclusion. On all these arguments, I think this is all fascinating. OK, so I asked Dario this exact question two years ago when I interviewed him, and it went viral over Twitter.

• 35:11Dwarkesh Patel?

  Didn't he say AGI in two years? That but Dara's always had short timelines. OK, but we are two years later. Did he say two years? I think he actually did say two years. Did he say three years?

• 35:20Speaker 01

  So we have one more year. Better work hard.

• 35:25Dwarkesh Patel?

  But he's, I mean, I think he's like, he in particular has not been that well calibrated. He's like, oh, like in 2018, he had like.

• 35:33Speaker 01

  I remember talking to like a very senior person who's now at Anthropic.

• 35:37Dwarkesh Patel?

  Yeah.

• 35:38Speaker 01

  In 2017, and then he told various people that they should they shouldn't do a PhD because by the time they completed it, you know.

• 35:45Dwarkesh Patel?

  That's right, that's right.

• 35:46Speaker 01

  Everyone will be automated.

• 35:47Dwarkesh Patel?

  Yeah, yeah. So anyways, I asked him this exact question, right? Because he has short timelines, which is that if a human knew the amount of things these models know, they would be finding all these different connections. And in fact, we this is, I was asking Scott about this the other day when I interviewed him, Scott Alexander, and he said, like, look, humans also don't have this kind of logical omniscience. I'm not saying we're omniscient, but we have examples of humans finding these kinds of connections. This is not an uncommon thing, right? I think his response to that was that these things are just not trained in order to find these kinds of connections. But if you if you like. Their view is that it would not take that much extra compute in order to build some RL environment in which they're incentivized to find these connections. Next token prediction just isn't incentivizing them to do this, but the RL required to do this would not be that.

• 36:32Dwarkesh Patel?

  Or set up some sort of scaffolds. I think actually Google DeepMind did do some similar scaffolds to make new discoveries. And I didn't look into how impressive the new discovery was. They claimed that some new discovery was made by an LLM as a result. On the Moravec's paradox thing, this is actually a super interesting way to think about AI progress. But I would also say there that if you compare animals to humans, long-term intelligent planning, like an animal is not gonna like help you book a flight either. An animal is not gonna do remote work for you. Or even do the kinds of things. I think what separates humans from other animals is that we can hold long-term, we can come up with a plan and execute on it,

• 37:16Dwarkesh Patel?

  whereas other animals often had to go by instinct or within the kinds of environments that they have evolutionary knowledge of, rather than like I'm put in the middle of the savanna or I'm put in the middle of the desert or I'm put in the middle of the tundra and I'll learn how to make use of the tools and whatever there. Like I actually think there's a huge discontinuity between humans and animals in their ability to survive in different environments, just based on their knowledge. And so it's like a recently optimized thing as well. And then I'd be like, okay, well, AI is like, we got it soon, AI will optimize it for fast.

• 37:49Speaker 02

  Right, so I would say if you're comparing animals to humans, it's kind of a different thing. I think animals, like if you could put the competences that the animals have into AI systems, that might just already get you to like AGI, like already. I think the reason why there is such a big discontinuity between animals and humans is because animals have to rely entirely on natural world data, basically, to train themselves. Like imagine that the only thing as a human that you saw was like nobody talked to you. You didn't read anything. You just had to learn by experience, maybe to some extent by imitating other people.

• 38:31Speaker 02

  But not like you have no explicit communication. Well, it would be very inefficient. Like what's actually happening is that you have this, I think some other people have made this point as well. Is that evolution is sort of this outer optimizer that's like improving the software efficiency of the brain in a bunch of ways. There's some genetic knowledge that you inherit, not that much because there isn't that much space in the genome. And then you have this lifetime learning, which is, you don't actually see that much data during lifetime learning. A lot of this is redundant and so on. So what changed, seems to have changed with humans compared to other animals is that humans became able to have culture.

• 39:08Speaker 02

  And they have language, which enables them to like have like a much more efficient training data modality compared to animals. They also have, I think, stronger ways in which they like tend to imitate other humans and learn from their skills. So that also enables this knowledge to be passed on. I think animals are pretty bad at that compared to humans. So basically as a human, you're just being trained on like much more efficient data. And that creates further insights to be then efficient at learning from it. And then that creates this feedback loop where the selection pressure gets much more intense. So I think that's roughly what happened with humans. But a lot of the capabilities that you need to be like a good worker in the human economy, animals already have.

• 39:52Speaker 02

  So they are able to, like they have quite sophisticated sensory motor skills. I think they are actually able to do, like animals are actually able to pursue long-term goals.

• 40:03Dwarkesh Patel?

  But ones that they have been instilled by evolution, like I think like a lion will find a gazelle. And that is a complicated thing to do and requires stalking and blah, blah, blah.

• 40:11Speaker 02

  But when you say it's been instilled by evolution, like there isn't that much information in the genome to.

• 40:16Dwarkesh Patel?

  But it just like, I think if you put the lion in the Sahara and you're like, go find lizards instead. Okay, so suppose you put a human and they haven't seen the relevant training data.

• 40:26Speaker 02

  I think they'd like.

• 40:28Dwarkesh Patel?

  They do slightly better.

• 40:29Speaker 02

  Slightly better, but not that much better. Like, I think a lot of the, like, again, didn't you have recently have an interview?

• 40:36Dwarkesh Patel?

  Joseph Henrick.

• 40:37Speaker 02

  Yeah, so like, he would probably tell you that.

• 40:40Dwarkesh Patel?

  That's right, okay, and I think what you're making is actually a very interesting and subtle point. That is an interesting implication. So often people point to, they say that ASI will be this huge discontinuity because, well, we have this huge discontinuity in the animal-to-human transition, where like something is like, not that much change between pre-human primates and humans genetically, but it resulted in this humongous change in capabilities. And so they say, well, why not expect something similar between human-level intelligence and superhuman intelligence? And the point you're making is that, or at least one implication of the point you're making, is that actually, it wasn't that we just gained this incredible intelligence.

• 41:23Dwarkesh Patel?

  Because of biological constraints, we, animals have just been like held back in this really weird way, that no AI system has been arbitrarily held back.

• 41:34Speaker 02

  Yes.

• 41:35Dwarkesh Patel?

  Of like not being able to communicate with other copies or with other knowledge sources. And so since AIs are not held back artificially in this way, there's not going to be a point where we should take away that hobbling, you know, and then now they're like, now they explode. Now, actually, I think I would disagree with that. The implication that I made, I would actually disagree with. I'm like a sort of like unsteerable chain of thought, I just, but because, as we wrote a blog post together about AI corporations, where we discussed, actually, there will be a similar unhobbling with future AIs, which is not about the intelligence, but a similar level of bandwidth and communication and collaboration with other AIs, which is a similar

• 42:17Dwarkesh Patel?

  magnitude of change from non-human animals to humans, in terms of their social collaboration, that AIs will have with each other because of their ability to copy all their knowledge exactly, to merge, to distill themselves, to scale.

• 42:28Speaker 01

  Maybe before we talk about that, I think just like a very important point to make here, which I think underlies some of this disagreement that we have with others, about both this argument from the transition from kind of non-human animals to humans is

• 42:44Speaker 01

  This like focus on intelligence and reasoning and, and you know, R and D.

• 42:50Dwarkesh Patel?

  Yeah.

• 42:51Speaker 01

  Which is enabled by that intelligence as being just enormously important. And so if you think that you get this very important difference from, you know, this, this transition from, from primates, non-human primates to, to humans, then, then, then you think that in some sense, you get this enormously important unlock from fairly small scaling and say brain size or something. And so then you might think, well, AI could be, you know, if we, if we scale beyond the size of, of, of training runs that, you know, the amount of training compute that the human brain uses, which is maybe on the order of, you know, 10, 24 flop or whatever, which we've recently surpassed, then maybe surpassing it just a little bit more enables us to unlock very sophisticated intelligence.

• 43:36Dwarkesh Patel?

  Yeah.

• 43:36Speaker 01

  In the same way that humans have much more sophisticated intelligence compared to non-human primates. And I think part of our disagreement is that intelligence is kind of important, but just having a lot of, a lot more intelligence and reasoning and good reasoning, isn't something that will kind of accelerate technological change and economic growth very substantially. Like, it isn't the case that the world today is just like totally bottlenecked by not having, you know, not having enough good reasoning. And that, that's not really what's bottlenecking the world's ability to grow much more substantially. I, I think that we, we might have some disagreement about this particular argument, but I think

• 44:18Speaker 01

  what's also really important is just that we have a different view as to how this acceleration happens. That it's not just having like a bunch of really good reasoners that give you this technology that then accelerates things very drastically, because that, that alone is not sufficient. You need kind of complementary innovations in other industries. You need the economy as a whole, growing and supporting the development of these various technologies. You need the various supply chains to be upgraded. You might need demand for the various products that, that are being built. And so we have this view where actually this very broad upgrading of your technology and, and your economy is important, rather than just having very good reasoners and very, very, very good reasoning tokens that,

• 45:03Speaker 01

  that gives us this acceleration.

• 45:04Dwarkesh Patel?

  Hmm. All right. So this brings us back to the intelligence explosion. Here's the argument for the intelligence explosion. Look, you're right that certain kinds of things might take longer to come about. But this core loop of software R and D that's required.

• 45:27Dwarkesh Patel?

  If you just look at like what kinds of progress is needed to make a more general intelligence. You may be right that it needs more experimental compute, but like we're just getting, as you guys have documented, we're just getting like a shit ton more compute every single year for the next few years.

• 45:41Speaker 01

  Yep.

• 45:41Dwarkesh Patel?

  So you can imagine that intelligence will be the next few years where in 2027, there'll be like 10x more compute than there is now for AI. And you'll have this effect where the AIs that are doing software R and D are finding ways to make running copies of them more efficient, which has two effects. One, you're increasing the population of AIs who are doing this research. So more of them in parallel can find these different optimizations.

• 46:12Dwarkesh Patel?

  And a subtle point that they'd often make here is software R and D in AI is not just Ilya type coming up with new transformer-like architectures. To your point, it actually is a lot of like, you gotta like, I mean, I'm not a AI researcher, but I assume there's like from the lowest level libraries to the kernels to making RL environments to finding the best optimizer to, there's just like so much to do. And in like parallel, you can be doing all these things or finding optimizations across them. And so you have two effects, going back to this. One is you, you know, if you look at the original GPT-4 compared to the current GPT-40, I think it's like, what, it's like, how much cheaper is it to run?

• 46:57Speaker 01

  It's like, maybe a hundred.

• 46:59Dwarkesh Patel?

  Yeah, yeah.

• 47:00Speaker 01

  So you have this fact. Times for the same capability or something.

• 47:02Dwarkesh Patel?

  Right, so you're, they're finding ways in which to run more copies of them at like, you know, 100x cheaper or something, which means that the population of them is increasing. And the higher population is then helping you find more efficiencies. And not only does that mean you have more researchers, but to the extent that what's the complementary input is experimental compute, it's not the compute itself, it's the experiments. And the more efficient it is to run a copy. Or to develop a copy, the more parallel experiments you can run because now you can do a GPT four scale training run for much cheaper than you could do it in 2024 or 2023. And so for that reason, also, this like software only singularity sees more researcher

• 47:44Dwarkesh Patel?

  copies who can run experiments for cheaper, dot, dot, dot. They initially are maybe handicapped in certain ways that you mentioned, but through this process, they are rapidly becoming much more capable. What is wrong with this logic?

• 47:56Speaker 01

  So I think the logic, like, the logic seems fine.

• 48:00Dwarkesh Patel?

  Yeah.

• 48:01Speaker 01

  I think this is like a decent way to think about this problem, but I think that it's useful to draw on a bunch of work that, say, economists have done for studying the returns to R and D and what happens if you 10x your inputs, the number of researchers, what happens to innovation or the rate of innovation. And there, you know, they point out these kind of two effects where, you know, as you do more innovation, then you get to kind of stand on top of the shoulders of giants and you get the benefit from past discoveries and it makes you as a scientist more productive. But then there's also kind of diminishing returns that the low hanging fruit has been picked and it becomes harder to make progress. And overall, you can summarize those estimates as thinking about the kind

• 48:43Speaker 01

  of returns to research effort. And, you know, we've looked into the returns to research effort in software specifically, and we look at a bunch of domains in traditional software or, you know, like linear integer solvers or SAT solvers, but also in AI, like computer vision and RL and language modeling. And there, like, if this model is true, that all you need is just cognitive effort. It seems like the estimates are a bit ambiguous about whether this results in this acceleration or whether it results in just merely exponential growth. And then you might also think about, well, it isn't just your research effort

• 49:24Speaker 01

  that you have to scale up to make these innovations because you might have complementary inputs. So as you mentioned, experiments are the thing that might kind of bottleneck you. And I think there's a lot of evidence that in fact, these experiments and scaling up hardware is just very important for getting progress in the algorithms and the architecture and so on. So in AI, This is true for software in general, where if you look at progress in software, it often matches very closely the rate of progress we see in hardware. So for traditional software, we see about a thirty percent roughly increase per year, which kind of basically matches Moore's law. And in AI, we've seen the same until you get to the deep learning era, and then you

• 50:09Speaker 01

  get this acceleration, which in fact coincides with the acceleration we see in compute scaling, which gives you a hint that actually the compute scaling might have been very important. Other pieces of evidence, you know, besides this coincidental rate of progress, other kind of pieces of evidence are, you know, the fact that innovation in algorithms and architectures are often concentrated in GPU-rich labs and not in the GPU-poor parts, you know, in the world, like academia or maybe smaller research institutes. That also suggests that having a lot of hardware is very important. If you look at specific innovations that seem very important, the big innovations

• 50:51Speaker 01

  over the past five years, many of them have some kind of scaling or hardware-related motivation. So, you know, you might look at the transformer itself was about how to harness more parallel compute. Things like flash attention was literally about how to implement the attention mechanism more efficiently, or things like the Chinchilla scaling law. And so many of these big innovations were just about how to harness your compute more effectively. That also tells you that actually the scaling of compute might be very important. And I think there's just like many pieces of evidence that points towards this complementarity picture. So I would say that not only, like even if you assume that experiments are not particularly

• 51:36Speaker 01

  important, the evidence we have, both from estimates of AI and other software, although the data is a bit... Is not great suggests that, you know, maybe you don't get this kind of hyperbolic, faster than exponential, you know, super growth in the, in the overall algorithmic efficiency of systems.

• 51:55Dwarkesh Patel?

  I'm not sure I buy the argument that because these two things compute and EA progress have risen so concomitantly that this is a sort of causal relationship. So broadly, the industry as a whole has been getting more compute and as a result, making more progress. But within, if you look at the top players, there's been multiple examples of a company with much less compute but a more coherent vision, more concentrated research effort, being able to beat incumbent who has much more compute. So OpenAI initially beating Google DeepMind. And if you remember, there was these emails that were released between Elon and Sam and so forth, where like, we gotta start this company because they've got this bottleneck

• 52:37Dwarkesh Patel?

  on the compute and like, look how much more compute Google DeepMind has. And then OpenAI made a lot of progress. Similarly, now with OpenAI versus Anthropic and so forth. And then I think just generally, your argument is just like too outside view, when we just do know a lot about like, what is, like, you're just like, this very macroeconomic argument that I'm like, well, why don't we just ask the AI researchers?

• 53:00Speaker 01

  I mean, AI researchers will often kind of overstate the extent to which just cognitive effort and doing research is important for driving these innovations. Because that's often kind of convenient or useful. They will say the insight was, you know, was derived from some kind of nice idea about statistical mechanics or some nice equation in physics that says that we should do it this way and then, and then, but often that's kind of an ad hoc story that they tell to make it a bit more compelling to, to the kind of reviewers.

• 53:37Dwarkesh Patel?

  So Daniel, Daniel mentioned this like survey he did, where he, Daniel Kukutala. He asked a bunch of AI researchers, if you had 1/30 the amount of compute, and he did 1/30 because EAs will be, as opposed to anything 30 times faster. If you had 1/30 the amount of compute, how much, how much did your progress slow down? And they say, I make a third of the amount of progress I normally do. So that's just a pretty good like substitution effect of. If you get one tenth the compute, your progress only goes down one third. I was talking to an AI researcher the other day, who's like, just like one of these like cracked people, gets paid millions and tens of millions of dollars a year, probably. And we asked him, how much do these AI models help you in domains you already are,

• 54:23Dwarkesh Patel?

  you know, how much does the AI models help you in AI research? And he said, in domains that I'm already quite familiar with, where I'm just closer to autocomplete, it's like, saves me four to eight hours a week. And then he said, but in domains where I'm actually less familiar, where it's like, I need to draw new connections, I need to understand how these different parts relate to each other and so forth. It saves me closer to 24 to 36 hours a week, right? So, and then that's like current models. And I'm just like, he didn't get more computed, but it still saved him like a shit ton more time. Like, just like, draw that forward, it's like, that's a crazy implication or crazy trend, right?

• 54:58Speaker 02

  I mean, I guess, have we seen, like, I'm skeptical of the claims that we have actually seen that much of an acceleration in the process of R and D. Like, these claims seem to me like they're not borne out by the actual data I'm seeing. So, I'm not sure how much to trust them.

• 55:17Dwarkesh Patel?

  I mean, on the general intuition, that cognitive effort alone can give you a lot of AI progress.

• 55:22Speaker 02

  Right.

• 55:22Dwarkesh Patel?

  We've had, it seems like a big, important thing the labs do is this like science of deep learning. Like, scaling laws is just, I mean, like, ultimately knitted out an experiment, but the experiment was motivated by cognitive effort.

• 55:35Speaker 02

  So, for what it's worth, when you say that A and B are complementary, you're not saying like, just as you can't get a lot of progress, like, just as A can't model like you, B can also model like you.

• 55:44Dwarkesh Patel?

  Yeah.

• 55:44Speaker 02

  So, when you say you need a computer and experiments and data, but you also need cognitive effort, like, that doesn't mean the lab who has the most compute is gonna win. Right, like that's a very simple point, like either one can be the bottleneck. I mean, if you just have a really dysfunctional culture and you can't, you don't actually prioritize using your compute very well and you just waste it, well then you're not going to make a lot of progress, right? So like, it's, it doesn't contradict the picture that someone with a much better vision, a much better team, much better prioritization can make better use of their compute if someone else was just bottlenecked heavily on that part of the equation. The question here is, once you get these automated AI researchers and you start this

• 56:26Speaker 02

  software singularity, your efficiency, software efficiency is going to improve by many orders of magnitude.

• 56:32Speaker 01

  Yep.

• 56:33Speaker 02

  While your compute stock, at least in sort of the short run, is going to remain fairly fixed. So how many ooms of improvement can you get before you become bottlenecked?

• 56:40Speaker 01

  Yeah.

• 56:41Speaker 02

  By the second part of the equation?

• 56:43Speaker 01

  Right.

• 56:44Speaker 02

  And once you actually factor that in, like, how much progress should you expect? That's the kind of question. And I think people don't have, I, I think it's hard for people to have good intuitions about this because people usually don't run the experiments. So you don't get to see at a company level or at an industry level, what would have happened if the entire industry had three times less compute? Maybe as an individual, like, what would happen if you had three times less compute? You might have a better idea about that, but that's a very local experiment and you might be benefiting a lot from spillovers from other people who like actually have more compute. So because this experiment was never run, it's sort of hard to get direct evidence about the strength of complementarity.

• 57:27Dwarkesh Patel?

  What is your probability of, if we live in the world where we get AGI in 2027, that there is a software-only singularity?

• 57:34Speaker 01

  Quite high because...

• 57:35Speaker 02

  Because you're conditioning on us getting it in 2027.

• 57:38Speaker 01

  Then you're conditioning on like, compute not being very large. So it must be that like, you know, you get a bunch of software progress.

• 57:44Speaker 02

  Yeah.

• 57:45Dwarkesh Patel?

  Right, right. Like, you just have a bunch of leverage from algorithmic progress in that world.

• 57:49Speaker 02

  Yeah, that's right.

• 57:51Dwarkesh Patel?

  So then, maybe the, because I was thinking, these are independent questions.

• 57:54Speaker 01

  I think a call out that I want to make is, I know that some labs do have multiple pre-training teams, and they give people different amounts of resources for doing the training, and different amounts of cognitive effort, different size of teams. But none of that, I think, has been published. And I'd love to see the results of some of those experiments. I mean, I think even that won't update you very strongly, just because it is often just very inefficient to do this very imbalanced scaling of your factor inputs. And in order to really get an estimate of how strong these complementarities are, you need to observe these very imbalanced scale-ups.

• 58:34Dwarkesh Patel?

  Yeah.

• 58:34Speaker 01

  And so that rarely happens. And so I think the data that bears on this is just really quite poor. And then, you know, the intuitions that people have also don't seem clearly relevant to the thing that matters about what happens if you do this very imbalanced scaling, and where does this net out?

• 58:53Dwarkesh Patel?

  One question I have, which I like, that I think it would be really interesting if somebody can provide an example of, is maybe through history, there was some point at which, because of a war or some other kind of supply shock, you had to ramp up production or ramp up some key output that was, people really cared about, while for some weird, like, historical reason, many of the key inputs were not accessible to a ramp-up, but you could ramp up one key input. I'm talking in very abstract terms. What do you see I'm saying, right? You need to make more bombers, but like, you ran out of aluminum, and you just like, need to figure out something else to do. And how successful these efforts have been, or whether you just keep getting bottlenecked?

• 59:35Speaker 02

  Well, for, I think that is not quite the right way to do it. Because I think, like, if you're talking about materials, then I think there's a lot of sense in which different materials can be substitutable for one another in different ways. Like, you can use aluminum. I mean, aluminum is a great metal for making aircraft, because it's sort of light and durable and so on. But you can imagine that you make aircraft with, like, worse metals, and then it just takes more fuel, and it's like, less efficient to fly. So there's a sense in which you can compensate and just cost more.

• 1:00:05Speaker 02

  Like, I think it's much harder if you're talking about something like a complementarity between labor and capital, complementary tree between like remote work and in-person work, or like skilled or unskilled work. And like, there are input pairs for which I would expect it to be much more difficult. Or like, for example, you're looking at the complementarity between like, the quality of leadership of an army and its number of soldiers, right? I mean, there is some effect there, but like, if you just scale up, you just have excellent like leadership, but your army only has a hundred people. You're not like, you're not gonna get like very far.

• 1:00:39Dwarkesh Patel?

  King Leonidas and Thermopylae?

• 1:00:42Speaker 02

  Well, they lost, right?

• 1:00:46Dwarkesh Patel?

  It would be funny if we're building models of software in singularity and we're like, what exactly happened in Thermopylae? You know, it's like somehow relevant.

• 1:00:54Speaker 02

  I mean, I can actually talk about that, but we probably shouldn't.

• 1:00:57Dwarkesh Patel?

  Okay, sure. By the way, so for the audience should know, my most popular guest by far is Sarah Payne. And not only is she my most popular guest, she's my most popular foreguest. Because like, all four of those episodes that I've done with her are like, from a viewer minute adjusted basis, I host the Sarah Payne podcast, where I occasionally talk about AI.

• 1:01:19Dwarkesh Patel?

  And anyways, we did this three-part lecture series. Where we're talking about, like, one of them was about India-Pakistan wars through history. One of them was about, was it the Japan, like Japanese culture before World War II? The third one was about the Chinese Civil War. And for all of them, my tutor, my history tutor, was Egge. And it just like, why does he know so much about like fucking random like 20th century conflicts?

• 1:01:50Dwarkesh Patel?

  But he did, and he suggested a bunch of the good questions I asked her. We'll get into that. Actually, what's going on there?

• 1:01:56Speaker 02

  I don't know. I mean, I don't really have a good question. I think it's interesting. I mean, I read a bunch of stuff, but it's a kind of boring answer, like, I don't know. Imagine you ask like a, like a top AI researcher, like, what's going on? Like, how are you so good? And then they will probably give you like a boring answer, like, I don't know, like, I did this.

• 1:02:12Dwarkesh Patel?

  That itself is interesting, that often these kinds of questions elicit boring answers. Like, it tells you the nature of, about like the nature of the skill. How'd you find them?

• 1:02:21Speaker 01

  We, we connected on like some, on a Discord for Metacius, which is this forecasting platform. And I was I was a graduate student at Cambridge at the time doing research in economics. And I was having conversations with my peers there. And I was occasionally having conversations with Egge. And I was like, this guy knows a lot more about economics. And he's, at the time, he was a computer science undergrad in Ankara. And he knows more about economics and about, you know, these big trends in economic growth and economic history than almost any of my peers at university. And so like, what the hell is up with that?

• 1:03:05Speaker 01

  You know, so, so we started like having frequent collaborations and ended up hiring Egge for Epoch because it, you know, clearly, clearly makes sense for him to work on, on these types of questions.

• 1:03:17Dwarkesh Patel?

  And it seems like at Epoch, you were just collecting this like a group of internet misfits and weirdos.

• 1:03:23Speaker 01

  Yeah, that's right.

• 1:03:24Dwarkesh Patel?

  How, how, how did you, so, but how did you start Epoch? And then how did you accomplish this?

• 1:03:27Speaker 01

  Yeah, so I was, I, I was at MIT doing, doing my research. And I was pretty unhappy with the bureaucracy there, where it was very hard for me to scale projects up, hire people. And I was pretty excited about a bunch of work that my P I wasn't excited about because it's maybe hard to publish or isn't, it doesn't confer the same prestige. And so, you know, I was chatting with Chaim Seviah, one of the co-founders, and we just collaborated on projects and, and then thought we should just start our own org because we can just hire people and work on the projects we were excited about. And then I just, you know, hired a bunch of the, the insightful misfits that like-

• 1:04:12Dwarkesh Patel?

  But did you like, was, was the thesis like, oh, there's a bunch of underutilized internet misfits and therefore like this org was successful? Or you started the org and then you're like-

• 1:04:20Speaker 01

  I, I think it's more the latter. So it was more like, we can make a bunch of progress because clearly like academia and industry is kind of dropping the ball on a bunch of important questions that academia is, is unable to publish interesting papers on. Industry is not really focused on producing useful insights. And, and so it seemed like very good for us to just do that. And also the timing was very good. So we started just before, you know, ChatGPT and we wanted to have much more grounded discussions of the future of AI.

• 1:04:51Dwarkesh Patel?

  Yeah.

• 1:04:51Speaker 01

  And I was frustrated with the quality of discussion that was, was happening in, on the, on the internet about the future of AI. And- I mean, to some extent, or to a very large extent, I still am.

• 1:05:03Dwarkesh Patel?

  Yeah.

• 1:05:03Speaker 01

  And that's like a large part of what, you know, motivates me to do this. It's just like born out of frustration with bad thinking and arguments about where AI is going to go.

• 1:05:15Dwarkesh Patel?

  The part about my job that I enjoy the least is the post-production. I have to re-watch the episode multiple times, make all these difficult judgment calls. And I've been trying to automate all this work with LLM scripts. And I found that Google's Gemini 2.5 Pro is the best model I've tried for these tools. So much of the post-production requires understanding, the delivery, the context, all these other things that you don't get from a text-only transcript. Unlike other models I've tested, I can actually just shove in the four-hour raw audio file into Gemini because of its multimodal capabilities. And it can generate super-high-quality transcripts, identify great snippets for clips, a bunch more other things. I've actually made a repo with all these tools and I've linked to the GitHub in the description below in case you might find it helpful.

• 1:05:59Dwarkesh Patel?

  I actually used 2.5 Pro in order to write the code for these scripts. It's actually quite interesting to read its reasoning traces as it's thinking through your gnarly list of requests and tasks. Gemini 2.5 Pro is now available in preview with higher rate limits. You can try it out at ai studio dot google dot com. Thanks to Google for sponsoring this episode, and now back to Egge and Tomei. Okay, so let me ask you about this. I can poke you from the... So just to set the scene for the audience. We're going to talk about the possibility of this explosive economic growth and like greater than 30% economic growth rates. So I want to poke you both from a perspective of

• 1:06:43Dwarkesh Patel?

  maybe suggesting that this isn't aggressive enough in the right kind of way, because it's maybe it's too broad. And then I'll poke you in the from the perspective of like the more normal perspective that, hey, this is fucking crazy. I imagine it would be difficult for you to do the second thing.

• 1:06:59Dwarkesh Patel?

  No, I mean, like, I think it might be fucking crazy. Let's see. The big question I have about this broad automation. Like, I get what you're saying about the industrial revolution, but in this case, we can just make this like argument that you get this intelligence. And then what you do next is you go like to the desert and you build this like Shenzhen of robot factories, which are building more robot factories, which are building. If you need to do experiments and you build bio labs and you build chemistry labs and whatever.

• 1:07:29Speaker 02

  If you build Shenzhen in the desert, I agree, that looks much more plausible than a software on the singularity.

• 1:07:35Dwarkesh Patel?

  But why, but the way you're framing it, it sounds like McDonald's and Home Depot and fucking whatever are growing at 30% a year as well and not just like, is it the you know, the aliens level view of the economy is that like, there's a robot economy in the desert that's growing at 10,000% a year? And everything else is the same old, same old? Or is it like?

• 1:07:57Speaker 02

  No, I mean, there is a question about what would be possible or physically possible? And what would be the thing that would actually be efficient, right?

• 1:08:07Dwarkesh Patel?

  Yep.

• 1:08:07Speaker 02

  So it might be, it might be the case. And again, once you're scaling up the hardware part of the equation as well as the software part. Then I think the case for this feedback loop gets a lot stronger. If you scale up data collection as well, I think it gets even stronger, like real world data collection by deployment and so on. But building Shenzhen in a desert, that's a pretty like, like if you look, if you think about the

• 1:08:32Speaker 02

  pipeline, so so far we have relied, first of all, we're relying on the entire semiconductor supply chain. That industry depends on tons of inputs and materials and whatever it gets from probably tons of random places in the world. And creating that infrastructure, like doubling or tripling, whatever, that infrastructure, like the entire thing, that's very hard work, right? So probably you couldn't even do it, even if you just have Shenzhen in a desert. Like, that will be even more expensive than that. On top of that, so far we have been drawing heavily on the fact that we have built up this huge stock of data over the past 30 years or something on the internet. Like, imagine you were trying to train a steady art model, but you only have like 100 billion tokens, right?

• 1:09:17Speaker 02

  To train on, that would be very difficult. So in a certain sense, we, our entire economy.

• 1:09:24Speaker 02

  Has produced this huge amount of data on the internet that we are now using to train the models. It's plausible that in the future, when you need to get new competencies added to these systems, the most efficient way to do that will be to try to leverage similar kind of modalities of data, which will also require this like, you would want to deploy the systems broadly, because that's going to give you more data. And maybe you can do the, maybe you can get where you want to be without that, but it would just be less efficient if you're starting from scratch compared to if you're collecting a lot of data. Like, I think this is actually a motivation for why labs want their L L Ms to be deployed widely. Because like, sometimes when you talk to Chat G P T, it's going to give you two responses and it's going to say, well, which one was good?

• 1:10:09Speaker 02

  Or like, it's going to give you one response and it's going to ask you, was this good or not? Well, why are they doing that? Right? That's a way in which they are getting user data through this extremely broad deployment. So I think you should just imagine that thing to be, continue to be efficient and continue to increase in the future. Because it just makes sense. And then there's a separate question of, well, suppose you didn't do any of that. Like, suppose you just try to imagine the most rudimentary, the most narrowest possible kind of infrastructure build out and deployment that would be sufficient to get this positive feedback loop that leads to like much more efficient A I s. I agree that loop could in principle be much smaller than the entire world.

• 1:10:53Speaker 02

  I think it couldn't, probably couldn't be as small as transatlantic desert, but it could be much smaller than the entire world. But then there's a separate question of, would you actually do that? Would that be efficient? I think some people have the intuition that there are just these like extremely strong constraints, maybe regulatory constraints, maybe social, political constraints to doing, like, doing this broad deployment. Yeah. They just think it's going to be very hard. So I think that's part of the reason why they imagine these like more narrow scenarios where they think it's going to be easier. But I think that's like, I think that's overstated. I think people's intuitions for like how hard this kind of deployment is comes from cases where the deployment of the technology wouldn't be like that valuable.

• 1:11:38Speaker 02

  So it might come from housing, like we have a lot of regulations on housing. Maybe it comes from nuclear power, maybe it comes from supersonic flights. I mean, those are all technologies that would be useful if they were like maybe less regulated. But they wouldn't like double economic output.

• 1:11:53Speaker 01

  I think the core point here is just the value of AI automation and deployment is just extremely large, even just for workers, at least the ones that, you know, at least after, you know, finding, you know, there might be some kind of displacement and there might be some transition that you need to do in order to find a job that that works for you. But but otherwise the wages could still be very high for for for a while at least. And and on top of that, the gains from owning capital might be very enormous. And in fact, a large share of the US population would benefit from this thing. They benefit, they own housing, they own, they have 401ks.

• 1:12:34Speaker 01

  Those would do enormously better when you have this process of broad automation and AI deployment. And so I I think there's, there could just be a very deep support for some of this, even when it's like totally changing the nature of labor markets and, you know, the skills and occupations that are in demand.

• 1:12:55Speaker 02

  So I would just say it's like complicated. I I think the like what the political reaction to it will be when this starts actually happening. I think like the easy thing to say is that, yeah, like this will become like a big issue. And then it will be maybe controversial or something. But like, what is the actual nature of the reaction in different countries? I think that's kind of hard to forecast. Like, I think the default view is like, well, people are going to become unemployed. So it will just be very unpopular. I think that's like very far from obvious. And I just expect heterogeneity, how different countries respond, and some of them are gonna be more liberal about this and gonna allow broader deployments, and those countries probably end up doing better. Just like during the industrial revolution, some countries were just ahead of others. I mean, eventually, almost the entire world adopted the sort of norms and

• 1:13:41Speaker 02

  culture and values of the industrial revolution in various ways.

• 1:13:43Speaker 01

  And actually, you say they might be more liberal about it, but they might actually like be less liberal. They might be less liberal in many ways. And

• 1:13:52Speaker 02

  Right.

• 1:13:52Speaker 01

  In fact, that might be like more functional in this world in which you have broad AI deployment. Like we might adopt the kind of values and norms that get developed in, say, you know, the UAE or something, which is like maybe focused a lot more on making an environment that is very conducive for AI deployment. And we might start, you know, emulating and adopting various norms like that. And they might not be kind of classical liberal norms, but norms that are just more conducive to AI being functional and producing a lot of value.

• 1:14:26Speaker 02

  This is not meant to be like a strong prediction. This is just an illustrative.

• 1:14:30Speaker 01

  Yeah, yeah, yeah.

• 1:14:31Speaker 02

  Like, it might just be that, like, like the freedom to deploy AI in the economy and build out lots of physical things at scale. Maybe that ends up being more important in the future. Maybe that is still missing something. Maybe there's other things that are also important. But like, I was just, like, the generic prediction that you should expect variance, and some countries do better than others. I think that's much easier to predict than like the specific countries that end up doing better.

• 1:14:54Dwarkesh Patel?

  Yeah, or the norms that that country will.

• 1:14:56Speaker 02

  That's right.

• 1:14:57Dwarkesh Patel?

  So, I mean, one thing I'm confused about is if you look at the world of today versus the world of 1750, the big differences is just like, like, we've got crazy tech that they didn't have back then. We've got these cameras, and we've got these screens, and we've got rockets and so forth. And that just seems like the result of technological growth and R and D and so forth.

• 1:15:22Speaker 02

  Capital accumulation.

• 1:15:23Dwarkesh Patel?

  Well, explain that to me, because. You're just talking about like this infrastructure build out and blah, blah, blah. I'm like, but why won't they just like fucking invented the kinds of shit that humans would have been invented by 2020, um, uh, 2020, 250.

• 1:15:37Speaker 02

  Producing this stuff takes a lot of infrastructure build up. It's not.

• 1:15:40Dwarkesh Patel?

  But, but that, that infrastructure is built out once you make the technology, right?

• 1:15:44Speaker 01

  I don't, I don't think that's, that's right. Like I, there isn't this like temporal, like difference where it's first you do the invention and then like often there's this interplay between the actual capital buildup and the innovation.

• 1:15:57Speaker 02

  And learning curves are about this, right? Fundamentally, like what has driven the increase in the efficiency of solar panels over the past 20, 30 years?

• 1:16:05Speaker 01

  Like it isn't just like people had the idea of like 2025 solar panels. Like no one ever had like, nobody 20 years ago had like the sketch for the 2025 solar panel. That, that's not, it's this kind of interplay between having ideas, building, learning, producing and.

• 1:16:23Speaker 02

  Other complementary inputs also becoming more efficient at the same time. Like you might get better materials. Like for example, uh, the, like the fact that like aluminum becomes something that's like, for example, the fact that smelting processes got a lot better towards the end of the 19th century. So it became a lot easier to work with metal. Maybe that was a crucial reason why aircraft technology later become, like became more popular. So it's not like someone came up with the idea of, oh, like you can just like use something that just has wings and has a lot of thrust and then that might be able to fly. Like, like that basic idea is not that difficult. But then that, well, how do you make it like actually a viable thing?

• 1:17:02Dwarkesh Patel?

  That's right.

• 1:17:03Speaker 02

  Well, that's much more difficult.

• 1:17:03Dwarkesh Patel?

  Have you seen the meme where, um, uh, two beavers are talking to each other and they're looking at the Hoover Dam and one of them's like, well, I didn't build that, but it's based on an idea of mine.

• 1:17:17Dwarkesh Patel?

  The point you're making is that when this invention focused.

• 1:17:22Dwarkesh Patel?

  Look on tech history underplays the work that goes into making specific innovations practicable and to deploy them widely.

• 1:17:33Speaker 02

  It's just hard, I think. Like, it's just hard to even, like, suppose you wanted to write a history of this, right? Like, you want to write a history of how was the, like, light bulb developed or something. It's just really hard because to understand why specific things happen at specific times, you probably need to understand so much about the economic conditions of the time. Like, for example, Edison spent a ton of time experimenting with, like, different filaments to be used in a light bulb. Like, the basic idea is very simple. You make something hot and it glows. But then, like, what actually, what filament actually works well for that in a product? What is, like, durable? What, like, has the sort of highest ratio of, like, light output versus, like, heat

• 1:18:15Speaker 02

  and, like, so that you have less waste, that it's more efficient? And then, even after you have the product, then you're faced with a problem, well, I mean, it's like 1880 or something, and then US homes don't have electricity, so then nobody can use it. So now you have to build power plants and build power lines to the houses so that people have electricity in their homes, so that they can actually use this new light bulb that you created. So he did that, and, but then, people present it as if it's, like, okay, he just came up with the idea, like, it's a light bulb.

• 1:18:45Dwarkesh Patel?

  Well, I guess what people would say is, like, you're right about how technology would progress if we were humans deploying for the human world, but what you're not counting is, there's just going to be this, like, AI economy where maybe they need to do this kind of innovation and learning by doing when they're figuring out, how, um, I want to make more robots because they're helpful, and so, like, we got, we're going to build more robot factories, we'll learn, and then we'll make better robots or whatever. But, like, That is just like a geographically, that is a small part of the world that's happening in. Or you understand what I'm saying, like it's not like, and then like then they walk in your building and then you do a business transaction with Lunar Society Podcast LLC and then you know.

• 1:19:30Speaker 02

  Yeah, I mean, for what it's worth, like if you look at the total surface area of the world, it might well be the case that the place that initially experiences this very fast growth is like a small percentage of the surface area of the world.

• 1:19:42Dwarkesh Patel?

  Right.

• 1:19:43Speaker 02

  Because I think that was the same for the industrial revolution, it was not different.

• 1:19:46Dwarkesh Patel?

  Yeah, so but then like, I'm just like, what concretely does this explosive growth look like? If I look at this heat map of growth rates on the globe, is there's gonna be like one area is like blinding hot and that's a, you know, that's like the desert factories with all these experiments and like.

• 1:20:03Speaker 02

  Yeah, so I would say our idea is that it's gonna be broader than that, but probably initially, so eventually it would be probably most of the world. But as I said, because of this heterogeneity, because I think some countries are gonna be faster in adoption than others, maybe some cities will be more faster option than others. And that will mean that there is differentials and some countries might like have much faster growth than other countries. But I would expect that at a like at a jurisdiction level, it will be like more homogeneous. So for example, like I expect the primary obstacles to come from things like like regulation. And so I would just imagine it being like more delineated by regulatory jurisdiction boundaries than anything else.

• 1:20:48Dwarkesh Patel?

  Got it. So you may be right that this infrastructure build out and capital deepening and whatever is necessary for a technology to become practical, but.

• 1:20:57Speaker 02

  Or even to be discovered, like there's an aspect of it that where you discover certain things by scaling up, learning by doing.

• 1:21:03Dwarkesh Patel?

  Yeah.

• 1:21:04Speaker 02

  Right, learning curve. And there's this separate aspect where you get to like, suppose that you become wealthier. Well, you can invest that increased wealth in, like, yeah, you use it to accumulate more capital, but you also can invest it in R and D and other ways.

• 1:21:21Speaker 01

  You get Einstein out of the patent office, like you need some amount of resources for that to make sense. And you need the economy to be of a certain scale. You also need demand for the product you're building. So like, you know, you could have the idea, but if the economy is just too small that, you know, there isn't enough demand for you to be specializing in producing the semiconductor, whatever, because there isn't enough demand for it, then it doesn't make sense. So you, you want the economy, like a much larger scale than economy is useful in, in very many ways, in delivering complementary innovations and the discoveries happening through serendipity, producing, like having there be consumers that would actually pay enough for you to recover your fixed costs of doing all the experimentation and the invention.

• 1:22:05Speaker 01

  You need the supply chains to exist to deliver the germanium crystals that you need to do, grow in order to come up with the semiconductor. You need a, a large labor force to be able to help you do all the experiments and so on.

• 1:22:19Dwarkesh Patel?

  I, I think the, the, the point you're illustrating is like, look, could you have just figured out that there was a big bang by first principles reasoning? Maybe, but what actually happened is we had World War II and we discovered radio, um, radio communications in order to fight and effectively communicate during the war. And then that technology helped us build radio telescopes and then we discovered cosmic microwave background and then we had to come up with an explanation for cosmic. And then, then we discovered like the big bang as a result of like World War II.

• 1:22:46Speaker 01

  People under emphasize that like giant effort that goes into this role, this kind of buildup of all the relevant capital and all the relevant supply chains and the technology. I mean, earlier you were making a similar comment when you were saying, oh, you know, reasoning models, actually in hindsight, they look pretty simple, but then you're kind of ignoring this giant kind of upgrading of the technology stack that happened, you know, that took five to 10 years.

• 1:23:13Dwarkesh Patel?

  Yeah.

• 1:23:14Speaker 01

  Prior to that. And, and so I think people just under emphasize, um, the support that is had from the overall upgrading of your technology, of the supply chains, of various sectors that are important for that. And people focus on just specific individuals of like, you know, Einstein had this genius insight and like, you know, he was. The kind of very pivotal thing in the causal chain that resulted in these discoveries, or Newton was just extremely important for discovering calculus without thinking about, well, there was this kind of all these other factors that produced lenses, that produced telescopes, that got the right data and that made people ask questions about dynamics

• 1:23:57Speaker 01

  and so on that motivated some of these questions. And those are also extremely important for science, scientific and technological innovation.

• 1:24:06Dwarkesh Patel?

  Yeah. You know conquest, what is it? One of conquest laws is like the more you understand about a topic, the more conservative you become about that topic. And so there might be like a similar law here where like the more you understand about an industry, the more sort of like, I obviously I'm just like a commentator or whatever, or a podcaster. But I understand AI better than any other industry I understand. And there I have the sense from talking to people like you that, oh, so much went into getting AI to the point where it is today. Whereas when I talk to journalists about AI, they're like, okay, who is the, who is a crucial person we need to cover? And they're like, should we get in touch with Jeffrey Hinton? Should we get in touch with Ilya? And I just have this

• 1:24:47Dwarkesh Patel?

  like, like you're kind of missing the picture. It's not, but then you should have that same attitude towards things you, maybe it's the more similar phenomenon is Joe Manamnesia. We should have a similar attitude towards other industries that it's like much more complicated.

• 1:25:00Speaker 02

  Right. I mean, it's Robin Hanson has this abstraction of like seeing things in near mode versus far mode, right? And I think if you don't know a lot about the topic, then you see it sort of in far mode and you sort of simplify and

• 1:25:10Dwarkesh Patel?

  That's right.

• 1:25:11Speaker 02

  Things you know, you see a lot more detail. Like in general, I think the thing I would say, and the reason I also believe that just like abstract reasoning and like sort of deductive reasoning, or even Bayesian reasoning by itself is not like sufficient or like is not as powerful as many other people think, is because I think there's just this like enormous amount of like richness and detail in the real world that like you just can't like reason about it. Like you need to see it. And obviously, that is not an obstacle to AI being incredibly transformative. Because as I said, you can scale your data collection, you can scale experiments. You do both in the AI industry itself and just more broadly in the economy. So you just discover more things.

• 1:25:54Speaker 02

  More economic activity means we have more exposed surface area to have more discoveries. Like, all of these are things that have happened in our past. Right. There's no reason that they couldn't speed up. Like, the fundamental thing is that there's no reason fundamentally why economic growth can't be much faster than it is today. Like, it's probably about as fast right now, just because humans are such an important bottleneck. They both supply the labor. They play crucial roles in the process of discovery of various kinds of productivity growth. There's just strong complementarity to some extent with capital, that you can't substitute machines and so on for humans very well. So, the growth of the economy and growth of productivity just ends up being bottlenecked by the growth of human population.

• 1:26:36Dwarkesh Patel?

  Publicly available data is running out. So, major AI labs like Meta, Google DeepMind, and OpenAI partner with Scale to push the boundaries of what's possible. Through Scale's data foundry, major labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. As AI races forward, we must also strengthen human sovereignty. Scale's research team, SEAL, provides practical AI safety frameworks, evaluates frontier AI system safety via public leaderboards, and creates foundations for integrating advanced AI into society. Most recently, in collaboration with the Center for AI Safety, Scale published "Humanity's

• 1:27:16Dwarkesh Patel?

  Last Exam," a groundbreaking new AI benchmark evaluating expert-level reasoning and knowledge capabilities across a wide range of fields. If you're an AI researcher or engineer, and you want to learn more about how Scale's data foundry and research lab can help you go beyond the current frontier of capabilities, go to scale.com/dwarkesh. Let me ask a tangential question. What has happened in China over the last 50 years? Would you describe that as, like, in principle, the same kind of explosive growth that you expect from AI? Because there's like a lot of labor that makes the marginal product of capital really high, which allows you to have, like, 10% plus economic growth rates. Is that basically in principle what you're from AI?

• 1:28:01Speaker 02

  So I would say in some ways it's similar, in some ways it's not. The way, probably the most important way in which it's not similar is that in China you see this relative, like you see a massive amount of capital accumulation.

• 1:28:14Dwarkesh Patel?

  Yeah.

• 1:28:14Speaker 02

  A substantial amount of adoption of new technologies and probably also human capital accumulation to some extent. And but you're not seeing a huge scale up in the labor force, the labor force. While for AI you should expect to see a scale up in the labor force as well, not in the human workforce, but in the AI workforce.

• 1:28:34Dwarkesh Patel?

  I think you did kind of like, maybe not consecutive increases in the labor force increase, but like you did.

• 1:28:40Speaker 01

  The key thing here is just the simultaneous scaling of both these things.

• 1:28:43Dwarkesh Patel?

  Got it.

• 1:28:44Speaker 01

  And so like you might ask the question of, isn't it like basically half of what's going to happen with AI that you scale up, you know, capital accumulation in China? But actually, that's really not like if you get both of these things to scale, that that gives you just much faster growth and a very different picture.

• 1:29:03Speaker 02

  But at the same time, if you're just asking like, what would thirty percent growth per year like look like? Like in terms of, like if you're just gonna have an intuition for how transformative that would be in concrete terms, then I think looking at China is not such a bad case, like you can, especially in the two thousands or maybe late nineties. Like that gives you a good, that's even slower than what we're forecasting.

• 1:29:23Speaker 01

  Right. I think also looking at the industrial revolution is pretty good.

• 1:29:26Speaker 02

  Well, national revolution is very slow, so.

• 1:29:28Speaker 01

  So but just in terms of the types of, the kind of, the margins along which we made

• 1:29:36Speaker 02

  Oh, yeah.

• 1:29:36Speaker 01

  progress in terms of products. So what what didn't happen, the thing that didn't happen during the industrial revolution is we just produced a lot more of things that people were producing prior to the industrial revolution, like producing a lot more crops and maybe a lot more kind of. Pre-industrial revolution style houses or whatever on farms. Instead, what we got is along pretty much every main sector of the economy. We just had many different products that are totally different from what was being consumed prior to that. So in transportation, in food.

• 1:30:12Speaker 02

  In healthcare, it's a very big deal.

• 1:30:14Dwarkesh Patel?

  Antibiotics. So another question, because I'm not sure I understand like how you're defining the learning by doing versus explicit R and D. Because there's like the way for taxes that companies say what they call R and D. But then there's like the intuitive understanding of R and D. So if you think about how AI is boosting TFP.

• 1:30:33Speaker 02

  Mm hmm.

• 1:30:33Dwarkesh Patel?

  You could say that like right now, if you just had replaced the TSMC process engineers with AIs. And they're finding different ways in which to improve that process and like, you know, improve efficiencies, improve yield.

• 1:30:47Speaker 02

  Right.

• 1:30:48Dwarkesh Patel?

  I would kind of call that R and D. On the other hand, you emphasize this, the other part of TFP, which is like better management and.

• 1:30:56Speaker 02

  Learning by doing.

• 1:30:57Dwarkesh Patel?

  Yeah, that kind of stuff.

• 1:30:58Speaker 02

  Learning by doing could be, you could.

• 1:31:00Dwarkesh Patel?

  I mean, like, how much, how much O M F R you're gonna, like, you're gonna get to the, you're gonna get to like the fucking Dyson sphere by better management?

• 1:31:05Speaker 02

  Well, not just, but, but that's not the argument, right? Like, the point is that there are all these different things that, like, some of them are maybe more complementary than others. The point is not that you can get to a Dyson sphere by just scaling labor and capital. Like, that's not the point. Like, you need to scale everything at once. So just as you can't get to a Dyson sphere by just scaling labor and capital, you also can't get to it by just scaling TFP. That doesn't work.

• 1:31:30Speaker 01

  I think there's a very important distinction between what is necessary, you know, to scale to get to this, you know, Dyson sphere world and what is important. Like, in some sense, producing food is necessary. But, but of course, producing food doesn't get you to a Dyson sphere, right? So I think R and D is necessary, but on its own, isn't sufficient. And scaling up the economy is also necessary. On its own, it's not sufficient. And then you can ask the question, what is the relative importance of each?

• 1:32:00Speaker 02

  Yeah, so I think our view here is like very much the same. We're like, like. It is very connected to our view about the software R and D thing where we're just saying like there are these bottlenecks. So you need to scale everything at once. Like this is just their general view. But I think people like misunderstand us sometimes as saying that like R and D is not important. Like, no, that's not what we what we're saying. We're saying it is important. It is less important in relative terms than some other things, none of which are by themselves sufficient to enable this growth. So the question is like, how do you do the credit attribution? I mean, one way in economics is standard to do that is to look at the elasticities of output to the different factors.

• 1:32:39Speaker 02

  Like, like capital is less important than labor because the output elasticity of like labor elasticity output is like point six. Well, for capitals like point three, but neither are by themselves sufficient. Like, if you just scaled one of them and the other remained fixed, then neither would be sufficient to indefinitely scale output.

• 1:33:00Dwarkesh Patel?

  One question that Daniel posed to me is like, because I made this perspective about everything being interconnected. When you were talking about like, another example people often bring up is what would it take to build the iPhone in the year 1000? And it's like unclear how you could actually do that without just like replicating every like intermediate technology or most intermediate technologies.

• 1:33:23Dwarkesh Patel?

  And then he made the point like, okay, fine, whatever. Nanobots, like nanobots, there's not a crux here. The crux, at least to the thing he cares about, which is human control, is just by when can the robot economy, whether or the AI economy, whether it's a result of capital deepening or whether it's a result of R and D, by when will they have just like the robots? And they've they have more sort of like human to live, physical power.

• 1:33:50Speaker 02

  Right, but he's imagining like a separate thing called the AI economy. That why would you imagine that? That seems like a, I think it's probably downstream of his views about the software only singularity. But again, like those are views that we don't share. So.

• 1:34:03Speaker 01

  Like, it's just much more efficient for AI to operate in our economy and benefit from the existing supply chains and existing markets, rather than like set up shop on some island somewhere and do its own thing.

• 1:34:15Speaker 02

  Yeah. And then he's sounding clear, like, for example, people might have the intuition, I brought this up before, like the distinction between what is the minimum possible amount of build out that would be necessary to get this. Feedback loop up and running and what will be the most efficient way to do it? Which are not the same question.

• 1:34:31Dwarkesh Patel?

  Right.

• 1:34:32Speaker 02

  But then people have this view that, oh, like the most efficient thing in principle, we can't do that because like.

• 1:34:37Dwarkesh Patel?

  I think the example he might give is when the conquistadors arrived to the New World or when the East India Trading Company arrived to India. They did integrate into the existing economy. In many cases, like it depends on how you define integrate, but like the Spanish relied heavily on New World labor in order to do silver mining and whatever.

• 1:35:00Speaker 02

  That's right.

• 1:35:01Dwarkesh Patel?

  East India Trading Company was like, it was just a ratio of British people to Indian people was just like not that high, right? So they just had to rely on the existing labor force. But they were still able to take over because of, I don't know what the analogous thing here is, but you see what I'm saying? And so he's concerned about by when will they, even if they're ordering components off of Alibaba or whatever, by, and sorry, I'm being, try to, you see what I'm saying? Like even if they're integrating those supply chains, by when are they in a position where because this like part of the economy has been growing much faster, they can, they could like take over the government or.

• 1:35:40Speaker 02

  If they wanted to.

• 1:35:41Dwarkesh Patel?

  That's right, yeah.

• 1:35:42Speaker 02

  Okay, so I think that is like, eventually you expect the AI systems to be driving most of the economy and I don't think that's a, like unless there is, there are some very strange coincidences where like humans are able to like somehow uplift themselves and like, like able to become competitive with the AIs by like stopping being biological humans or whatever, which seems very unlikely early on, then AI is just going to be much more powerful. Okay. So, and I agree that in that world, if the AI is just somehow coordinated and decide, okay, like we should just like take over or something, like they just, like just somehow coordinated to have that goal, then they could probably do it. Okay.

• 1:36:26Speaker 02

  But, okay, that's also probably true in our world. Like in our world, if the, I don't know, if the US wanted to invade Central Island, then probably they could do it. I don't think anyone could stop them. But like, what does that actually mean? I mean, there, there is this dramatic power imbalance. But that doesn't mean, like, that doesn't tell you what's gonna happen, right? Like, why doesn't the US just invade Guatemala or something? Like, why don't they do that? Seems like they could easily do it.

• 1:36:53Dwarkesh Patel?

  Because the value to the US of-

• 1:36:55Speaker 02

  Not that high, right?

• 1:36:56Dwarkesh Patel?

  Yeah, but I, like, there's rea- so basically, it just seems, I agree with, that might be true for AIs, because like, they, most of the shit is in space, and once you get like, you wanna do the capital deepening on Mars, and like, the surface area of the sun, instead of, um, instead of like, uh-

• 1:37:11Speaker 02

  I mean, I think before-

• 1:37:12Dwarkesh Patel?

  Instead of New York City, but-

• 1:37:13Speaker 02

  So it's deeper than that, like, there are also the, there's also the fact that if the AIs are gonna be integrated into our economy, so basically, they start out as like a smaller part of our economy or our workforce, and then over time, they grow, and over time, they might, like, they become the vast majority of the actual sort of work power in the economy. But they are, uh, they are growing in this existing framework, where we have like, norms and rules for better coordination, and then undermining those things has a cost. So if, like, getting the things that is making the humans sort of wealthier than they used to be before, and more comfortable, like, yeah, like, you would probably be better off if you could just take that from them.

• 1:37:56Speaker 02

  But, uh, the benefit to you, if you already are getting almost all of the income in the economy, will be fairly small.

• 1:38:03Dwarkesh Patel?

  I mean, I, I feel like the sentient nihilist thing is not the, there's one reference class that includes that, but historically, there's like a huge reference class that includes, East India Trading Company could have just kept trading with the Mughals. They just like took over, right? Like, they could have like kept trading with the, um, the, uh, the, like, the fifty different nation-states in pre-colonial India, but, yeah.

• 1:38:22Speaker 02

  That's right, I mean, that's what they were initially doing, and then, whatever, like, I'm not gonna-

• 1:38:27Dwarkesh Patel?

  But, but, like, that is the reference class of, like, you took, yeah.

• 1:38:30Speaker 02

  I agree, so like, i- i- if the question is, um- Like if they are entirely, if they have some totally different values and then they represent most of the economy, then like would they take over? I still don't know because I'm not sure to what extent the class of all AIs is like a natural like class. Like it's sort of like, why don't, why don't the young people in the economy like?

• 1:38:54Dwarkesh Patel?

  I mean, I think it's a closer. So I agree that sometimes these kinds of class arguments are misused. For example, when Marxists are like, why don't like this like class uprising against the others. Dana made the interesting argument that if you look at the history of the conquistadors, when Cortez was making his way through the new world, he had to actually go back and fight off a Spanish fleet that had been sent to arrest him.

• 1:39:17Speaker 02

  That's right.

• 1:39:17Dwarkesh Patel?

  And then go back, right? So you can have this fight within this like conquering AIs and then that's still, that's out in the, the Native Americans getting disempowered. But with AIs in particular, they're just like copies of each other. In many other ways, they might, they just have a lot, they have lower transaction costs when they trade with each other or interact with each other. There's, there's other reasons to expect them to just be more compatible coordinating with each other than coordinating with the human world.

• 1:39:48Speaker 02

  Sure. But then like, I guess I, I'm still not seeing the, I mean, I, if the question is just that, is it possible for that to happen, which is like a weaker claim, then yeah, I mean, it seems possible. But there are, I think, a lot of arguments just pushing against it. Probably actually the biggest one is the fact that AI preferences are just not like, just look at the AIs we have today. Like, can you, can you imagine them doing that? I think people just don't put a lot of weight on that because they think, once we have enough optimization pressure and once they become super intelligent, they're just going to become misaligned. But I just don't see the evidence for that.

• 1:40:22Dwarkesh Patel?

  No, I mean, I, I think that's actually, like, I agree, there's some evidence that they're like good boys.

• 1:40:27Speaker 02

  Wait, no, there's more than some evidence.

• 1:40:29Dwarkesh Patel?

  No, no, but there's also some evidence again, like there's a new OpenAI paper where in Chain of Thought, like reward hacking is such a strong basin that if you're like, hey, let's go solve this coding problem, in the Chain of Thought, they'll just Be like, okay, let's hack this and then like figure out how to hack it.

• 1:40:47Speaker 02

  Yeah, so imagine that you gave students at a school a test and then the answer key was like.

• 1:40:52Dwarkesh Patel?

  Right, but but like the the reference class of humans does include like Cortez and the East Indian Trading Company.

• 1:40:57Speaker 02

  Sure.

• 1:40:58Speaker 01

  So I think one one issue here is that I think people are doing this very kind of partial equilibrium analysis or something where they're thinking about just this raw abilities of AI systems. In a world where AI systems are kind of dominant and and like human civilization has done very little in terms of integrating itself and the AIs integrating itself into the human world. Maybe making, you know, insofar as it's poor at communicating and coordinating with AIs, addressing those deficiencies and improving that. Insofar as that's posing a risk or creating inefficiencies because it's unable to benefit from coordinating and trading, then it should have this enormous incentive to address that.

• 1:41:43Speaker 01

  Insofar as there is a lot of value to be gained from dominating and taking over humans. Like what you might get is a more negotiated settlement. You if that's indeed the case, then a war would just be inefficient. And so you would want to negotiate some settlement that results in some outcomes that are mutually beneficial.

• 1:42:05Dwarkesh Patel?

  Compared to the counterfactual, not compared to like there was a mutually beneficial.

• 1:42:12Dwarkesh Patel?

  Trade that was made between the Qing Dynasty and the British after like in the Opium Wars, right? But it was like, like it was like maybe better than like China, pre-industrial China going to war with the British Empire. But it wasn't better than like never having interacted with the British Empire in the first place.

• 1:42:27Speaker 01

  So I I think like, you know, one mistake that I I feel people make is they they do they have this very naive analysis of what creates conflict. And I think Matthew has written a bit about this, a colleague of ours, where they say, you know, there's misalignment. And so that that then creates conflict. But that's actually not what the literature on what causes conflict says creates conflict. It's not just misalignment, it's also other issues like not being able like. Being, having bad understanding of the relative strengths of your armies versus theirs,

• 1:43:05Speaker 01

  or maybe having these very strong commitments that you think some grounds are secret, and so you're not willing to do any trade in order to give up some of that, in order to gain something else. And so then you have to posit some additional things other than just the base value misalignment part.

• 1:43:27Dwarkesh Patel?

  I think you're making a good argument. It's like, let's, humans take up the spears and the machetes and go to war against the AI data centers. Because maybe like that, yeah, there's not this asymmetric information that often leads to conflicts in history. But this argument does not address at all the risk of like, just like takeover, which can be the result of a peaceful end negotiation, or human society is just like, look, we're totally outmatched, and we'll just like, take these meager concessions rather than going to war.

• 1:43:57Speaker 01

  But insofar as it's more peaceful, then I think it's like, much less, much less of a thing to worry about, right? Like, I think there could be this, you know, trend where we indeed have this gradual process where, you know, AI is much more important in the world economy and actually deciding and determining what happens in the world. But this could be beneficial for humans, where, you know, we're getting access to this, you know, much vast, much, much larger economy and much more advanced technological stock.

• 1:44:29Dwarkesh Patel?

  Yeah, so I think it's important to be clear about like, what is the thing that you're actually worried about? Like, I think some people are just like, say that, oh, like, we're gonna, like, humans are gonna lose control of the future. Like, we're not gonna be the ones who are like, making the important decision. We, however conceived, that's also kind of nebulous. But, okay, so is that something to worry about? If you just think like, biological humans should remain in charge of all important decisions forever, then I agree, the, like, the development of AI seems like...

• 1:44:57Speaker 02

  Kind of a problem for that, but in fact, other things also seem like a problem for that. I just don't expect it to generically be true. Like in a million years from now, if even if you don't develop AI, biological humans, the way we recognize them today, are still making all the important decisions and they have something like the culture that we would recognize from ourselves today. I'll be pretty surprised by that. So I think there is a, I think Robin Hanson has again talked about this, where he said, a bunch of the things that people fear about AI are just things they fear about change and fast change. So the thing that's different is that AI has a prospect of accelerating a much of this change, so that it happens in a narrower period.

• 1:45:35Dwarkesh Patel?

  So no, I think, I think there's like not the, I think it's not just the kind of change that would have happened from, let's say genetically modifying humans and blah, blah, blah, is instead happening in a compressed amount of time. I think the worry comes more from like, it is a very, it's not just that change compressions. It's a very different like vector of change.

• 1:45:53Speaker 02

  Yeah, but what is the argument for that? I have never seen a good arguing for this.

• 1:45:58Speaker 01

  You should expect a bunch of change if you accelerate just human change as well. Like you might expect different values to become much more dominant. You might expect people that don't discount the future as much to be much more influential, because they save more and they make good investments that gives them more control.

• 1:46:17Speaker 02

  Higher risk tolerance.

• 1:46:18Speaker 01

  Higher risk tolerance, like because they're more willing to make kind of bets that maximize expected value and so get much more influence. So just generically, the accelerating human change would also result in a lot of things being lost that you might care about. And so I do think there's.

• 1:46:36Dwarkesh Patel?

  I think the argument is that like maybe the speed of the change determines what fraction of the existing population or stakeholders or whatever have some causal influence on the future. And maybe the thing you care about is like, look, there's going to be change, but it's not just going to be like one guy presses a button.

• 1:46:59Speaker 01

  That's like the software singularity extreme.

• 1:47:02Speaker 02

  Right.

• 1:47:03Speaker 01

  And it's more like, you know, like, like, over time, norms change and so forth.

• 1:47:07Speaker 02

  So if you're looking at the software singularity picture, I agree, that picture looks different. And again, I'm coming back to this because obviously Daniel and maybe Scott, to some extent, they probably have this view that the software only singularity is like more plausible. And then one person would just be in a position to like, we could end up in a situation where their idiosyncratic preferences or something end up.

• 1:47:26Speaker 01

  That's right, there, yeah.

• 1:47:27Speaker 02

  Yeah. Even in, I agree, that is, like, that makes the situation look different from if you just like have this broader process of automation. But even in that world, like, I think a lot of people have this view about things like value lock-in, where like they think this moment is like a pivotal moment.

• 1:47:45Speaker 01

  Yeah.

• 1:47:45Speaker 02

  In history. And then we're just gonna have, like, someone is gonna get this AI, which is very powerful because, say, of a software only singularity, and then they're just gonna like lock in some values. And then those values are just gonna be stable for like millions of years. And I think that just looks like very unlike anything that has happened in the past. So I'm kind of confused why people think it's very plausible. I think people have the argument that, like, they see the future, again, in my view, in sort of far mode. They think there's gonna be one AI, it's gonna have some kind of utility function. That utility function is gonna be very stable over time. It's not gonna change. There won't be this messiness of like, lack of coordination between different AIs or

• 1:48:29Speaker 02

  different, like, over time, values drifting for various reasons, maybe because they become less functional or in a new environment, maybe because of other reasons. And so they just don't imagine that. They say, well, I mean, utility functions, we can preserve them forever. We have the technology to do that. So it's just gonna happen. And I'm like, well, that's, like, that seems like such a weak argument to me. Like, I'm not sure.

• 1:48:50Speaker 01

  Actually, if you look at just like, if the, often the idea is because this is like digital, you can preserve the information better and copy it with higher fidelity and so on. But actually, if you look. Even if you look at just like information on the internet, you have this thing called link rot, which happens very quickly. And actually, information that's digital isn't preserved for very long at all.

• 1:49:15Dwarkesh Patel?

  Or and and the point that Matthew was making is that also this like the fact that the information is digital has led to, not maybe led to, but at least been associated with faster cultural change.

• 1:49:24Speaker 01

  Cultural change, exactly.

• 1:49:25Speaker 02

  I mean, basically, technological changes can create incentives for cultural change.

• 1:49:29Dwarkesh Patel?

  Yeah.

• 1:49:30Speaker 02

  Just as they make preserve.

• 1:49:31Dwarkesh Patel?

  That's right. I mean, I I think one of the there's two key arguments that I've heard. One is that we will soon reach something called technological maturity.

• 1:49:39Speaker 02

  Yeah.

• 1:49:40Dwarkesh Patel?

  And once you one of the key ways in which society has been changing recently is that it's not like maybe actually its culture would have changed even more. Actually, no, no, I think this argument is wrong that you're making because we do know that language actually changed a lot more. Like, we can read everything that was written after like 1800s, when literacy became more common. But it's actually even just go back a couple hundred years after that and you're reading Old English and it's hard to understand. And that is a result of literacy and the codification of language.

• 1:50:08Speaker 02

  Well, that information was better preserved. What about like other kinds of cultural practices?

• 1:50:12Dwarkesh Patel?

  But but I I think the argument would be that maybe they would have actually changed more if that change was a result of technological change in general, not the result of information being digitized. And maybe that culture would have actually changed more if information wasn't as well preserved or technology technology had continued to proceed. And the argument is, in the future, we're going to reach some point at which like you've done all the tech. Like, ideas have just gotten way too hard to find and you need like entire galaxies worth of like you need to make a CERN that's the size of a galaxy to make progress physics an inch forward. And at that point, there's this this like growth in technology just churning over civilization goes away. And then you just have the digital thing,

• 1:50:56Dwarkesh Patel?

  which which does mean that like a lock-in is more plausible.

• 1:51:00Speaker 01

  So the technological maturity thing, I I agree that results in this slowdown in change and growth and so on and certain things might get more locked in relative to what preceded it. But then what do we do today about that? Well, like, what could you do to have a kind of positive impact by our lights?

• 1:51:16Dwarkesh Patel?

  Yeah.

• 1:51:17Speaker 01

  And so, asking that question, I mean, Robin Hanson had this question of what could someone do in the 1500s to have a A positive impact on the world today from their point of view, knowing all they knew back then. I think this question is even worse than that because I think the amount of change that happens between today and technological maturity is just orders of magnitude greater than whatever change happened between the 1500s and today. So it's in even worse position than someone in the 1500s thinking about what they could do to have an impact, positive impact in expectation, predictably positive today. And so I think it's just pretty hopeless.

• 1:51:56Speaker 01

  Like, I don't know if we could do anything or find any candidate set of actions that would just make things better, you know, post-lock in.

• 1:52:05Speaker 02

  Or, I mean, that's assuming locking isn't even gonna happen, which is not.

• 1:52:07Dwarkesh Patel?

  In the 1700s, a bunch of British abolitionists were making the case against slavery. And I think the world has, like, we could just live in a slave society. Like, I don't think there's any, in principle, reason why we couldn't have been a slave society to this day, or the world couldn't have slavery. And I think what happened is just, like, they convinced enough British people that slavery is wrong. The British Empire put all its might into abolishing slavery and making that a norm. I think another example is Christianity and the fact that, like, Jesus has these ideals. You could, like, talk about these ideals. I think, like, the world is a more Christian place.

• 1:52:45Speaker 02

  Oh, it is a more Christian place, sure.

• 1:52:46Dwarkesh Patel?

  And also is, like, more of the kind of place, like, I'm not saying Jesus Christ would endorse every single thing that happens in the world today. I'm just saying he endorses this timeline more than one in which he doesn't exist and doesn't preach at all.

• 1:52:59Speaker 02

  I don't know, actually. I mean, I'm not sure if that's true. It seems like a hard question, but.

• 1:53:04Dwarkesh Patel?

  I think, like, some, from the Christian perspective, favorable cultural developments.

• 1:53:08Speaker 02

  I mean, you don't know the counterfactual.

• 1:53:09Dwarkesh Patel?

  I agree, that is always true. I'm just saying, like, the world does have people who, like, read the Bible and are like, "I'm inspired by these ideals to do certain things." And it just seems like that's more likely to lead to, like.

• 1:53:20Speaker 02

  That is what I would call, like, a legacy effect or something. I mean, you can say the same thing about languages, like, some cultures might just become more prominent and their languages might be spoken more, or some symbols might become more prominent. But then there are things like, like, how do cities look and how do cars look and what do people? Spend was there a time doing in their day and what did they spend their money on? And like, and those questions seem much more determined by the like, by how your values change as circumstances change.

• 1:53:49Dwarkesh Patel?

  That might be true. But I I'm in the position with regards to the future where I'm like, I expect a lot of things to be different and I'm okay with them being different. I care much more about the equivalent of slavery, which in this case is literally slavery. But like I was just to put a final point on it. Like the thing I really care about is there's going to be trillions of digital beings. I wanted I wanted it to be the case that they're not like tortured and put into conditions in which they don't want to work and whatever. Or like I don't want galaxies worth of suffering. Okay, but that seems closer to like British abolitionists being like, let's put our empires right against fighting slavery.

• 1:54:25Speaker 02

  I agree, but I would I would distinguish between the case of Christianity and the case of end of slavery. Because I think the end of slavery, like, I agree, you can imagine a society like technologically, it's like feasible to have slavery. But like, I think that's not the relevant thing which brought it to an end. The relevant thing is that the the change in values associated with the industrial revolution made it so that slavery just became like an inefficient thing to sustain in a bunch of ways. And a lot of countries at different times like phased out different things you could call slavery. So for example, Russia abolished serfdom in the 1860s. They were not under British pressure to do so. Like Britain couldn't force Russia to do that, they just did that on their own.

• 1:55:10Speaker 02

  There are there were various ways in which people in Europe were like tied to their land and they couldn't like move, they couldn't go somewhere else. Those movement restrictions were lifted because they were inefficient. There were there were ways in which it used to be like the kind of labor that needed to be done in the colonies to grow sugar or to grow various crops. It was very hard labor. It was not the kind of thing that probably you could have paid people to do because they just wouldn't want to do it because the health hazards and so on were very great, which is why they needed people to force people to do them. And that kind of work over time became less prevalent in the economy.

• 1:55:52Speaker 02

  So again, that reduces the economic incentive to do it. I agree, you could still do it, but.

• 1:55:58Dwarkesh Patel?

  But I would emphasize it's like. Uh, the way you're painting the counterfactual is like, oh, but then in that world, they would have just phased out the remnants slavery. But like, there's a lot of historical examples where there's not necessarily hard labor, only hard labor, like Roman slavery.

• 1:56:14Speaker 02

  Yes, everyone's different.

• 1:56:16Dwarkesh Patel?

  And I interviewed a historian about it recently, the episode hasn't come out, but he wrote a book about, like, the scope, I think it was like 20% of Roman, people under Roman control were slaves. And this was not just agricultural slavery, this was like every, like, and his point was that it was this division of, like, the maturity of the Roman economy is what led to this level of slavery, because the reason slavery collapsed in Europe after the fall of the Roman Empire was because the economy just lost a lot of complexity.

• 1:56:51Speaker 02

  Well, I'm not sure if I would say that slavery collapsed. I mean, I think this just depends on what you mean by slavery. I mean, you know, a lot of ways, people who, in feudal Europe, were like, they-

• 1:56:58Dwarkesh Patel?

  But his point is that actually serfdom was not the descendant institution for Roman slavery.

• 1:57:03Speaker 02

  No, I agree, it was not descendant, but like, in fact, this is sort of the point I'm trying to make, is that values that exist at a given time, like what the values of, we will have in 300 years, or like, from the perspective of someone 1,000 years ago, what values people are gonna have in 1,000 years. Those questions are much more determined by the technological and economic and social environment that's gonna be there in 1,000 years. Which values are gonna be functional, which societies, which values end up being more competitive and being more influential, so that other people, like, adopt their values?

• 1:57:40Dwarkesh Patel?

  Yeah.

• 1:57:40Speaker 02

  And it depends much less on the individual actions taken by people 1,000 years ago. So I would say that the abolitionist thing, it's not, it's not the cause of why slavery came to an end. It is, and slavery comes to an end also because people's own- Like people just have natural preferences that I think are suppressed in various ways during the agricultural era, where like it's more efficient to have settled societies in like cities which are like fairly authoritarian and don't allow for that much freedom. And you're in this Malthusian world where people have very low wages, perhaps compared to what they enjoyed in the hunter gather era.

• 1:58:24Speaker 02

  So it's just a different economic period. And I think people were not, they didn't evolve to have the values that would be functional in that era. So what happened is that there had to be a lot of cultural assimilation, where people had to adopt different values. And in the industrial revolution, people become also very wealthy compared to what they used to be. And that, I think, leads to different aspects of people values being expressed. Like people just have a, put a huge amount of value on equality. Like it's always been the case. But I think when it is sufficiently functional for that to be suppressed, they are capable of suppressing it.

• 1:59:01Dwarkesh Patel?

  I mean, if that's a story, then this puts all the more reason, this makes alignment all the more important, or like the value alignment all the more important, because then you're like, oh, if the AIs become wealthy enough, they actually will make a concerted effort to make sure the future looks more like the utility function we put into them, which I think you have been underemphasizing.

• 1:59:17Speaker 02

  No, I'm not, I'm not underemphasizing that. I think like there are, what I would say is, there are certain things that are like path dependent in history, such that if someone had done something different, like something had gone differently a thousand years ago, then today in some respects would look different. I think, for example, which languages are spoken across which boundaries, or like which religions people have, or like which, like those kinds, or fashion maybe to some extent, though not entirely, like those things are more path dependent. But then there are things that are not as path dependent. So for example, if some empire, like if the Mongols had been more successful and like they somehow I don't know how realistic it is, but they became very authoritarian and had slavery everywhere.

• 1:59:58Speaker 02

  Would that have actually led to slavery being a much more enduring institution a thousand years later? That seems not true to me. Like, the forces that led to the end of slavery seemed like they were not contingent forces. They seemed like deeper forces than that. And if you're saying, well, if we aligned A S today to some bad set of values, then that could affect the future in some ways which are more fragile, that seems plausible. But I'm not sure how much of the things you care about the future and how much, the ways in which you expect the future to get worse, you actually have a lot of leverage on at the present moment.

• 2:00:40Dwarkesh Patel?

  I mean, another example here might be factory farming, where you could say, like, oh, us having, it's not like us having better values over time led to suffering going down. In fact, your suffering might have gone up because the

• 2:00:50Speaker 02

  A lot of people say that.

• 2:00:52Dwarkesh Patel?

  The incentives that led to factory farming emerging are just like

• 2:00:56Speaker 02

  And probably when factory farming comes to an end, it will be because the incentives start doing that right.

• 2:01:01Dwarkesh Patel?

  Right. But suppose I care about making sure the digital equivalent of factory farming doesn't happen, where it may be more efficient, maybe in the, all else being equal, it's just more economically efficient to have suffering minds doing labor for you than non-suffering minds because of the intermediary benefits of suffering, or something like that, right? What would you say to somebody like me, where I'm like, I really want that not to happen. I don't want the like home filled with suffering workers or whatever. Is it just like, we'll give up because this is the way economic history is

• 2:01:40Speaker 02

  No, I don't think you should give up. It's more like, it's hard to anticipate the consequences of your actions in the very distant future. So, like, I would just recommend that you should just discount the future, not for like a moral reason, not because like the future is worth less or something, but because it's just very hard to anticipate the effects of your actions. In the near term, I think there are things you can do that seem like they would be beneficial. Like, for example, you could try to

• 2:02:08Speaker 02

  Align your present AI systems to value the things that you're talking about. Like, they should like value happiness and they should dislike suffering or something. You might want to support like political solutions that would like, basically, you might want to build up the capacity so that in the future, if you notice something like this happening, then we might have some ability to intervene. Like, maybe you would think about the prospect of, well, eventually, we're gonna maybe colonize other stars and like civilization might become very large and communication delays might be very long between different places. And in that case, competitive pressures between different local cultures might become much stronger because there's like, it's harder to centrally coordinate.

• 2:02:50Speaker 01

  That's right.

• 2:02:51Speaker 02

  And so in that world, you might expect competition to take over in a stronger way. And if you think the result of that is gonna be a lot of suffering, maybe you would try to stop that. Again, I think at this point, it's not, it's very far from obvious that trying to say limit competition is actually a good idea. I would probably think it's a bad idea. But maybe in the future, we will receive some information and we'll be like, oh, like, we were wrong. Actually, we should stop this. And then maybe you want to have the capacity so that we can make that decision.

• 2:03:20Speaker 01

  Right.

• 2:03:21Speaker 02

  But that's a very hard, like, that's a nebulous thing. How do you build that up? Well, I don't know. I mean, you would need to, like, that's the kind of thing I would be trying to do.

• 2:03:27Speaker 01

  Yeah, I think the overall takeaway I take from the way that I think about it and I guess we think about it is be more humble in what you think you can achieve.

• 2:03:38Speaker 02

  Yeah.

• 2:03:38Speaker 01

  And like, just focus on the nearer term, not because it's more morally important than the longer term, but just because it's much easier to have a predictably positive impact on that.

• 2:03:49Dwarkesh Patel?

  One thing I've noticed over the last few weeks, thinking about these bigger fixture topics and interviewing Daniel and Scott and then you two, is how often I've changed my mind about everything from the smallest questions about when AI will arrive. It's funny that that's the small question in the grand scheme of things, to whether there will be an intelligence explosion or whether there'll be an R and D explosion, to whether there'll be explosive growth or how to think about that. And if you're in a position where you are just like...

• 2:04:23Dwarkesh Patel?

  Incredibly epistemically uncertain about what's going to happen. I think it's important to just like directly acknowledge like this is instead of just instead of becoming super certain about like your next conclusion, just being like, well, let me just, at least from my perspective, I'm just like, let me just take a step back. I'm like, I'm not sure what's going on here. And I think a lot more people should be from that perspective, unless you've had the same opinion about AI for many years, in which case I have other questions for you about why that's the case. And in other situations, I mean, generally, how we as a society deal with topics on which we are this uncertain is just to have freedom, decentralization, both decentralized knowledge

• 2:05:05Dwarkesh Patel?

  and decentralized decision-making take the reins and not to do super high volatility, centralized moves like, hey, let's nationalize so we can make sure that we can make sure the AI, the software only singularities aligned or not to do, make moves that are just incredibly contingent on one worldview that are brittle under other considerations. And that's become a much more salient part of my worldview. I think just classical liberalism is the way we deal with being this epistemically uncertain. And I think we should be more uncertain than we've ever been in history, as opposed to many other people who seem to be more certain than they are about other sort of more mundane topics.

• 2:05:44Speaker 01

  Yeah, I think it's, it's very hard to predict what happens because of this acceleration basically means that you find it much harder to predict what the world might be in 10 years' time. I think these questions are also just like very difficult and we don't have very strong empirical evidence. And then there's like a lot of this kind of disagreement that exists.

• 2:06:10Speaker 02

  Like, I would say that it's important to, like, the, like, in a lot of cases and a lot of situations, it's much more important to be, like, maintain flexibility and ability to adapt to new circumstances, new information than it is to get a specific plan that's gonna be like correct and like that's gonna be very detailed and has a lot of specific policy recommendations and things that you should do. So, like, that's actually also the thing that I would recommend, like, if I wanna make the- Make the transition to AI in this period of explosive growth go better. I would just prefer it if we, like, in general, had, like, higher-quality institutions.

• 2:06:46Speaker 01

  Yeah.

• 2:06:46Speaker 02

  And, but I am much less bullish on someone sitting down today and working out, okay, like, what will this intelligence explosion or explosive growth be like? What should we do? Like, I think plans that you work out today are not gonna be that useful when the events are actually occurring, because you're gonna learn so much stuff that you're gonna, like, update on so many questions that these plans are just gonna become obsolete.

• 2:07:12Speaker 01

  That's right. Because one thing you could do is you could look at, say, the history of war planning and how successful war planning has been for, like, actually anticipating what actually happens when the war actually happens.

• 2:07:22Speaker 02

  So for one example, I think I might have mentioned this in, like, off the record at some point, but before the Second World War happened, people were, obviously, people saw that there were all these new technologies, like tanks and airplanes and so on, which were now, like, they existed in World War I, but in a much more primitive setting. So they were wondering what is gonna be the impact of these technologies now that we have them in much greater scale. And the British government had estimates of how many casualties there will be from aerial bombardments in the first few weeks of the Second World War, and they expected hundreds of thousands of that, casualties, basically, in, like, two weeks, three weeks after the war begins. So the idea was that air bombing is basically this unstoppable force. All the major urban centers are gonna get bombed, tons of people will die.

• 2:08:08Speaker 02

  So, basically, like, we can't have a war, because if there's a war, then it will be a disaster, because, like, we will have this aerial bombardment. But later, it turned out that that was totally wrong. In fact, in all Britain, there were fewer casualties from air bombing in the entire, sort of, six years of the Second World War than the British government expected in the first few weeks of the war. Like, they had less casualties in six years than they expected in, like, three weeks. So why did they get it wrong? Well, I mean, there are lots of boring practical reasons. Like, for example, it turned out to be really infeasible to bomb, especially early on, to bomb cities in daytime because your aircraft would just get shot down.

• 2:08:48Dwarkesh Patel?

  Yeah.

• 2:08:49Speaker 02

  But then if you tried to bomb at nighttime, then your bombing was really imprecise and only a very small fraction of it actually hit. And then people also underestimated the extent to which people on the ground could, like firefighters and so on, could just sort of go around the city and then put out fires from bombs that were falling on structures. They overestimated the amount of economic damage that it would do. They underestimated how economically costly it would be. Like, basically, you're sending these aircraft and then they're getting shot down. Well, an aircraft is very expensive. So in the end, what turned out is when the Allies started bombing Germany, they were, like, for each dollar of capital they were destroying in Germany, they were spending like four to five dollars on the aircraft and fuel and training of the pilots and so on that they were sending in missions.

• 2:09:34Speaker 02

  And the casualty rate was very high, which later got covered up by the government because they didn't want people to worry about, you know.

• 2:09:41Dwarkesh Patel?

  Yeah.

• 2:09:42Speaker 02

  So that is a kind of situation where all the planning that you would have done in advance predicated on this assumption of like, air bombing is gonna be this, like, nuclear weapons light, basically. Like, it's extremely destructive, there's gonna be some aspect to which.

• 2:09:56Dwarkesh Patel?

  I mean, it was, though, right? Like, eighty-four thousand people died in one night of firebombing in Tokyo. Like, Germany, like, large fractions of their.

• 2:10:07Speaker 02

  But that was over the period of six years of war.

• 2:10:10Dwarkesh Patel?

  But there were, like, single firebombing attacks. I mean, it was a case that during the end of World War II, when they were looking for the place to launch the atomic bombs.

• 2:10:20Speaker 02

  That's right.

• 2:10:20Dwarkesh Patel?

  They just had to go through like a dozen cities because they're like, they just wouldn't be worth nuking them because they're already destroyed by the firebombing.

• 2:10:28Speaker 02

  That's right. But, but the level of, if you look at the level of destruction, that was, it was expected within the space of a few weeks. And then this level of destruction took many years. So there was like a two-order magnitude mismatch or something like that, which is pretty huge.

• 2:10:42Dwarkesh Patel?

  Yeah.

• 2:10:43Speaker 02

  And that affected the way people think about it.

• 2:10:45Dwarkesh Patel?

  Right.

• 2:10:45Speaker 01

  An important, like, underlying theme of much of what we have discussed is like, how powerful just reasoning about things is to making progress about what specific plans you wanna make. To prepare and make this transition to advanced AI go well.

• 2:11:02Dwarkesh Patel?

  Yeah.

• 2:11:02Speaker 01

  And our view is, well, it's actually quite hard and you need to make contact with the actual world in order to inform most of your beliefs about what actually happens. And so it's somewhat futile to think, to do a lot of kind of war gaming and figure out, you know, how AI might go and what we can do today to make that go a lot better, because a lot of the policies you might come up with might just look fairly silly. And I think there's in the, in the thinking about how AI actually has this impact, again, people think, oh, you know, just AI reasoning about doing science and doing R and D just has this drastic impact on, you know, the overall economy or, or technology. And our view as well, actually,

• 2:11:46Speaker 01

  again, making contact with the real world and getting a lot of data from experiments and from deployment and so on, it's just very important. So I think there is this underlying, underlying kind of latent variable, which explains some of this disagreement, both on the policy prescriptions and about the extent to which we should be humble versus ambitious about what we ought to do today, as well as for thinking about the mechanism through which AI has this impact. And this underlying latent thing, it's like, what is the power of reason? Like, how much can we reason about what might happen? How much can reasoning in general figure things out about the world and about technology? And, you know, so that is like a kind of core underlying disagreement here.

• 2:12:27Dwarkesh Patel?

  Yeah, yeah. I do want to ask, you say in your announcement, we want to accelerate this broad automation of labor as fast as possible. As you know, many people think it's a bad idea to accelerate this.

• 2:12:42Speaker 02

  Yes.

• 2:12:43Dwarkesh Patel?

  The broad automation of labor and AGI and everything that's involved there. Why, why do you think this is, this is good?

• 2:12:49Speaker 02

  So the argument for why it's good is that we're going to have this enormous increase in economic growth, which is going to mean like enormous amounts of wealth and like incredible new products that you can't even imagine and like healthcare or whatever. And like the quality of life of the typical person is probably going to go up a lot. Early on, probably also their wages are going to go up because the AI systems are going to be automating things that are complementary to their work. Or like it's going to be automating part of their work and then you'll be doing the rest and then you'll be getting paid much more on that. And in the long term, eventually, we do expect wages to fall just because of arbitrage with the AIs. But by that point, we think humans will own enormous amounts of capital. And there will also be like ways in which even the people who don't own capital,

• 2:13:34Speaker 02

  we think are just going to be much better off than they are today. Like, I think it's just hard to express in words the amount of wealth and increased variety of products that we would get in this role. It will be probably more than a difference between like 1800 and today. So if you imagine that difference, it's like such a huge difference. And then imagine like two times, three times, whatever.

• 2:13:57Dwarkesh Patel?

  The standard argument against this is why does the speed to get there matter so much? Because especially if the trade off against the speed is the probability that this transition is achieved successfully in a way that benefits humans.

• 2:14:11Speaker 01

  I mean, it's unclear that this trades off against the probability of it being achieved successfully or something like.

• 2:14:18Dwarkesh Patel?

  There might be an alignment tax.

• 2:14:20Speaker 01

  I mean, maybe. I like, you can also just do the calculation of how much a year's worth of delay costs for current people. Like, you know, this is this enormous amount of utility that people are able to enjoy. And that gets brought forward by by year or pushed back by year. If you delay things by year, and how much is this this worth? Well, you know, you can you can look at simple models of how concave people's utility functions are and do some calculations. And maybe that's worth on the order of tens of trillions of dollars per year in consumption. That that is roughly the amount consumers might be willing to defer in order to get, you know, bring forward the date of automation one year.

• 2:15:03Dwarkesh Patel?

  In absolute terms, it's high. In relative terms, relative to if you did think it was going to nudge the probability one way or another. Of building systems that are aligned and so forth. Then it's like, just so small compared to all of the future.

• 2:15:17Speaker 02

  I agree. So like, there are a couple of things here. First of all, I think the way you think about this matter.

• 2:15:24Speaker 02

  So first of all, we don't actually think that it's clear whether speeding things up or slowing things down actually makes like doom-y outcome more or less likely. Like, I think that's just a question that doesn't seem obvious to us. Like, we don't, like, partly because of our views on the software R and D side, we don't really believe that if you just pause and then you like do research for 20 years at a fixed level of compute scale, that you're actually going to make that much progress on relevant questions on alignment or something. Like, I think, like, imagine you were trying to make progress in alignment in 2016 with the compute budgets of 2016 and the, like, well, you would have gotten nowhere, basically. Like, you would have discovered none of the things that people have today discovered and that turned out to be useful.

• 2:16:08Speaker 02

  And I think if you pause today, then we will be in a very similar position in 10 years, right? Like, we have not made a bunch of discoveries. So the scaling is just really important to make progress in alignment, in our view. And then there's a separate question of how long-term is should you be in a very different senses? So there's a moral sense where like, how much should you actually care about people who are alive today as opposed to people who are not yet born? That's just a moral question. And there's also a practical question of, as we discussed, how certain can you be about the impact your present actions are actually going to have on the future?

• 2:16:42Dwarkesh Patel?

  Okay, maybe you think it really doesn't matter whether you slowed things down right now or you speed things up right now. But is there some story about why speeding them up from the alignment perspective actually helped? It's good to have that extra progress right now rather than later on? Or is it just that, well, if it doesn't make a difference either way, then it's better to just get that extra year of people not dying and having cancer cures and so forth?

• 2:17:05Speaker 02

  I think I would say the second, but like, but it's just important to understand the Like value of that, right?

• 2:17:11Dwarkesh Patel?

  Yep.

• 2:17:11Speaker 02

  Even in, even in purely economic terms, like, uh, in, like, imagine that you would be, like, each year of delay might cause like, maybe a hundred million people, maybe more, uh, maybe one hundred fifty, two hundred million people who are alive today to end up dying, right? So the, even in purely economic terms, the value of a statistical life is like, pretty enormous, uh, especially in western countries, so it's like, uh, sometimes people use numbers as high as ten million dollars for a single life. So imagine you do like ten million dollars times a hundred million people, that's like a huge number, right?

• 2:17:46Dwarkesh Patel?

  Yeah.

• 2:17:47Speaker 02

  So like, I, I think, uh, like, that is just so enormous, that unless you're just, so, so I think for you to think that speeding things up is a bad idea, you have to first be, like, just have this long-termist view, where you look at the long-run future, you think your actions today have high enough leverage that you can predictably affect the direction of long-term future.

• 2:18:10Dwarkesh Patel?

  Well, in this case, it's kind of different, because you're not saying, "I'm going to affect what some emperor a thousand years from now does," like somebody in the year zero would have to do to be a long-termist. In this case, you just think there's this incredibly important inflection point that's coming up, and you just need to have influence over that crucial period of explosive growth of intelligence explosion or something. So I think it is a much more practicable, uh, prospect than...

• 2:18:36Speaker 02

  So I agree in relative terms, so, like, in relative terms, I agree the present moment has, like, is a moment of higher leverage.

• 2:18:43Dwarkesh Patel?

  Yeah.

• 2:18:43Speaker 02

  And you can expect to have more influence. I just think in absolute terms, the amount of influence you can have is still quite low.

• 2:18:48Dwarkesh Patel?

  Yeah.

• 2:18:49Speaker 02

  So it might be orders of magnitude greater than it would have been two thousand years ago, and still be quite low.

• 2:18:54Speaker 01

  And, and again, I think there's this, like, difference in opinion about how broad and diffuse this transformation ends up being versus how concentrated within a specific labs, where the very idiosyncratic decisions made by that lab will end up having very large impact.

• 2:19:10Dwarkesh Patel?

  Yeah.

• 2:19:10Speaker 01

  If you think, if you think those developments will be very concentrated, then you think the leverage is especially great. And so then you might be especially excited about having the ability to influence

• 2:19:22Dwarkesh Patel?

  Yeah.

• 2:19:22Speaker 01

  how that transition goes. But our view is very much that this transition happens very diffusely by way of many, many organizations and companies doing, doing things. And for those actions to be determined a bunch by economic forces rather than have idiosyncratic preferences, you know, on the part of labs or these kind of decisions that have these kind of founder effects that last for very long.

• 2:19:47Dwarkesh Patel?

  Right. Okay, let's go through some of the objections to explosive growth, which is most people are actually more conservative, not more aggressive about the forecasts you have. So obviously, one of the people who has articulated their disagreements with your view is Tyler Cowen. He made an interesting point when we did the podcast together, and he said, "Most of Sub-Saharan Africa still does not have reliable clean water. The intelligence required for that is not scarce. We cannot so readily do it. We are more in that position than we might like to think along other variables."

• 2:20:22Speaker 01

  I mean, we agree with this. Like, I think, like, intelligence isn't the bottleneck that's holding back, you know, technological progress or economic growth.

• 2:20:32Dwarkesh Patel?

  Right.

• 2:20:32Speaker 01

  It's like many other things. And so I think that this is very much consistent with our view that scaling up your over economy, accumulating capital, accumulating human capital, having, you know, all these factors scales.

• 2:20:45Speaker 02

  In fact, this is even consistent with what I was saying earlier, that I was pointing out this like, oh, like, good management and like, good policies and those just contribute to TFP and they can be bottlenecks. And I heard like.

• 2:20:55Dwarkesh Patel?

  But like, right now, we could just plug and play our better management into Sub-Saharan Africa.

• 2:21:02Speaker 01

  No, we can't. It's hard.

• 2:21:02Speaker 02

  I don't think.

• 2:21:03Dwarkesh Patel?

  Okay, so that's what I was, maybe I should have said it. One could theoretically imagine plugging and playing with.

• 2:21:09Speaker 01

  Yeah. I can imagine many things.

• 2:21:14Dwarkesh Patel?

  But we cannot so readily do it because of, it's like, hard to articulate why and you wouldn't, it wouldn't be so easy to do it just in terms of capital or labor. Why not think that the rest of the world will be in this position with regards to the advances that AI will make possible?

• 2:21:31Speaker 01

  I mean, if the AI advances are like the kind of geniuses in a data center, then I agree that that might. Be bottlenecked by the rest of the economy not scaling up and being able to accumulate the relevant capital to make those changes feasible. So I kind of agree with this picture. And I think this is like, you know, an objection to the geniuses in a data center type view.

• 2:21:55Speaker 02

  Yeah.

• 2:21:55Speaker 01

  And like, I buy basically this.

• 2:21:57Speaker 02

  And also the fact that like, it's also plausible, you're going to have the technology. But then some people are not going to want to deploy it, or some people are going to have norms and laws and cultural things that are going to make it so that AI is not able to be widely deployed in their economy, or not as widely deployed as otherwise it might be. And that is going to make those countries or societies just slower. That's, I mean, that's like, like, some countries will be growing faster, just like Britain and the Netherlands were sort of the leaders in the industrial revolution. They were the first countries to start experiencing rapid growth. And then other countries, even in Europe, sort of had to come from behind.

• 2:22:33Speaker 02

  Well, again, I just think we expect the same thing to be true for AI. And, I mean, the reason that happened was exactly because of these kinds of reasons, where like those countries that are culture or governance systems or whatever, which were just worse and bottlenecked the deployment and scaling of the new technologies and ideas. It seems very plausible.

• 2:22:52Dwarkesh Patel?

  But you're saying as long as there's one jurisdiction.

• 2:22:55Speaker 02

  Yeah.

• 2:22:56Dwarkesh Patel?

  But then again, you also previously emphasized the need and the need to integrate with the rest of the global economy and the human economy. So doesn't that contradict?

• 2:23:05Speaker 01

  Doesn't often require cultural homogeneity. Like, we can, we trade with countries, like the US trades with China quite a lot, actually, and there's like a bunch of disagreements.

• 2:23:14Dwarkesh Patel?

  What if the US is like, I don't like the UAE is doing explosive growth with AI, we're just going to like embargo them?

• 2:23:22Speaker 01

  That seems plausible.

• 2:23:23Dwarkesh Patel?

  And then would that not not prevent explosive growth?

• 2:23:26Speaker 01

  I mean, like, I think that would be plausible at the point at which it's revealing a lot about the capabilities and the power of AI.

• 2:23:34Dwarkesh Patel?

  Yeah.

• 2:23:34Speaker 01

  And you should also think that that creates both an incentive to embargo, but also an incentive to adopt the very similar. Styles of governing that enable AI to be able to produce a lot of value.

• 2:23:47Dwarkesh Patel?

  What do you make of this? I think people interpret explosive growth from an arms race perspective. And that's often why they think in terms of public-private partnerships for the labs themselves. But just this idea that you have the geniuses in the data center, they like, you can have them come up with the mosquito drone swarms. And then those drone swarms will, you know, like if China gets to the swarms earlier. I mean, even within your perspective where it's like not, is this a result of your whole economy being advanced enough that you can produce mosquito drone swarms? You being six months ahead means that you could decisively win. Does it? I don't know. Maybe you being like a year ahead in explosive growth means you could decisively win a war against China or China could win a war against you.

• 2:24:30Dwarkesh Patel?

  So would that lead to an arms race like dynamic?

• 2:24:33Speaker 02

  I mean, I think it would to some extent, but like, I'm not sure if I would expect that like a year of lead to be like enough to take a risk. Because like if you go to war with China, I mean, for example, if the US went to war with, if you replace China today with China from 1990. Or if you replace Russia today with Russia from like 1970 or 1980. It's possible that their ICBM and whatever technology is already enough, like it's already enough to make, like have very strong deterrence. So maybe even that lead, a technological lead is not sufficient so that you would feel comfortable going to war. So that seems possible.

• 2:25:12Dwarkesh Patel?

  Yeah. And actually this relates to a point that Gorn was making, which is, he was like, okay, this is going to be a much more unstable period than the industrial revolution. Even though industrial revolution saw the, saw many countries gain rapid increases in their capabilities. Because this is just like, within this span, if you've got a centuries worth of progress compressed within a decade. One country gets to like ballistic missiles first, then the other country gets to railroads first and so forth.

• 2:25:46Dwarkesh Patel?

  But if you have this more integrated perspective about what it takes to get to ballistic missiles and to railroads. Then you might think, no, basically, this isn't some orthogonal vector. It just like, you're just like churning on the tech tree further and further.

• 2:26:01Speaker 02

  Yeah, I mean, for what it's worth, I do think like it's possible if you have it just happen in a few countries which are relatively large and have enough land or something like those countries could just like they would be starting from a lower base compared to the rest of the world. So they would need to catch up to some extent. So like if they are just going to sort of grow internally and they're not going to depend on the external supply chains. But like that doesn't seem like something that's impossible to me.

• 2:26:25Dwarkesh Patel?

  Yeah.

• 2:26:25Speaker 02

  Some countries could do it, but it would just be like more difficult. But in this setting, if some countries have like a significant policy advantage over the rest of the world and they start growing first and then they won't necessarily have a way to get other countries to adopt their norms and culture. So in that case, they might just, it might be more efficient for them to do the growth locally, right? So that's why I was saying the growth differentials will probably be determined by like regulatory jurisdiction, boundaries more than anything else.

• 2:26:53Dwarkesh Patel?

  Right.

• 2:26:54Speaker 02

  But like, I'm not saying say the US by itself, it had AI, but it couldn't like, it couldn't get the rest of the world to adopt AI. I think that would still be sufficient for access to growth.

• 2:27:04Dwarkesh Patel?

  How worried should we be about the fact that China today just has, because it industrialized relatively recently, just has more industrial capacity and know how and all the other things of learning by doing and so forth. If this, if we buy your model of how technology progresses with or without AI, how like, are we just underestimating China because we have this perspective that like, what fraction of your GDP you're spending on research or matters, when in fact it's the kind of thing where like I am, I've got all the factories in my backyard and I know how they work and I'm like, I can go buy a component whenever I want.

• 2:27:40Speaker 01

  I don't think people are necessarily underestimating China. I mean, it depends on who you're looking at, but, but it seems like the discussion of China is this very big discussion when in, in, in these AI circles, right? And so people are like very much appreciating the power and the potential threat that China poses. But I think the key thing is not just like the scale in terms pure, in terms of pure number of people or like number of firms or something, but the scale of the overall economy, which is just measured in how much is being produced in terms of dollars.

• 2:28:12Speaker 02

  But

• 2:28:12Speaker 01

  And there, you know, the US is, is ahead and

• 2:28:14Speaker 02

  But doesn't the

• 2:28:17Dwarkesh Patel?

  Like, we're not expecting all this explosive growth to come from financial services. We're expecting it to start from a base of industrial technology and industrial capacity.

• 2:28:25Speaker 02

  I know financial services can be important if you want to scale very big projects very quickly.

• 2:28:29Speaker 01

  Financial services are very important for like raising funding and getting investments in data centers and.

• 2:28:35Dwarkesh Patel?

  But if I understood you correctly, it just seems like, man, you know how to do all the, like, you know how to build the robot factories and so forth. That like know-how, which in your view is still crucial to technology growth and just general economic growth, is lacking. And you might have more advanced financial services, but like, it seems like the more you take your view seriously, the more it seems like the

• 2:28:58Dwarkesh Patel?

  that having the Shenzhen locally matters a lot.

• 2:29:00Speaker 01

  I mean, relative to like what starting point, like, I think people already appreciate that China is very important. And then I agree that there are some domains where China is leading, but then there are very many domains in which the US is leading or the US and its allies where, you know, countries that are producing relevant inputs for AI that the US has access to, but China doesn't. So I think, I think the US is just like ahead on many dimensions and there's some that China is ahead or at least very close. So I don't, I don't think this should cause you to update very strongly in favor of China being a much bigger deal. At least depending on where you started.

• 2:29:40Speaker 02

  I think people already think China is a big deal, like this is the big underlying thing here. Like, if we were just very dismissive of China, then maybe this would be a reason to update, but.

• 2:29:47Dwarkesh Patel?

  I get your argument that thinking about the economy-wide acceleration is more important than focusing on the IQ of the smartest AI. But at the same time, do you believe in the idea of superhuman intelligence? Is that a coherent concept? In the way that you don't necessarily stop at human-level go-playing, you just go way beyond it in Elo Score. Will we get to systems that are like that with respect to the broader range of human abilities? And maybe that doesn't mean they become god because there's other ASIs in the world. But they, you know what I mean? Like, will there be systems with such superhuman capabilities?

• 2:30:26Speaker 01

  Yeah, I mean, I, I do expect that. I think there's a question of how useful is this concept for thinking about this transition to a world with much more advanced AI and. I I I don't find this like a particularly meaningful or helpful concept. Like, I I think people introduce some of these notions that on the surface seem useful, but then actually, when you delve into them, they're just like very vague and kind of unclear what you're supposed to make of this. And you have this notion of AGI, which distinguishes from narrow AI in the sense that it's much more general and and maybe, you know, can do everything that a human can do on average. I mean, AI systems have these very jagged profiles of capability. So you have to somehow take some notion of average capabilities. And what exactly does that mean?

• 2:31:11Speaker 01

  It just feels really unclear. And then you have this notion of ASI, which is AGI in the sense that it's very general, but then it's also better at humans than on every task. And, you know, is this is this a meaningful concept? I guess it's coherent. I think this is not a super useful concept because I prefer just thinking about, you know, what actually happens in the world. And you could have a drastic acceleration without having an AI system that can do everything better than humans can do. I guess you could have no acceleration when you have an ASI that is better at humans than, you know, better than humans at everything, but it's just very expensive or very slow or something.

• 2:31:55Speaker 01

  So I don't I don't find that particularly meaningful or useful. I just prefer thinking about, you know, the overall effects on the world and what AI systems are capable of producing those types of effects.

• 2:32:06Dwarkesh Patel?

  Yeah, I mean, one intuition pump here is compare John von Neumann versus a human from the standard distribution. If you added a million John von Neumanns to the world, what would the impact on growth be as compared to just adding a million people from the normal distribution?

• 2:32:25Speaker 02

  Well, I agree, it would be a much greater.

• 2:32:27Dwarkesh Patel?

  Right. And but then, like, Because of Moravax paradox type arguments that you made earlier, that evolution has not necessarily optimized us for that long along the kind of spectrum on which John von Neumann is distinguished from the average human. And given the fact that already within this deviation, you have this much greater economic impact. Why not focus on going optimizing on this thing that evolution has not optimized that hard on further?

• 2:32:51Speaker 02

  I don't think we shouldn't focus on that. I think it's but what I would say is, for example, if you're thinking about the capabilities of Go-playing AIs, then the concept of a superhuman Go AI, yeah, you can say like that could be, that is a meaningful concept. But like if you're developing the AI, it's not a very useful concept. If you're like, if you just look at the scaling curve, it's just like it just goes up and there is some human level somewhere, but like the human level is not privileged in any sense. So the question is like, is it a useful thing to be thinking about? And the answer is probably not, depends on what you care about. So I'm not saying we shouldn't focus on trying to make the system smarter than humans are. Like, I think that's a good thing to focus on.

• 2:33:31Dwarkesh Patel?

  Yeah, I guess I'm trying to understand whether we will stand in relation to the AIs of 2100 that humans stand in relationship to other primates. Like, is that like, is that the right mental model we should have? Or is there going to be a much greater familiarity with their cognitive horizons?

• 2:33:47Speaker 01

  I mean, I think AI systems will be very diverse. And so it's not super meaningful to ask something about, you know, this very diverse range of systems and where we stand in relation to them.

• 2:33:58Dwarkesh Patel?

  I mean, we'll be able to like cognitively access the kinds of considerations they can take on board. Like, humans are diverse, but no chimp is able, going to be able to understand this argument in the way that another human might be able to, right? So I'm just like, if I'm trying to think about my place or a human's place in the world of the future, I think it's sort of, it is a relevant concept of is it just that the economy has grown a lot and there's much more labor? Or are there beings who are in this crucial way, super intelligent?

• 2:34:28Speaker 01

  I mean, there will be many things that we just like will fail to understand. And to some extent, there are many things today that People don't understand about how the world works and how certain things are made. And and then, you know, how important is it for us to have access? Or in principle, be able to access those considerations?

• 2:34:48Dwarkesh Patel?

  Mm hmm.

• 2:34:48Speaker 01

  And I think it's not clear to me that that's particularly important, that like any individual human should be able to access all the relevant considerations that produce some outcome. Like, that just seems like overkill. Like, why do you need that to happen? I think it would be nice in some sense, but I think if you want to have a very sophisticated world where you have very advanced technology, those things will just not be accessible to you. And then, like, so you have this trade off between accessibility and maybe how advanced the world is. And, you know, from my point of view, I'd much rather live in a world which has very advanced technology,

• 2:35:29Speaker 01

  has a lot of products that I'm able to enjoy and a lot of inventions that I can, you know, improve my life with, if that means that I just don't understand them. I mean, I think this is like a very simple trade that I like, I'm very willing to make.

• 2:35:45Dwarkesh Patel?

  Hmm. Okay, so let's get back to objections to explosive growth. We discussed a couple already. Here's another, which is more a question than an objection. Where is all this extra output going? Like, who is consuming it? If the economy is a hundred x bigger in a matter of a decade or something, like, to what end?

• 2:36:05Speaker 02

  So first of all, I think even if you view that along what you might call the intensive margin in the sense that you just have more of the products you have today. I think there is just a lot of, like, there will be a lot of appetite for that. Maybe not quite a hundred x, but like, that might start hitting some emission turns.

• 2:36:23Speaker 01

  Current GDP per capita on average in the world is 10 K a year or something, right? And there are people who enjoy millions of dollars. And so there's A gap between what people enjoy and like don't seem to be super diminished in terms of marginal utility. And so there's a big room, there's a lot of room on just purely the intensive margin of just consuming the things we consume today, but more. And then, and then there's this maybe much more important dimension along which we will expand, which is

• 2:36:51Speaker 02

  Product variety.

• 2:36:52Speaker 01

  Yeah. Extensive margin of what, of what is the scope of things that you're consuming. And if you look at something like the industrial revolution, that seemed to have been the main dimension along which we kind of expanded to consume more. There's just on any kind of sector that you care about, transportation, medicine, you know, entertainment and food, there's just this massive expansion in terms of variety of things that we're able to consume that is enabled by new technology or new trade routes or new methods of producing things. And so that is, I think, the really the key thing that we will see, you know, come along with this kind of expansion in consumption.

• 2:37:35Speaker 02

  Hmm, yeah.

• 2:37:36Dwarkesh Patel?

  Another point that Tyler makes is that there will be some mixture of Baumwinkle cost disease where you're bottlenecked by this lowest growing thing, which grows in proportion, the fastest productivity things basically diminish their own

• 2:37:56Speaker 02

  Share in output, yeah.

• 2:37:57Dwarkesh Patel?

  That's right, yeah.

• 2:37:58Speaker 01

  I mean, like, we totally agree with that. I would say that that's just like a kind of qualitative consideration. It doesn't itself, it isn't so, it isn't itself sufficient to make a prediction about what growth rates are permitted, given these Baumwinkle effects versus not. It's just like a qualitative consideration and then you might need to make additional assumptions to be able to make a quantitative prediction. So I think it's a little bit

• 2:38:25Speaker 02

  So the like convincing version of this argument would be If you did the same thing that we were doing earlier with the software and the singularity argument, where we were pointing to essentially the same objection, where there are multiple things that can bottleneck progress. So I would be much more convinced if someone pointed to like an explicit thing, they would be like, like here, like healthcare is like this very important thing. And why should we expect AI to like make that better? Like, that doesn't seem like that would get better because of AI. So like, that maybe healthcare just becomes a big part of the economy and then that bottleneck. So like, if there was some specific sector- But maybe the argument is that if there's even one- No, if there's one, though, like if that's a small part of the economy, then you could just still get a lot of growth. You just automate everything else. And that is going to produce a lot of growth.

• 2:39:08Speaker 01

  So it has to like quantitatively work out. And so you actually have to be quantitatively specific about what this objection is supposed to be.

• 2:39:15Speaker 02

  Right. So first of all, you have to be specific about, okay, what are these tasks? What are the current share in economic output? The second thing is you have to be specific about how bad do you think the complementarities are? So in numerical terms, economists use the concept of elasticity of substitution to quantify this. So that gives you a numerical estimate of if you just have much more output on some dimensions, but not that much on other dimensions, how much does that increase economic output overall? And then there's a third question. You can also imagine you automate a bunch of the economy. Well, a lot of humans were working on those jobs. So now, well, they don't need to do that anymore because those got automated. So they could work on the jobs that haven't yet been automated.

• 2:39:56Speaker 02

  So for example, as I gave the example earlier, you might imagine a world in which remote work tasks get automated first, and then sensory motor skills lag behind. So you might have a world in which software engineers like become physical workers instead. Of course, in that world, the wages of physical workers will be much higher than their wages are today. So that reallocation also produces a lot of extra growth, even in the, like if bottlenecks are maximally powerful. Like, even if it's literally, you just look at all the tasks in the economy and literally take the worst one for productivity growth, you would still get a lot of increase in output because of this reallocation.

• 2:40:35Speaker 01

  So I think one point that I think is useful to make, our experience talking to economists about this is that they will bring up these kind of more qualitative considerations. Whereas the arguments that we make are like make specific quantitative predictions about growth rates. So for example, you might ask like, how fast will the economy double? And then we can think about, you know, an H one hundred does about. There are some estimates of how much computation the human brain does per second. And it's about one E fifteen flop or so it's a bit unclear. But and then it turns out that an H one hundred roughly does on that order of computation. And so you can ask the question of how long does it take for an H one hundred to pay itself back?

• 2:41:20Speaker 02

  If you run the software of the human brain.

• 2:41:22Speaker 01

  If you run the software of the human brain, you can then deploy that in the economy and earn, say, human wages on the order of fifty to a hundred K a year or whatever in the US. And so then it pays itself back because it costs on the order of thirty K per H one hundred. And so you get a doubling time of maybe on the order of a year, right? And so this is like a very quantitatively specific prediction about, you know. And then there's the response, well, you have BOM effects. And they're like, OK, well, what does this mean? Like, does it double? Do you does this predicted doubles every two years or every five years? Like, you need just more assumptions in order to make this a coherent objection.

• 2:42:03Speaker 01

  And so I think a thing that's a little bit, you know, confusing is just that there are these qualitative objections that I agree with. Like, botonex are indeed important, which is part of the reason I'm more skeptical of this software singularity story. But I think this is not sufficient for blocking explosive growth.

• 2:42:23Dwarkesh Patel?

  The other objection that I've heard often, and it might have a similar response from you. Is this idea that a lot of the economy is comprised of O-ring type activities? And this refers to, I think, in the Challenger, the Challenger space shuttle explosion. There was just like one component. I forgot what the exact problem with the O-ring was, but because of that being faulty, the whole thing collapsed.

• 2:42:49Speaker 01

  I mean, I think it's quite funny, actually, because the O-ring model is, is, is taking the product of many, many inputs. And then the overall output is the product of very many things.

• 2:42:59Dwarkesh Patel?

  That's right.

• 2:42:59Speaker 01

  And so, but actually, this is like, pretty optimistic from the point of view of having fewer bottlenecks.

• 2:43:07Speaker 02

  I think we pointed this out before, where, again, talking about software in the singular, I said, like, if it's the product of computer experiments with research.

• 2:43:14Dwarkesh Patel?

  But if one of those products, because of human.

• 2:43:17Speaker 01

  But you have constant marginal product there, right?

• 2:43:19Speaker 02

  No, but, no, but, yeah, but i- if, if one of those products doesn't scale, that is, that doesn't limit, like, yeah, it means you're less efficient at scaling than you otherwise would be. But you can still get a lot.

• 2:43:29Speaker 01

  You can unbounded, you can just have unbounded scaling in, in the O-ring world. So, I, actually, I, I disagree with Tyler, that he's not conservative enough, that he should take his, you know, bottleneck view more seriously than, than he actually is. And yet, I, I disagree with him about, like, the conclusion, and I think that we're gonna get explosive growth once we have AI that can flexibly substitute.

• 2:43:49Dwarkesh Patel?

  I'm not sure I understand, like, there would be entirely new organizations that AIs come up with. We've written a blog post about one such with the AI firms.

• 2:43:56Speaker 02

  Yeah.

• 2:43:57Dwarkesh Patel?

  And you might be a productive worker, or a productive contributor in this existing, the organizations that exist today. In the AI world, many humans might just be like, zero, or even minus, I agree.

• 2:44:14Dwarkesh Patel?

  Why won't that, put that in the multiplication?

• 2:44:18Speaker 01

  But why would you put that in the multiplication? Humans be in the loop there?

• 2:44:21Speaker 02

  Like, you're both saying that humans would have, like, humans would be, like, negatively contributing to output.

• 2:44:27Dwarkesh Patel?

  Yeah.

• 2:44:27Speaker 02

  But then you're also saying that, like, we should put them into the, like, it seems like these.

• 2:44:32Dwarkesh Patel?

  Okay, fair, fair, fair. The main objection often is regulation, and I think we've addressed it implicitly in different points, but might as well just explicitly address, why won't regulation stop this?

• 2:44:42Speaker 02

  Yeah, so for what it's worth, like, we do have, like, a paper where we go over all the arguments for against explosive growth.

• 2:44:48Dwarkesh Patel?

  Yeah.

• 2:44:49Speaker 02

  And regulation, I think, is the one that seems like stronger, says against. Because like the reason it seems strong is because even though we have made arguments before about international competition and like variation of policies among jurisdictions and these strong incentives to adopt this technology both for economic and national security reasons. So I think those are pretty compelling when taken together. But even still, like the world does have a surprising ability to like coordinate on just not pursuing certain technologies.

• 2:45:18Dwarkesh Patel?

  Right. Human cloning.

• 2:45:20Speaker 02

  That's right. So I think like it's hard to be extremely confident that this is not gonna happen. Like I think it's less likely that we're gonna do this for AI than it is for human cloning, but because I think human cloning touches on some other taboos and so on.

• 2:45:37Dwarkesh Patel?

  Right. Also less valuable.

• 2:45:39Speaker 02

  Also less valuable, but and probably less important also for national security in an immediate sense. But at the same time, as I said, it's just hard to rule this out. So I wouldn't say like if someone said, well, I think like there's a 10% or 15%, whatever, 20% chance that like there will be some kind of global coordination and of regulation and that's gonna just be very effective. Maybe it will be enforced through like sanctions on countries that defect or, you know, and then that is gonna like maybe it doesn't prevent AI from being deployed, but maybe it just like slows things down enough that you never quite get explosive growth. Like I don't think that's an unreasonable view if it's like 10% chance. Could be.

• 2:46:17Dwarkesh Patel?

  I think I don't know if there's any, I don't know, do you encounter any other, any other objections?

• 2:46:24Speaker 02

  What should I be hassling you about? Yeah, I mean, some things that we've heard from economists, like again, there was this argument that like people sometimes respond to our argument about explosive growth, which is a, there's an argument about growth levels. So we're saying we're gonna see 30% growth per year instead of 3%. They responded to that with an objection about levels. So they say, well, how much more efficient, how much more valuable can you make like hair dressing or like taking flights or whatever, or going to a restaurant? And like that is just fundamentally the wrong kind of objection. Like we're talking, we're talking about the rate of change. And you're objecting to it by making an argument about the absolute level

• 2:47:07Dwarkesh Patel?

  Right.

• 2:47:07Speaker 02

  of productivity. And as I said before, it is not an argument that economists themselves would endorse if it was made about a slower rate of growth continuing for a longer time.

• 2:47:16Dwarkesh Patel?

  Yeah, yeah, yeah.

• 2:47:17Speaker 02

  So it seems more like a like special pleading, like.

• 2:47:20Dwarkesh Patel?

  I mean, why not just the deployment thing where the same argument you made about AI, where you do learn a lot just by deploying to the world and seeing what people find useful. ChatGPT was an example of this. Why won't a similar thing happen with AI products and services where you're just, if you, one of the components is you put it out to the marketplace and people play with it and you find out what they need. And it like, it clings into the existing supply chain and so forth. Doesn't that take time?

• 2:47:49Speaker 01

  Well, I mean, it takes time, but it is often quite fast. In fact, ChatGPT grew extremely fast, right?

• 2:47:55Dwarkesh Patel?

  Right, but that was just purely digital service, but.

• 2:47:58Speaker 02

  Well, I think the important thing would be like, yeah. One reason to be optimistic is if you think the AIs will literally be dropping remote workers or dropping workers in some cases. If you have robotics, then companies are already kind of experienced at onboarding humans. Like onboarding humans doesn't take like a very long time. Like maybe it takes six months for like a, even in a particularly difficult job for a new worker to like start being productive. Well, that's not that long. So I don't think that would rule out like companies being able to onboard AI workers, assuming that they don't need to make like a ton of new complementary innovation discoveries, like take advantage. I think one way in which current AI systems are being inhibited and the reason

• 2:48:40Speaker 02

  we're seeing the growth maybe be slower than you might otherwise expect is because companies in the economy are not used to working with this new technology. They have to rearrange the way they work in order to take advantage of it. But if AI systems were literally able to substitute for human workers, then, well, the complementary innovations might not be as necessary, so.

• 2:49:00Dwarkesh Patel?

  Actually, this is a good excuse to maybe go to the final topic, which is AI firms. So this is this blog post we wrote together about what it would be like to have a firm that is. Fully automated and the crucial point we were making was that people tend to over emphasize and think of AI from the perspective of how smart individual copies will be. Yep. And if you actually want to understand the ways in which they are superhuman, you want to focus on their collective advantages, which because of biology, we are just precluded from, which are the fact that they can, they can be copied with all their tacit knowledge. Yep. You can copy a Jeff Dean or Elias Satskow or whatever the relevant person is in a different domain. You can even copy Elon Musk and he can be the guy who's every single engineer in the

• 2:49:45Dwarkesh Patel?

  SpaceX rig. And if that's not an efficient way to-

• 2:49:48Speaker 01

  An equivalent of them.

• 2:49:50Dwarkesh Patel?

  Yeah.

• 2:49:52Dwarkesh Patel?

  And if that's, it's not best to have Elon Musk or anything, you just copy the relevant like team or whatever that. Yep. And we have this problem with human firms where there can be very effective teams or groups, but over time their culture dilutes or the people leave or die or get old. And this is one of the many problems that can be solved with these digital firms where you actually, firms right now have two of the three relevant criteria for evolution. They have selection and they have variation, but they don't have high fidelity replication. Yep. And you could imagine a much more fast paced and intense sequence of evolution for firms.

• 2:50:36Dwarkesh Patel?

  Once you, once you have this final piece click in. Yep. And that relates to the onboarding thing where right now, right now, you know, you like, they are just aren't smart enough to be onboarded as full workers. But once they are, I just imagine for my own, like the kinds of things I try to hire for, it would just be such an unlock. Yep. It doesn't even matter, like the salary is a totally secondary. The fact that I can like, this is the skill I need or the set of skills I need and I can have a worker and just like, I can have a thousand workers in parallel if there's something that has a high elasticity of demand, I think is like, probably along with the transformative AI, the most underrated-

• 2:51:17Dwarkesh Patel?

  Tangible thing that like you need to understand about what the future AI society will look like.

• 2:51:22Speaker 02

  Right. I mean, I think there's a point, there's a first point about this like very macroeconomic picture where you just expect a ton of scaling of all the relevant inputs. And I think that is like the first order thing.

• 2:51:31Dwarkesh Patel?

  Yeah.

• 2:51:32Speaker 02

  But then you might have more like micro questions about, okay, like how, like how does this world actually look like? Like, how is it different from a world in which we just have a lot more people and a lot more capital and a lot more like, you know, because it should be different. And then I think these considerations become important.

• 2:51:49Speaker 02

  I think another important thing is just that AIs can be aligned. Like you get to control the preferences of your AI systems in a way that you don't really get to control the preference of your workers.

• 2:51:59Dwarkesh Patel?

  Yeah.

• 2:51:59Speaker 02

  Like your workers, you can just select, you don't really have any other option. But for your AIs, you can like fine tune them, you can like build AI systems which have the kind of preferences that you want. And you can imagine that's like dramatically changing basic problems that determine the structure of human firms. Like, for example, the principal agent problem might go away. Like, this is a problem where the, you as a worker have incentives that are either different from those of your manager or those of the entire firm or those of the shareholders of the firm.

• 2:52:29Dwarkesh Patel?

  I actually think the incentives is a smaller piece of the puzzle. I think it's more about like bandwidth and information sharing where it's often with a large organization, it's very hard to have a single coherent vision.

• 2:52:43Speaker 02

  Yep.

• 2:52:44Dwarkesh Patel?

  And the most social firms we see today is where for an unusual amount of time, a founder is able to keep their vision instilled in the organization, like SpaceX or Tesla are examples of this. People talk about Nvidia this way, but just imagine a future version where there's this hyper-inference scaled mega Jensen, who you're spending hundred billion dollars a year on inference on, and copies of him are constantly, you know, like writing every single press release and reviewing every pull request and answering every customer service request and so forth. And monitoring the whole organization, making sure it's like proceeding along a coherent vision and getting merged back into the hyper, uh, hyper Jensen.

• 2:53:26Speaker 02

  Hyper Jensen.

• 2:53:28Dwarkesh Patel?

  Megagens and whatever. Yeah.

• 2:53:31Speaker 02

  Yeah, I agree. That's a bigger deal. At the same time, I would point out that like part of the reason why it's important to have like a coherent vision and culture and so on and human companies might be that there's incentive problems exist otherwise. Like, I mean, I wouldn't rule that out, but I agree that the like, aside from the overall macroeconomic thing, I think the fact that they can be replicated is probably the biggest.

• 2:53:50Dwarkesh Patel?

  That's right. That's right. That's right. Yeah.

• 2:53:51Speaker 02

  Like, that also enables additional sources of economies of scale, where if you have like twice the number of GPUs, you can run not only twice the number of copies of your old model, but then you can train a model that's even better. So you double your training computers and your inference compute, and that means you not only double, like, you don't get just twice the number of workers you would have had otherwise. You get more than that, because they are also smarter, right? Because you spend more training computers.

• 2:54:15Dwarkesh Patel?

  Right.

• 2:54:15Speaker 02

  So then that is additional source of economies of scale. And then there's this benefit that you can, like, for humans, you, like, every human has to learn things from scratch, basically. Like, they are born and then they have a certain, a lot of lifetime learning that they have to do. So in human learning, there is a ton of duplication. While for an AI system, it could just learn once. It could just have one huge training run, which are tons of data, and then that run could be deployed everywhere.

• 2:54:39Dwarkesh Patel?

  Yeah.

• 2:54:40Speaker 02

  So that's like another massive advantage that the AIs have over humans.

• 2:54:43Dwarkesh Patel?

  Yeah. This, maybe we'll close up with this one debate we've had, often had offline, which is, will central planning work with these economies of scale?

• 2:54:53Speaker 02

  So I would say that, I mean, again, the question of like, will it work?

• 2:54:56Dwarkesh Patel?

  Will it be optimal?

• 2:54:57Speaker 02

  Right. So, I mean, my guess is probably not optimal, but like, I think it's hard, like, I don't think anyone has, like, thought this question through in, like, a lot of.

• 2:55:10Speaker 01

  So it's worth thinking about just like why one might expect

• 2:55:13Dwarkesh Patel?

  Yeah.

• 2:55:13Speaker 01

  central planning to be slightly better in this world.

• 2:55:16Speaker 02

  Right.

• 2:55:16Speaker 01

  So, so one consideration is just communication bandwidth being potentially much, much greater than it is today. And then, like, in the current world, the information gathering and the information processing are like co-located. Like, humans observe and also process what they observe.

• 2:55:39Speaker 01

  In an AI world, you can disaggregate that.

• 2:55:42Dwarkesh Patel?

  That's actually a really interesting point, yeah.

• 2:55:44Speaker 01

  So you can have the sensors and not do much processing, but just collect and then process centrally. And that processing centrally might make sense for a bunch of reasons. And you might get economies of scale from having more GPUs that produce better models and also be able to, you know, think more deeply about what it's seeing.

• 2:56:05Dwarkesh Patel?

  It's weird to know that certain things already work like this. For example, Tesla FSD, it will benefit from the data collected at the periphery from millions of miles of driving. And then the improvements which are made as a result of this.

• 2:56:18Speaker 01

  Centrally directed, it's like coming from this HQ being like, we're gonna push an update.

• 2:56:23Dwarkesh Patel?

  That's right.

• 2:56:24Speaker 01

  And so you do get some of this more centralized.

• 2:56:27Dwarkesh Patel?

  And it can be a much more intelligent form than just whatever gradient averaging that they, I mean, I'm sure it's more sophisticated than Tesla, but it can be a much more like deliberate, intelligent.

• 2:56:35Speaker 02

  That's right.

• 2:56:36Dwarkesh Patel?

  Update.

• 2:56:36Speaker 01

  So that's one reason to expect. And the other reason, I guess, is just having like current leaders or CEOs don't have bigger brains than the workers do, maybe a little bit.

• 2:56:49Dwarkesh Patel?

  I don't know if you want to open that.

• 2:56:51Speaker 01

  But not by orders of magnitude. And so you could have just orders of magnitude more scaling of the size of the models that are doing the planning than the people or the agents or workers doing the actions.

• 2:57:04Speaker 02

  And I think a third reason is the thing about like the incentive thing. Or like you wouldn't face this problem that like part of the reason you have a market is that it gives people the right kind of incentives. But you might not need that as much if you're using AI. So I think there's an argument that if you just list the traditional arguments people have made against like, why does central banking not work, then you might expect them to become weaker. Now, I think that is still, like, there's a danger when you're doing that kind of analysis to fall into the same kind of like partial equilibrium analysis where you're like only considering some factors and then you're not considering other things. Like, for example, you consider.

• 2:57:43Speaker 01

  Get more complex. You just have a much bigger economy. And so like, on the one hand, your ability to. Kind of kind of collect information and process it improves. But also, the need for doing that also increases as things become more complex.

• 2:57:58Dwarkesh Patel?

  And I mean, one way to illustrate that is like, imagine if Apple, the organization today, with all its compute and whatever, was tasked with managing the economy of work. Right? I think it like, it actually could centrally plan the economy. Maybe like, the economy of work might work even better as a result. But like, Apple, as it exists today, cannot manage the economy, the world economy as it exists today.

• 2:58:18Speaker 02

  That's right.

• 2:58:19Dwarkesh Patel?

  I mean, that's a good, yeah. Yeah. All right, actually, this will be the final question.

• 2:58:24Dwarkesh Patel?

  Look, one of the things that makes AI so fascinating is that there's no domain of human knowledge that is irrelevant to studying it, because what we're really trying to do-

• 2:58:33Speaker 01

  I don't know about that.

• 2:58:35Dwarkesh Patel?

  Fine, fine, fine. There's no serious domain of human knowledge.

• 2:58:40Speaker 01

  That's, that's better.

• 2:58:42Dwarkesh Patel?

  That is not relevant to studying it, because you're just fundamentally trying to figure out what a future society will look like. And so it's like, obviously, computer science is relevant, but also economics, as we've been discussing, history, and how to understand history, and many other things we've been discussing, right? Especially if you have longer timelines and there is enough time for somebody to pursue a meaningful career here. What would you recommend to somebody? Because both of you are quite young. I mean, you especially, I guess, but like, both of you. So it's like, this is not, you would think this is the kind of thing which requires crystallized intelligence or whatever. Especially given what we said earlier about, look, as we get more knowledge, we're going to have to factor in what we're learning into building a better model of what's going to happen to the world. And if somebody is interested in this kind of career that you both have, what advice do you have for them?

• 2:59:28Speaker 02

  Yeah, that's a hard question.

• 2:59:32Speaker 02

  I mean, I'm not sure. Like, I think the, there is an extent to which it's, like, it's difficult to deliberately pursue, like, the implicit strategy that we would have pursued. Like, it probably works better if it's spontaneous and like, more driven by curiosity and interest than, like, you make a deliberate choice. Okay, like, I'm, I'm just going to learn about a bunch of things so that I can contribute to the discourse on AI. I, I would think that strategy is probably less effective. At least, I haven't seen anyone who deliberately used that strategy and then was successful. It seems like-

• 3:00:04Dwarkesh Patel?

  Yeah, I guess I, not that I've contributed to the discourse directly, but maybe facilitated other people contributing.

• 3:00:11Dwarkesh Patel?

  I guess it wasn't deliberate strategy on my end, but it was a deliberate strategy to do the podcast, which inadvertently gave me the opportunity to learn about multiple fields.

• 3:00:19Speaker 01

  Yeah, so given like if you're already interested and curious and reading a bunch of things and studying a bunch of things and thinking about these topics, on the margin, there are a bunch of things you can do to like make you more productive at having this of making some contributions to this. And I think just speaking to people and writing your thoughts down and finding like especially useful people to chat with and collaborate with, I think that's very useful.

• 3:00:44Dwarkesh Patel?

  Right.

• 3:00:45Speaker 01

  So just seek out people that have similar views and you're able to have very high bandwidth conversations with and seemingly, you know, and kind of make progress on these topics. And I think that's just pretty useful.

• 3:01:00Dwarkesh Patel?

  But I mean, like, how exactly like should they DM you? Like, how do they get?

• 3:01:04Speaker 02

  Yeah, sure.

• 3:01:05Speaker 01

  And like, I don't know, set up signal chats with your friends or whatever.

• 3:01:09Dwarkesh Patel?

  Yeah, yeah. I've done a lot of, actually, it's like crazy how much I've gotten out of that, but.

• 3:01:13Speaker 02

  Yeah, I mean, I think like the one of, in fact, one advice I would give to people in general, even if they're not like thinking about AI specifically, but I think it's also helpful for that. It's just that people should just be much more aggressive about reaching out.

• 3:01:25Dwarkesh Patel?

  Like, that's right.

• 3:01:25Speaker 02

  Yeah, like a lot of the communication that like, maybe people have an impression that if you like reach out to someone who looks really important and like they're not going to respond to you. But like if what you send to them is just interesting and like high quality, then it's very, very likely that they will respond.

• 3:01:41Speaker 01

  Yep.

• 3:01:42Speaker 02

  Like, there's like a lot more edge there that you can get, which is just being more aggressive and like less ashamed or something of like looking dumb. Like, that's the main advice I would give because if you want to be productive, then again, like there are these complementarities and so on, like different, you know, you need to be part of like some community or some organization.

• 3:02:02Dwarkesh Patel?

  And it goes back to the thing about reasoning alone not being that helpful.

• 3:02:05Speaker 01

  Yeah, yeah, exactly.

• 3:02:05Dwarkesh Patel?

  It's just like, other people have thought a long time and have randomly stumbled upon useful ideas that you can take advantage of.

• 3:02:11Speaker 02

  That's right. So you should just like try to place yourself in a situation where you can become part of something larger.

• 3:02:17Dwarkesh Patel?

  Right.

• 3:02:18Speaker 02

  Which is at working on it from, that's just a more effective. A way of contributing and to do that you have to well let people know.

• 3:02:25Dwarkesh Patel?

  That's right, that's right. And I think just coming to the Bay Area is especially for instance in AI in particular.

• 3:02:30Speaker 02

  Yeah, coming to Bay Area is nice, just post like just writing things and like posting them in a way people can see them, just aggressively reaching out to people with like interesting comments.

• 3:02:38Speaker 01

  Provided your like thoughts are interesting and so on.

• 3:02:41Dwarkesh Patel?

  I mean, but they probably are like in many cases, I think it's like my thoughts weren't, my thoughts still might not be interesting, but people tolerate my cold emails and are like, you know, will still like do collaborate with me and so forth.

• 3:02:53Speaker 02

  Yep.

• 3:02:56Dwarkesh Patel?

  The other thing I've noticed, tell me this is actually the wrong pattern or the wrong, yeah, with people like you or with Carl or something is that as compared to a general person who's intellectually curious or reading widely, you tend to focus much more on key pieces of literature than say, I'm gonna go read the classics or just generally read like it's like I'm gonna, I'm gonna just like put like a ton more credence in something like the Roamer paper and a normal person might not even read the, a normal person who's like intellectually curious would just like not be reading key pieces of literature.

• 3:03:30Speaker 02

  Yeah, I think like you have to be very mindful of the fact that you have a very limited amount of time, like you're not an AI model. So you have to kind of aggressively prioritize what you're going to spend your time read, like reading.

• 3:03:42Speaker 01

  Even AI models don't prioritize that heavily. They read Reddit mostly or like a large part of their corpus is.

• 3:03:48Dwarkesh Patel?

  Yeah, key pieces of empirical literature at least, at least among you guys. I mean, it might not be the most productive thing in general, but.

• 3:03:54Speaker 01

  I think that's useful. I also just think it's useful to read Twitter. I think we were having this conversation about people often say that they should like they're spending too much time reading Twitter and they wish they spend more time reading archives. But actually, like the amount of information per unit time you get reading Twitter is often just much higher.

• 3:04:10Dwarkesh Patel?

  Yeah.

• 3:04:11Speaker 01

  And and it's just much more productive for them to read Twitter. I think there are key pieces of literature that are kind of important. And I think it's, it's useful to figure out what people who have spent a lot of time thinking about this find important in their worldview. So you know, in AI, this might be, you know, key papers, like, I don't know, like the Andy Jones paper about scaling loss for inference is like a big thing. And in economics, like this Roamer paper or the paper on.

• 3:04:48Speaker 01

  Uh, explaining long run population from Kramer or from uh, from um, the, the David Rudman and so on. I think, I think just like if, if people who you think are having really good, uh, who think very well about this, suggests a certain paper and they like highly recommend it, then I think you should take that seriously and actually read those papers.

• 3:05:09Dwarkesh Patel?

  And for me, it's been especially helpful to, instead of just skimming a bunch of things, just like really stop on like, yeah, if there's a key piece of literature or for, yeah, to in order to, for example, understand the transformer, I like, there's always a car product lectures, but one resource that was really useful is the Anthropic's original transformer circuit paper. And I just like, just spending a day on that paper instead of skimming it and making a bunch of space repetition cards and so forth, was much more useful than just like generally reading widely about AI.

• 3:05:42Speaker 02

  Yeah, I think it's just much more important here to, if you want to prioritize things correctly to be, again, to be part of a community or to be getting input from a community or get from people who have thought a lot and have a lot of experience about what is important and what is not.

• 3:05:56Dwarkesh Patel?

  Yeah.

• 3:05:57Speaker 02

  Like this is true even in academic field. So if you want to do math research, but you're not part of like a graduate program, you're not at, at a university where there are tons of people who like do math research all day for many years, then you're not even gonna know like what are the open problems that I should be working on? What is reasonable to attack? What is not reasonable to attack? Like what, what papers in this field are important, contain important techniques? You're just gonna have no idea.

• 3:06:22Dwarkesh Patel?

  Right.

• 3:06:22Speaker 02

  So it's very important to be like plugged into that video information somehow.

• 3:06:26Dwarkesh Patel?

  Yeah. But how did you know all this shit before being plugged in? Cause you weren't talking to anybody in Ankara.

• 3:06:30Speaker 02

  I mean, you, you don't need to talk, I mean, the internet is a pretty, like pretty useful thing in this respect. And you don't need to necessarily talk to people. Like you can get a lot of benefit from reading. Like, like you just need to identify, okay, like who are the people who seem like consistently most interesting? And you can also get a lot of benefit or maybe you found one person and then uh, often that person will know some other people who are interesting.

• 3:06:51Dwarkesh Patel?

  Right.

• 3:06:52Speaker 02

  And then you can like start tracing the social network. So for example, maybe um, I don't know, like one example I can give, which I think is actually accurate is like, maybe you know about Daniel Ellsberg. So you like look for a podcast where he appears on it. You notice that he's appeared on 80,000 Hours Podcast, which he has. And then you notice like there are some other guests on 80,000 Hours Podcast. So maybe there's Brian Kaplan, who has also appeared on the podcast. And then maybe Robin Hanson has also appeared on the podcast. And then, you know, maybe there are some people those other people know. And then like just tracing that kind of social network and like figuring out who to listen to like that. I think that can be.

• 3:07:25Speaker 01

  And I think you're doing a very big service to making that possible. Where like, I think your selection is often very good.

• 3:07:33Dwarkesh Patel?

  I'm actually curious if you're offline, what I got wrong. Well, actually, I think I know the answer to that.

• 3:07:38Speaker 01

  So, so, you know, and I think that that makes it a bunch easier to track. Like, you know, who are the people doing the most interesting thinking on on various topics?

• 3:07:47Dwarkesh Patel?

  That's right. Cool. I think that's a good place to end with you praising me.

• 3:07:53Dwarkesh Patel?

  Okay. I again, I highly recommend people follow Epoch. There's a great weekly newsletter, Gradient Updates, which I mean, like, people plug newsletters, but this is like, I can't believe this is a thing that comes out on a weekly basis. And it's like, it's like, anyways. And you now have a new podcast, which I will not plug as a competitor.

• 3:08:16Dwarkesh Patel?

  But you can check it out.

• 3:08:17Speaker 01

  Thanks for lending your studio to our podcast. That is very generous.

• 3:08:24Dwarkesh Patel?

  Anyways, cool. Thanks, guys.

• 3:08:26Speaker 01

  All right, thanks.