Is AGI Far? With Robin Hanson, Economist at George Mason University and Researcher at Future of Humanity Institute

In this episode, Nathan sits down with Robin Hanson, associate professor of economics at George Mason University and researcher at Oxford’s Future of Humanity Institute. They discuss the comparison of human brains to LLMs and legacy software systems, what it would take for AI and automation to significantly impact the economy, our relationships with AI and the moral weight it has, and much more. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api LINKS: Robin’s Book, The Age of Em: https://ageofem.com/ Robin’s essay on Automation: https://www.overcomingbias.com/p/no-recent-automation-revolutionhtml Robin’s Blog: https://www.overcomingbias.com/ AI Scouting Report: https://www.youtube.com/watch?v=0hvtiVQ_LqQ&list=PLVfJCYRuaJIXooK_KWju5djdVmEpH81ee&pp=iAQB Dr. Isaac Kohane Episode: https://www.youtube.com/watch?v=pS5Vye671Xg SPONSORS: The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. X/SOCIAL: @labenz @robinhanson (Robin) @CogRev_Podcast TIMESTAMPS (00:00) Preview (07:10) Why our current time is a “dream time” and the move back to a Malthusian world (13:30) What sort of world should we be striving for? (13:40) Sponsor - Brave (17:50) Distinguishing value talk from factual talk (18:00) Comparing and contrasting Ems to LLMs (22:30) The comparison of human brains to legacy software systems (30:52) Sponsor - Netsuite (41:01) AIs in medicine (53:30) A several century innovation pause (55:30) Achieving full human level AI in the next 60-90 years (1:03:55) Chess and routine benchmarks not a good predictor of AI performance in the economy (1:07:44) Reaching and exceeding human-level AI in the next 1000 years (1:11:40) Losing technologies tied to scale economies (1:12:00) Why AI is hard to maintain in the long run (1:12:20) Standard deviation in automation (1:14:05) Computing power grows exponentially but automation grows steadily (1:15:50) AI art generation and deepfakes (01:21:42) The economics of AI-powered coding (1:33:51) Merging LLMs (1:36:02) Rot in software and the human brain (1:40:18) Parallelism in LLMs and brain design (1:41:00]) Moral weight for AIs, enslavement, and cooperation with AI (1:47:10) What would change Robin’s mind about the future (1:49:18) Wrap This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We're on this upward growth trajectory. We have the potential to taking a big chunk of the universe and doing things with it. And I'm excited by that potential. So I want us to keep growing. And I see how much we've changed to get to where we are. My book, Age of M, is about brain emulations. So that's where you take a particular human brain and you scan it and find spatial chemical detail, where you fill in for each cell, a computer model of that cell. And if you've got good enough models for cells and a good map of the brain, then basically the IO of this model should be the same as the IO of the original brain. If we can get full human level AI in the next 16 to 90 years with the progress, then this population decline won't matter so much because we will basically have AIs take

• 0:42Speaker 00

  over most of the jobs and then that can allow the world economy to keep growing.

• 0:46Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. My guest today is Robin Hanson, professor of economics at George Mason University and author of the blog, Overcoming Bias, where Robin has published consistently on a wide range of topics since 2006 and where Elias Yudkowsky published early versions of what has become some of his most influential writing on AI.

• 1:31Nathan Labenz?

  Robin is an undeniable polymath, whose approach to futurism is unusually non-romantic. Rather than trying to identify value buddies, Robin aims to apply first principles thinking to the future and to describe what is likely to happen without claiming that you should feel any particular way about it.

• 1:51Nathan Labenz?

  I set this conversation up late last year after my deep dive into the new Mamba state space model architecture. Because Robin's 2016 book, The Age of M, which analyzes a scenario in which human emulations can be run on computers, suddenly seemed a lot more relevant. My plan originally was to consider how his analysis from the age of M would compare to similar analyses for a hypothetical age of LLMs, or perhaps even an age of SSMs. In practice, we ended up doing some of that, but for the most part, took a different direction, as it became clear early on in the conversation that Robin was not buying some of my core premises. Taking the outside view, as he's famous for doing, and noting that AI experts have repeatedly

• 2:35Nathan Labenz?

  thought that they were close to AGI in the past, Robin questions whether this time really is different, and doubts whether we are really close to transformative AI at all. This perspective naturally challenged my worldview, and I listened back to this conversation in full to make sure that I wasn't missing anything important before writing this introduction. Ultimately, I do remain quite firmly convinced that today's AIs are powerful enough to drive economic transformation. And I would cite the release of Google's Gemini 1.5, which happened in just the few short weeks between recording and publishing this episode, as evidence that progress is not yet slowing down. Yet, at the same time, Robin did get me thinking more about the disconnect between feasibility

• 3:19Nathan Labenz?

  and actual widespread implementation and automation. Beyond the question of what AI systems can do, there are also questions of legal regulation, of course, and perhaps even more importantly, just how eager people are to use AI tools in the first place. When Robin reported that his son's software firm had recently determined that LLMs were not useful for routine application development, I was honestly kind of shocked. Because if nothing else, I'm extremely confident about the degree to which LLMs accelerate my own programming work. Since then, though, I have heard a couple of other stories, which, combined with Robin's, helped me develop, I think, a bit better theory of what's going on.

• 4:00Nathan Labenz?

  First, an AI educator told me that failure to form new habits is the most common cause of failure with AI in general. In his courses, he emphasizes hands-on exercises because he's learned that simple awareness of AI capabilities does not lead to human behavioral change. Second, a friend told me that his company hosted a Microsoft GitHub salesperson for a lunch hour demo, and it turned out that one of their own team members had far more knowledge about GitHub Copilot than the rep himself did. If Microsoft sales reps are struggling to keep up with Copilot's capabilities, we should perhaps adjust our expectations for the rest of the economy.

• 4:41Nathan Labenz?

  And third, in my own experience helping people address process bottlenecks with AI, I've repeatedly seen how unnatural it can be for people to break their own work down into the sort of discrete tasks that LLMs can handle effectively today. Most people were never trained to think this way, and it's going to take time before it becomes common practice across the economy. All this means that change may be slower to materialize than those of us on the frontiers of AI adoption might expect. And while that does suggest more of an opportunity and indeed advantage for us in the meantime, on balance, I do have to view it as a negative sign about our preparedness and our ability to adapt overall. Regardless of your views, and I do suspect that most listeners will find themselves

• 5:25Nathan Labenz?

  agreeing with me more than with Robin, his insights are always thought-provoking, and I think you'll find it very well worthwhile to engage with the challenges that he presents in this conversation. As always, if you're finding value in the show, we would appreciate it if you'd share it with friends, post a review on Apple Podcasts or Spotify, or just leave a comment on YouTube. And of course, I always love to hear from listeners, so please don't hesitate to DM me on the social media platform of your choice. Now, I hope you enjoy this conversation with Professor Robin Hanson.

• 5:58Speaker 02

  Robin Hanson, professor of economics at George Mason University and noted polymath. Welcome to the cognitive revolution.

• 6:04Speaker 00

  Nice to meet you, Nathan. Let's talk.

• 6:07Speaker 02

  I'm excited about this. So I have followed your work for a long time. It's super wide ranging and always very interesting. People can find your thoughts on just about everything, I think, over the years on overcoming bias, your blog. But today I wanted to revisit what I think is one of your destined to be perhaps one of your most influential works, which is the book, The Age of M, which came out in 2016 and envisions a future, which basically amounts to putting humans on machines, and we can unpack that in more detail, and then explores that in a ton of different directions. Where we actually are now as we enter into 2024 is not exactly that, certainly.

• 6:49Speaker 02

  But I've come to believe recently that it's maybe bending back a little bit more toward that, certainly more than my expectations a year ago. So I've revisited the book and I'm excited to bring a bunch of questions and kind of compare and contrast your scenario versus the current scenario that we seem to be evolving into.

• 7:07Speaker 00

  Okay, let's do it.

• 7:09Speaker 02

  One big theme of your work always, I think, is that we live in this strange dream time and that our reality as modern humans is quite different than the reality of those that came before us and likely those that will come after us for some pretty fundamental reasons. Do you want to just sketch out your kind of big picture argument that our times are exceptional and not likely to go on like this forever?

• 7:35Speaker 00

  The first thing to notice is that we were in a period of very rapid growth, very rapid change, which just can't continue for very long on a cosmological time scale. 10,000 years would be way longer than it could manage. And therefore, we're going to have to go back to a period of slower change. And plausibly, then a period of slower change will be a period where population could grow faster relative to the growth rate of the economy in the universe. And therefore, we will move back more toward a Malthusian world if competition remains, such as almost all our ancestors were until a few hundred years ago. So we're in this unusual period of being rich per person and in very rapid change.

• 8:22Speaker 00

  And also sort of globally integrated. That is, our distant ancestors were fragmented culturally across the globe. And, you know, each talked to a small group of people near them. And our distant descendants will be fragmented across the universe. And they won't be able to talk all across the universe, you know, instantaneously. So future culture and past culture were both very fragmented. And we are in a period where our entire civilization can talk rapidly to each other. The, you know, the time delay of communication is very small compared to the doubling time of our very rapid growth economy. So we are now an integrated civilization. We're rich, growing very fast.

• 9:03Speaker 00

  And there's a number of consequences of being rich, which is that we don't have to pay that much attention to functionality. Those we're not pressured to do what it takes to survive in the way our ancestors and our descendants would be. So we can indulge our delusions or whatever other inclinations we have. They aren't disciplined very rapidly by survival and functionality. That makes us a dream time. That is, our dreams drive us. Our abstract thoughts, our vague impressions, our emotions, our visions.

• 9:42Speaker 00

  We, we do things that are dramatic and exciting and meaningful in our view, according to this dream time mind we have, which isn't, again, that disciplined by functionality. That is, the mind we inherited from our distant ancestors. It was functional there. It was disciplined there. We're in a very different world, but our mind hasn't changed to be functional in this world. And so we are, you know, expressing this momentum of what we used to be in this strange new world. That's the dream time.

• 10:18Speaker 02

  So let me just try to rephrase that or frame it slightly differently and tell me if you agree with this framing. I would maybe interpret it as we're maybe in a punctuated equilibrium sort of situation where we're, we're in the transition from one equilibrium to another. There have probably been. However, many of these through history, not like a huge number, but a decent number. I think of, you know, such phrases as the Cambrian explosion, perhaps as another dream time. These moments happen when some external shock happens to the system, whether that's like an asteroid that takes out a lot of life or human brains, you know, come on the scene. And there's a period in which the normal constraints are temporarily relaxed.

• 11:01Speaker 02

  But then in the long term, there's just like no escaping the logic of natural selection. Is that, is that basically the framework?

• 11:08Speaker 00

  So your analogy of the Cambrian explosion could be, you know, we discovered multicellularity. We discovered being able to make large animals. And that was happened at a moment. There was the moment of multicellularity. And then evolution took time to adapt to that new opportunity. And the Cambrian explosion is the period of adaptation. Then after the Cambrian explosion, we've adapted to that new opportunity and then we're more in a stasis. And then you're imagining this period of adaptation to a sudden change. But for humans today, we keep having sudden changes and they keep coming fast. And so there wasn't this one thing that happened 300 years ago or 10,000 years ago that we're slowly adapting to.

• 11:51Speaker 00

  We keep having more big changes that keep changing the landscape of what it is to adapt to. So we won't see this slow adaptation to the new thing until we get a stable new thing, which we haven't gotten yet. We, things keep changing.

• 12:06Speaker 02

  I want to maybe circle back in a minute to what would be the conditions under which things would restabilize. I think, I guess the M scenario is one of them, but there may be others that might even be more imminent at this point. Before doing that, I just wanted to touch on another big theme of your work, which is, and I really appreciate how you introduced the book this way with the idea that I'm just trying to figure out what is likely to happen in this scenario. I'm not telling you, you should like it. I'm not telling you, you should dislike it. I'm not trying to judge it. I'm just trying to extrapolate from a scenario using the tools of science and social science to try to figure out what might happen.

• 12:47Speaker 02

  I love that and I, I try to do something similar with this show around understanding AI. I think there's so much, you know, emotional valence brought to so many parts of the discussion. And I always say, we need to first kind of figure out what is. And I, even in the current moment, like what capabilities exist, you know, what, what can be done? What, what is still out of reach of current systems before we can really get serious about what ought to be done about it? I guess I'd invite you to add, you know, any additional perspective to that. And then I'm also curious, like, I think that's very admirable, but could you give us a little window into your own kind of biases or preferences? Like what sort of world do you think we should be striving for?

• 13:30Speaker 02

  Or do you think that's just so futile to even attempt to influence against these, you know, grand constraints that it doesn't matter?

• 13:37Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 13:42Speaker 02

  Pretty much all big grand talk is mostly oriented around people sharing values. That's what people want to do when they talk big politics, when they talk world politics or world events, when they went, when they talk the future, people want to jump quickly to, do I share your values? Here's my values. What are your values? Do we agree on values? Are we value buddies? And people are so eager to get to that, that they are willing to skip over the analysis of the details, say, if you want to talk about, I don't know, the war in Ukraine, people want to go, which side are you on and who, you know, are we, do we have the right values?

• 14:24Speaker 02

  And then they don't care to talk about like, who has how much armaments that will run out soon, or who can afford what, or what they, you know, all those details of the war. They don't want to go there. They just want to go to the values and agree about them.

• 14:37Speaker 00

  And that happens in the future too, futurism too. People just wanna jump to the values. So for the purposes people have, they're doing roughly the right thing. They don't really care about the world and they don't really care about the future. What they care about is be finding value buddies. Or if you find a value conflict, having a value war. That's what people just wanna do. And so if you actually wanna figure out the world or national politics or national policy, or you wanna figure out the future, you really have to resist that and you have to try to pause and, you know, go through an analysis first, a neutral analysis of what the options are, what the situation is. I mean, I am afraid literally that if I express many values,

• 15:20Speaker 00

  that that the discussion will just go there and you'll never talk about anything else. And that's why I resist talking about that. But I think, you know, my simplest value with respect to the future is, I really like the fact that humanity has grown and achieved vast things compared to where it started. We're on this upward growth trajectory. We have the potential to taking a big chunk of the universe and doing things with it, and I'm excited by that potential. So my first cut is I want us to keep growing. And I see how much we've changed to get to where we are. And I can see that had people from a million years ago insisted that

• 16:00Speaker 00

  their values be retained, maintained and that the world be familiar and comfortable to them, if they'd been able to enforce that, we would not have gotten where we are now. That would have prevented a lot of change. So I kinda see that if I want us to get big and grand, I'm gonna have to give a lot on how similar the future is to me and my world. Uh, I'm gonna have to compromise a lot on that. I just don't see any way around that. So I get it that if you want the future to be really comfortable for you and to share a lot of your values and your styles, you're gonna have to prevent it from changing. And you may have a shot at that and I would not like that, but you might. You know, so again, even as part of the value framework,

• 16:42Speaker 00

  even when I talk values with you, I want to be clear to distinguish my value talk from the factual talk. I'm gonna be happy to tell you what it would take for you to get your values, even if they aren't mine. So maybe we should talk about the facts of LLMs. So you want to go there in terms of comparing M's and LLMs, right? So first of all, our audience, we should say for our audience, my book Age of M is about brain emulations. So that's where you take a particular human brain and you scan it and find spatial chemical detail to figure out which cells are where connected to what other cells through what synapses. You make a map of that and then you make a computer model that matches that map,

• 17:26Speaker 00

  where you fill in for each cell a computer model of that cell. And if you've got good enough models for cells and a good map of the brain, then basically the IO of this model should be the same as the IO of the original brain, which means you could hook it up with artificial eyes, ears, hands, mouth. And then it would behave the same as the original human would in the same situation, in which case you can use these as substitutes for humans throughout the entire economy. And then my exercise of the Age of M book was to figure out what that world looks like. And a primary purpose was to actually be able to show that it's possible to do that sort of thing. It's possible to take a specific technical assumption and work out a lot of consequences. And many people have said they didn't want so many details. They'd rather have fiction or something else.

• 18:11Speaker 00

  But I was trying to prove how much I could say. And I hope you'll admit, I proved I could say a lot. And that almost no other futurist's work does that. And so I'm trying to inspire other futurists to get into that level of detail, to try to take some assumptions and work out a lot of consequences. So that's my book, The Age of M. You'd like us to compare that to current large language models and to think about what we can say about the future of large language models. So in my mind, the first thing to say there is, well, an M is a full human substitute. It can do everything a human can do, basically. A large language model is not that yet.

• 18:55Speaker 00

  So a key question here would be, how far are we going to go in trying to imagine a descendant of a large language model that is more capable of substituting for humans across a wide range of contexts? We stick with current large language models. They're really only useful in a rather limited range of contexts. And so if you're going to do forecasting of them, it's more like forecasting the future with a microwave oven or something. You think about, well, where can you use a microwave oven? And how much will it cost? You know, what other heating methods will it displace? And what sort of inputs would be compliments to that? It would be more of a small-scale futury forecasting exercise.

• 19:37Speaker 00

  Whereas the age of M was purposely this very grand exercise because the M's actually change everything. Whereas most futurism, like if you're trying to analyze consequences of microwave oven, you have a much more limited scope because in fact it will have a limited impact. So that would be the question I have for you first, which is, are we going to talk about the implications of something close to the current large language models? Or are we going to try to imagine some generalized version of them that has much wider capabilities?

• 20:10Speaker 02

  Yeah, very good question. I think maybe two different levels of this would be instructive. One of the key things that jumps out, and I think a lot of stuff flows from, is the assumption that M's can be copied cheaply, paused and stored indefinitely cheaply, but not understood very well in terms of their internal mechanism, right? Very much like this similar understanding to what we have of the brain where we can kind of poke and prod at it a little bit, but we really don't have a deep understanding of how it works. We can't do like very localized optimizations. But we do have this like radical departure from the status quo, which is you can infinitely clone them, you can infinitely freeze and store them.

• 20:54Speaker 02

  And so this creates like all sorts of elasticities that just don't exist in the current environment.

• 21:00Speaker 00

  So a number of those features are going to be general to anything that can be represented as computer files and run on a computer. So any form of artificial intelligence will be of the sort in general that you could have a digital representation of, archive it, make a copy of it, pause it. Run it faster or slower, that's going to be just generically true of any kind of AI, including M's. The ability to sort of modify it usefully, I mean, yes, with human brains initially, they're just a big mess, you don't understand them. But honestly, most legacy software systems are pretty similar. So today, large legacy software systems, you mostly have to take them as they are.

• 21:44Speaker 00

  You can only make modest modifications to them. That's close to what I'm assuming for M's. So I'm actually not assuming that they are that different from large legacy software systems. They're just a big mess. That even though you could go look at any one piece and maybe understand it, that doesn't really help you usefully in modifying the entire thing. You basically have to take the whole thing as a unit and can only make some minor changes. But you can copy the whole thing, you can run it fast or slow. You can move it at speed, transfer it the speed of light around the earth, through the universe. You know, those things are true of pretty much any AI that could be represented and as a computer file, run on a computer.

• 22:27Speaker 02

  Yeah, I think these these dimensions are a really useful way to break this down. And I I took some inspiration from you in a presentation that I created called the AI scouting report, where I have the tail of the cognitive tape that compares human strengths and weaknesses to LLM strengths and weaknesses. And I think for the purposes of this discussion, maybe we we might even have like four different kind of things to consider. One is humans, second would be M's. Third is, let's say, transformer language models of the general class that we have today. Although I think we can predictably expect at a minimum that they will continue to have longer context windows

• 23:08Speaker 02

  and have, you know, generally more pre-training and generally more capability, at least, you know, within a certain range. And then the fourth one that I'm really interested in and has been kind of an obsession for me recently is the new state space model paradigm, which actually has some things now in common again with the humans and the M's that the transformer models lack. The state space models, you know, this has been, of course, you know, a line of research that's been going on for a couple of years, kind of in parallel with transformers. Transformers have taken up, you know, the vast majority of the energy and the public, you know, focus because they have been the highest performing over the last couple of years.

• 23:53Speaker 02

  But that has maybe just changed with a couple of recent papers, most notably one called Mamba, that basically shows parity, rough parity with the transformer on kind of your standard language modeling tasks, but does have like a totally different architecture that I think opens up like some notably different strengths and weaknesses. Whereas the transformer really just has the weights and then the sort of next token prediction. The state space model has this additional concept of the state, which is, and I recall

• 24:33Speaker 02

  from the the book, you sort of say, you know, taking an information processing lens to the human or, you know, where you spend more of your focus is on the M, you have the current state plus some new input information, you know, sensory or whatever. And then that propagates into some action, some output and a new internal state. And that I think is really the heart of what the new state space models do is that they they add that additional component where they have not only the weights, like a transformer has static weights, but they also have this state which is of a fixed size, evolves through time and is something that gets output, you know, at each kind of inference step so that

• 25:18Speaker 02

  there is this internal state that, you know, propagates through time and can kind of change and, you know, have long history. I think it is likely to bring about a much more integrated medium and long-term memory than the transformers have and create more sort of long-episode conditioning where these these models I think will be more more amenable to like employee onboarding-style training, you know, which is something also that the M's have in your, you know, in your scenario, right? You can kind of train a base M to be an employee for you. You can

• 26:00Speaker 02

  even put it in that, you know, mental, get it to that mental state where it's like really excited and ready to work, and then you can freeze it, store it, boot it up when necessary, you know, boot it up in times as necessary. The transformers don't really have that same feature right now. They're just kind of their, you know, monolithic base form at all times. But the state state models start to add some of that back. Obviously, it's not going to be one-to-one with the the humans or the M's.

• 26:30Speaker 00

  Here's going to be my problem with that number four. If I look at sort of the history of AI over, you know, the history of computers and even the history of automation before that, we see this history where a really wide range of approaches have been tried, a really wide range of paradigms and concepts and structures have been introduced. And over time, we've found ways in some sense to subsume prior structures within new ones. But we've just gone through a lot of them. And there's been this tendency, unfortunately, that when people reach the next new paradigm, the next new structure, they get really excited

• 27:12Speaker 00

  by it, and they consistently say, are we almost done? They said that centuries ago, they said that half a century ago. Every new decade, every new, you know, kind of approach that comes along, people go, there's basically typically some demo, some new capability that's a new system can do that none of the prior systems are able to do. And it's exciting, and it's shocking even, and, you know, exciting. But people consistently say, so it must be almost done, right? Like, surely this is enough to do everything, and pretty soon, humans will be displaced by automation based on this new approach. And that just happens over and over again.

• 27:55Speaker 00

  Over and over again. And so we've had enough of those that I gotta say the chance that the next exciting new paradigm is the last one we'll need is a priori pretty low. We've had this long road to go and we still have a long way to go ahead of us. And therefore it's unlikely that the next new thing is the last thing. So that's, that's my stance. I would think, okay, I can talk to you about L L Ms because they're the latest thing. We can talk about what new things they can do and what exciting options that generates in the near future. And then we could ask, well, what's the chance it's the last thing we'll need? Or that the next one is the last thing.

• 28:37Speaker 00

  And so one way to cash that out is to ask, what do we think the chances are that within a decade or even two, basically all human jobs will be replaced by machines based on this new approach? And, you know, most of the forecasting that's done out there is excited about near term progress in a lot of ways, but when you ask the question, when will most jobs be replaced? They give you forecasts that are way out there because they think, no, we're not close to that. And I don't think we're close to that. So then the question is, now we could say, what will happen when we eventually get to the point where AIs are good enough to do everything? And we don't know what that approach is, but we can still talk about that point and what's likely

• 29:19Speaker 00

  to what the transition rate would be and the transition scenario and who would get rich and would be unhappy and, you know, all the different things we could, we could talk about there. But now we're talking about whatever approach eventually gets us past the, you know, being able to have to do pretty much all human tasks, which is not where we are now. Or we can talk about where we are now and what these things can do. And, you know, what exciting things might happen in the next decade.

• 29:48Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:52Speaker 02

  Well, I'm tempted by all of those options. So maybe for starters, I would be interested to hear how you would develop a sort of cognitive tale of the tape between, you know, humans and M's by presumption have kind of the same cognitive abilities, but these kind of different, you know, sort of external properties of copyability and so on. The large language model today, you know, transformer remarkably simple architecture, you know, when you really just look at like the wiring diagram, it's way simpler than the human brain is. And not shockingly, you know, it can only do certain things. There's, there's like really important traits that the human brain has that the language models don't

• 30:37Speaker 02

  have. I identified one of those as kind of integrated, you know, ever evolving medium and long-term memory. I wonder what else you would kind of flag there. I don't know if you have a taxonomy of what are the kind of core competencies of humans that you could then say, oh, and here's the things that language models currently lack. I'm trying to develop something like this in general because it does seem to me that the large language models have hit not genius human level, but like closing in on expert human level at some very important, dare I say even like core aspect of information processing, right? Like they can do things that I would say are qualitatively different than any

• 31:21Speaker 02

  earlier AI system could do. It certainly seems like we're, we're getting close, you know, whatever the last step is like, we're definitely closer to it than we used to be.

• 31:29Speaker 00

  But just notice that phrase you just gave was true, or most of all the previous ones as well. They could also do a thing that the previous ones before it couldn't do. It's always been exciting. We've found a new fundamental capability that each new paradigm, you know, structures, approach has been of this sort, that it was allowed the system to do fundamental things you couldn't do before that seemed to be near the core of what it was to think. So there's apparently a lot of things near the core of what it is to think. That's the key thing to realize. What it is to think is a big thing. There's a lot of things in there.

• 32:06Speaker 02

  Well, let's list some. I can't come up with that many, honestly. Like, I would love to hear, you know, how many? How many can you name? I have all day. So, you know, could you begin to break down what it is to think into key components?

• 32:19Speaker 00

  I was an AI researcher from eighty-four to ninety-three. That was, I was a full-time at NASA and then Lockheed. And certainly at that time, I understood the range of approaches people had and could talk about the kinds of things systems then could do or not do in expert terms relating to the, you know, then current tasks and issues. I am not up to date at the moment on the full range of AI approaches. So I don't want to pretend to be an expert on that. But I have listened to experts and the experts I hear basically consistently say, this is exciting, this is great, but we're not close to being able to do all the other things.

• 33:04Speaker 00

  And they would be much better than I in making a list of that. And I feel like they should make the list, not me. I mean, as a polymath, you call me, I want to be very careful to know when I'm an expert on something and when I'm not. And I want to defer to other people on areas where I can find people who know more than I. And when I think I'm near the state of the art, as good as anyone on a topic, then I will feel more free to generate my own thoughts and think they're worth contributing.

• 33:29Speaker 02

  Fair, certainly. I think most people, where I think you do still bring something very differentiated to the discussion is just the, the sort of willingness to stare reality in the face, or at least try to.

• 33:48Speaker 00

  The simplest thing is if I start talking to a large language model, there's a whole bunch of things I can ask it to do that it just can't do. I'm not so sure how to organize that in terms of the large major categories, but it's really obvious that there's a certain kind of thinking it can do and a bunch of other kind of thinking it can't do. And it's just, and I don't know exactly why it can't do them. But, you know, I'm talking to you. There's a bunch of things I could ask you to do in this conversation. You would probably do a decent job of them. And then if I were talking to the large language model, it just couldn't do those things. So it's just really obvious to me that this has a limited capability. It's really impressive compared to what you might have expected five or 10 years ago. It's wow, I never would have thought that would be feasible this soon.

• 34:28Speaker 00

  But, you know, you just try asking it a bunch of other things and it just can't do them, right?

• 34:34Speaker 02

  Yeah, I mean, I, I think that in my view, a lot of those things are kind of overemphasized relative to what maybe really matters. You know, you see a lot of things online where people, and there's different categories of this. Some of the things you'll see online are literally people just using non-frontier models and, you know, kind of confusing, muddying the waters. So always watch out for that. Um, I have a, you know, longstanding, uh, practice of first thing I do when I see somebody say GPT-4 can't do something is try it myself. And I would honestly say like two thirds of the time it's just straight up misinformation.

• 35:18Speaker 02

  And it in fact like can do it, but there's still the one third of the time that matters. They're not very adversarially robust. They're easy to trick. They're easy to sort of get on the wrong track. Um, and then they seem to get kind of stuck in a, a mode is a good term for it. I think where once they're kind of on a certain, this is kind of how they can, they can often get jailbroken. You know, if you can get them to say like, okay, I'll be happy to help you with that. Then they'll go on and do whatever, you know, you asked because they've already kind of got into that mode.

• 35:49Speaker 00

  Yeah, I'm much less worried about them doing things you don't want them to do than being able to get them to do things at all. That is, you know, humans can be made to do all sorts of things you might not want them to do. That we survive that. Uh, I mean, to me, the, the main thing is if you imagine. You know, treating large language model as a new employee in some workplace where you're trying to show them how to do something and get them to do it instead of you. That's the main thing that will be economically valuable in the world. That is, when you have a thing like that, that can be introduced into a place, you know, trained roughly and said, watch how I do this, you try to do it now, et cetera. Then that will be the thing that, you know, makes an enormous difference in the economy, because that's how we get people to do things, right?

• 36:34Speaker 00

  So that, I think, is, in a sense, the fundamental main task in the economy, which is a bunch of people are doing something. You have a new thing and you say, would you Kim watch us and ask us questions? And we'll ask you questions and like figure out how to help us and be part of what we're doing. That is the fundamental problem in the economy. So that, in some sense, is the fundamental task that any AI has to be held up to. I mean, in the past, of course, we don't even bother to have a conversation to show how to do. We actually say, well, let's make a machine to do this thing. And then we design a machine to do this thing. And then we train it up to do this thing, all with the idea, the whole thing, having in mind the thing we're going to have it do. That's how AI has been usually in the economy so far.

• 37:17Speaker 00

  But now, if you're imagining a thing that could just be trained to do a new job, well, that would be great. Sure, then we won't have to design the AI ahead of time for the particular task. But you'll have to have a thing that's up to that. And large language models today are just clearly not up to that. You can't say, I'm about to train you how to do the following thing. Pay attention, I just did this, now would you do it, you know?

• 37:39Speaker 02

  Well, you can do that quite a bit, right? I mean, that was the, the main kind of finding in GPT three was, and I'm not sure if I have this verbatim, but the title of that paper was, Large Language Models Are Few-Shot Learners. And the big kind of breakthrough observation there, which I don't think they designed, this was, you know, there's a whole quagmire of what should count as emergent or not emergent. But my understanding is they didn't specifically train for this few-shot imitation capability. But they nevertheless got to the point where, at runtime today, you can

• 38:14Speaker 02

  Give a few examples of what you want. And in fact, that is like a best practice that OpenAI and you know Anthropic recommend for how to get the most from their systems. They'll say, some things are hard, you know, they also have now trained them to follow instructions just verbatim or explicitly, but they will still say that, you know, some things are better shown by example than described in terms of, you know, what to do. So do that, you know, and you'll get like a lot better performance. It seems to me that there is on that kind of watch, you know, watch it to kind of, you know, to borrow from medicine, watch one, do one, teach one. It seems like we're on the like, do one step. And that does seem to be a pretty

• 38:57Speaker 02

  qualitative like threshold that has been passed. Now, they obviously can continue to get better at that.

• 39:03Speaker 00

  Right. But it's the range of things they can do. That's the question. Yes, it's great that they can, you can say, here's some examples, give me another one. But the range of things you can do that for is limited. Most people in most jobs, they couldn't have large language model swap in for, you know, many of their main tasks that way. That's just, but there are some, and that's exciting. And I hope to see people develop that and improve it. But again, the key question is, how close are we to the end of this long path we've been on for a while?

• 39:32Speaker 02

  Yeah, I guess I think about it a little bit differently in terms of rather than thinking about the end of the path, I think of how close are we to key thresholds that will bring in qualitatively different dynamics relative to the current situation. So one threshold that I think has recently been passed and in a pretty striking way that this is, should get more discussion than it does, in my view, is Google DeepMind just put out a paper not long ago where they showed basically a two to one advantage for a large language model in medical diagnosis versus human doctors. And then, of course, they also compared to

• 40:14Speaker 02

  human plus AI. And that was in the middle. So on these cases that they lined up and the, you know, the scenario is like, you're chatting with your doctor, 60% accuracy from the language model, 30% accuracy from the human.

• 40:29Speaker 00

  I was in AI from 83 to 94.

• 40:34Speaker 00

  And at the beginning, one of the reasons I came into AI was there were these big journal articles and national media coverage about studies where they showed that the best AI of the time, which they called expert systems, were able to do human level medical diagnosis. This was in the early nineteen eighties, right? We're talking forty years ago. And obviously, the computer capacity is vastly larger than that. So either they were lying back then, you know, messing with the data, or they did have human level diagnosis back then, but they weren't allowed to apply it because of medical licensing. So, and we're still not allowed to apply it because of medical licensing.

• 41:19Speaker 00

  So, you know, this is exactly the sort of ability that won't give substantial economic impact because we had it forty years ago and it didn't have an impact then.

• 41:30Speaker 02

  Yeah, I don't, I don't know. So if I had, I think one qualitative difference between that earlier system and this system, which, you know, I won't claim to be an expert in the earlier expert systems, but I would guess that a huge difference is that you can take today a totally uninitiated person who has a medical concern and say, sit in front of this computer, talk to this doctor. They don't even need to know it's an AI doctor. They can just talk to it.

• 42:00Speaker 00

  That wasn't the problem back then. They could have made these expert systems usable by ordinary people with modest effort. If that wasn't the problem in using them. The problem was just you're not legally allowed to use them. Only doctors are allowed to give medical diagnoses. And so only doctors are allowed to use these systems to talk to people. That was the main obstacle, and it still is today, the obstacle. You, you could make such a system today that ordinary people could talk to, but they're not allowed to talk to it and they won't be allowed to talk to it for a long time.

• 42:31Speaker 02

  I think there is a qualitative difference between these systems. If I were to sit down in front of the early eighties thing and I were to say, you know, what's different today is that the chat system could say, Robin, tell me how you're feeling. Tell me about your experience and you can just go on in your own language, however you want to express yourself and it can get you. And then it can ask you specific follow up, but you're not going through a wizard and, you know, going down an expert system tree and ask for like numeric scores. You don't understand and don't know. You can literally just express yourself. That was not there then, right? I mean, nothing.

• 43:02Speaker 00

  But, but that's not the limiting factor. Right? I mean, you couldn't have a, you know, fancy graphics interface back then either. This was early 1980s, right? But again, the limiting factor is the legal barrier. It was back then and still is. And that legal barrier doesn't look like it's about to go away. So if you're going to make us excited about applications, it'll have to be something that's legal.

• 43:25Speaker 02

  My model of this is that the consumer surplus of this type of thing is going to be so great.

• 43:32Speaker 00

  It already was 40 years ago. It would have been a huge consumer surplus 40 years ago. It was not allowed.

• 43:37Speaker 02

  But there was never a ground swell of, I don't know. I'm, I'm just not buying this. I'm not buying that there was an experience that is qualitatively like the one that we have today, such that I think today, if you show people what Google has, they will say, it is not acceptable to me that you keep this locked up behind some paywall. I don't think that was the, the general consumer reaction to early eighties expert systems. And it seems like that, that political economy pressure could change things.

• 44:08Speaker 00

  Consider the analogy of nuclear power. The world has definitely been convinced for a long time that nuclear power is powerful. It is full of potential and power. And if we had let it go wild, we would have vastly cheaper energy today. But it was that power that scared people, which is why we don't have that energy today. The very vision of nuclear energy being powerful is what caused us not to have it.

• 44:35Speaker 00

  We overregulated it to death and we made sure that the power of nuclear power was not released. We believed the power was there. It was not a, at all, an issue of not believing that nuclear power was powerful. It was believing it was too powerful. Scary, dangerous, powerful. And there's a risk that we'll do that with AI today. We will make people believe it's powerful, so powerful that they should be scared of it and it should be locked down and not released into the wild where it might do us terrible danger.

• 45:07Speaker 02

  Yeah, well, that's certainly a tragic outcome in the case of the nuclear power. And I think it would also be a tragic outcome if people are denied their AI doctors of the future on that basis. And it could happen. I, you know, I certainly wouldn't rule out the possibility that, you know, just AI research broadly gets made illegal. This time, we do have, I mean, again, it is, I do think we're in a different regime now where enough has been discovered, enough has been put into the hands of, of millions. There is sort of the open source kind of hacker level.

• 45:41Speaker 00

  Not medical diagnosis is not. We have not put medical diagnosis AI in the hands of ordinary people. And if you tried it, you would find out just how quickly you'd get slapped down.

• 45:51Speaker 02

  Yeah, I think, I think I know someone who actually may be about to try this and it'll be very interesting to see how quickly and how hard they get slapped down and how they may respond from it. I've actually been very encouraged by the response from the medical community. I would say, you know, obviously, it's not a monolithic thing, but I did an earlier episode with Zach Kahane, who is a professor at Harvard Medical School and who had early access to GPT-4. He came out with a book, basically to coincide with the launch of GPT-4 called GPT-4 and the Revolution in Medicine. And broadly, I have been encouraged by how much the medical establishment has seemingly been inclined to embrace this sort

• 46:36Speaker 02

  of stuff. I don't know if it's just that they're also overworked these days or?

• 46:40Speaker 00

  Well, they'll embrace the internal use of it. Again, it's always been doctors allowed to use these things. And the main reason they didn't get more popular is doctors couldn't be bothered to type in and input all the information because they want to have short meetings with, with patients. Even today, of course, if you've gone to a modern doctor, most of your meeting with the doctor is them typing in information to their computer as they talk to you. And they don't want to spend much more time, you know, typing in more. And so they don't want to use computer aids in their diagnosis, and that's been true for a long time. They've computer diagnosis aids been available for a long time that would give them better diagnoses at the cost of them having to spend more time with them, and they've chosen not to spend more time. That's been true for many decades now.

• 47:24Speaker 02

  Have you personally used GPT four for any like advanced things like this, medical or legal advice or whatever?

• 47:32Speaker 00

  Uh, no, I'm a economics professor, so I've used it to check to see what my students might try to use it to answer my exam questions or essay questions or things like that. I've asked it things that I wanted to know and and tried to check on them. I haven't used it for legal or medical questions. Those are areas which are heavily regulated. It's always been possible for other people to uh offer substitutes. So for example, many decades ago, there were experiments where we for, you know, basically for the purpose of general practice for doctors, we compare doctors to nurses, nurse practitioners or paramedics. We found that those other groups did just as well and much cheaper at doing the first level of general practice, but they haven't been allowed. So that right there is enormous value that could have been released.

• 48:17Speaker 00

  We could have all this time been having nurse practitioners and doctors and paramedics do our first level of general practice medicine. And they would save at least a factor of two or three in cost. And that's been true for decades. We've had randomized experiments showing that for decades.

• 48:31Speaker 02

  So going back to the age of M then for a second, are we, are you just assuming that that scenario doesn't happen in M land for some reason? Or like, why wouldn't it be? Uh, the first objection to the age of M seems like it maybe should be M's will be made illegal. Nobody will be allowed to do it.

• 48:50Speaker 00

  Absolutely.

• 48:51Speaker 02

  And and in and basically you're just kind of in the analysis saying, well, let's just assume that doesn't happen because it'll be, you know, it's a short, it's a short book if they, if they just get made illegal too early. Is that the idea?

• 49:01Speaker 00

  Well, so first of all, I say transitions are harder to analyze than equilibrium of the world. So I try to avoid analyzing the transition. Although I do try to discuss it some toward the end of the book. Uh, but I admit I can just say less about a transition. It does seem like that, uh, you know, compared to a scenario where everyone eagerly adopted M technology as soon as it was available, more likely there will be resistance. There will be ways in which there are obstacles to M technology early on. And therefore, at some point, there would basically be the, you know, breaking of a dam flooding out where a bunch of things that had been held back were released and then caused a lot

• 49:43Speaker 00

  of disruption, faster disruption than would have happened had you adopted things as soon as they were available. That's part of, that can be a very disturbing transition then. You know, if all of a sudden, large numbers of people are disrupted in ways they weren't expecting in a very rapid way because of, you know, a dam suddenly broke open, then I think there will be a lot of unhappy people in that sort of a transition and maybe a lot of dead people. So imagine the M technology slowly just gets cheaper over time, but it's not very widely adopted. Then there'll be a point at which it eventually gets so cheap that if some, say ambitious nation like, say, North Korea said, gee, if we went whole hog in adopting this thing,

• 50:28Speaker 00

  we could get this big, you know, economic and military advantage over our competitors, then eventually somebody would do that. Now, it might take a long time. That is, the world could coordinate to resist this technology for a long time, but I don't think they could hold it back for a thousand years. So then I feel somewhat confident eventually in the age of M happens, and then eventually there's a thing to think about, and then I'm analyzing that world. So I don't want to presume in the age of M that this transition happens smoothly or soon or as fast as it could. But I want to say eventually there'll be this new world and here's how it would play out. So I don't know if you know that in the last few months, I've dramatically changed my

• 51:11Speaker 00

  vision of the future to say that there was probably going to be a several century innovation pause, probably before the age of M happens, and then the world that would eventually produce AI and M's would be a very different world from ours and somewhat hard to think about. That is, rising population will stop rising and will fall due to falling fertility. That will basically make innovation grind to a halt, then the world population will continue to fall until...

• 51:42Speaker 00

  Insular fertile subcultures like the Amish grow from their very small current levels to become the dominant population of the world. And then when that becomes large enough compared to our current economy, then innovation would turn on again. And then we would restart the AI and M path. And then eventually the age of M would happen. Trying to anticipate how transitions would happen in a world we can just hardly even imagine seems tough, right? That is, okay, imagine the descendants of the Amish become a large, powerful civilization. They've always been somewhat resistant to technology and, you know, very picky about which technologies they're allowed. But eventually, I would predict, you know, there would be competition within them and

• 52:25Speaker 00

  that would push them to adopt technologies like AI and M's. But we're, we're looking a long way down the line. And this isn't what I wish would happen. To go back to your initial thing, I, I would rather we continued growing at the past rate of the past century and continue that for a few more centuries, by which time I'm pretty sure we'll eventually get M's and human level AI, although question of what order. But I gotta say at the moment, that's not looking so good. So basically, you know, I'm estimated that if we were to continue on a steady growth path, we would eventually reach a point where we had the same amount of innovation as we will get over the entire integral of this several centuries pause.

• 53:07Speaker 00

  And I've estimated that to be roughly 60 to 90 years worth of progress. So if we can get full human level AI in the next 60 to 90 years with the progress, then this population decline won't matter so much because we will basically have AIs take over most of the jobs and then that can allow the world economy to keep growing. I think that's iffy whether we can do that, whether we can achieve full human level AI in 60 to 90 years. And I know many people think it's going to happen the next 10 years, they're sure. So sure, of course it'll happen in 60 to 90 years, but I look at the history and I go, look, I've seen over and over again, people get really excited by the next new kind of AI. And they're typically pretty sure, a lot of them are pretty sure that we must be near

• 53:52Speaker 00

  the end and pretty soon we'll, we'll have it all. And it just keeps not happening.

• 53:58Speaker 02

  The main change I want to suggest to that paradigm is replacing the end with meaningful thresholds along the way. I think there are probably several that we will hit on some time scale. And it feels to me like at least a couple of the big ones are pretty close. And then, you know, like at the the end is very, you know, my crystal ball gets very foggy beyond like a pretty short time scale. But I'm struggling with the the early 80s expert systems, but it really does seem like in my lifetime, I have not seen anything that remotely resembles the experience of going to a doctor.

• 54:41Speaker 02

  I've seen WebMD, I've seen, you know, I'm familiar with like expert systems to a degree. But I've never seen anything that, you know, I didn't think Ilya Sutskever from OpenAI puts this really well. He's like, the most shocking thing about the current AIs is that I can speak to them and I feel that I'm understood. And that is like a qualitatively different experience. And, you know, clearly, I think reflects some qualitative, you know, advance in terms of what kind of information processing is going on. If I had to say like, what is that under the hood? I would say it's like a high dimensional representation of

• 55:21Speaker 02

  of concepts that are like really relevant to us that, you know, have previously been kind of limited to like language level compressed encoding. But now we are actually starting to get to the point where we can like look at the middle layers of even just the systems we have today, you know, the transformers and say, you know, can we identify concepts like positivity, you know, or paranoia or love? And we are starting to be able to, you know, it's still pretty messy. We have the same, you know, not the same, but we have a analogous problem to like understanding what's going on inside the brain and it's just a it's a mess in there still in the transformers. But we are starting to be able to see these like high dimensional representations where it's like.

• 56:06Speaker 02

  That is, you know, a numeric representation of some of these big concepts. And we're even starting to get to the point where we can steer the language model behavior by like injecting these concepts. So you can say, for example, you know, inject safety into the middle layers of a transformer and get, you know, a safer response or, you know, danger, right? Or, you know, rule breaking and then they'll, it'll be more likely to like break their rules.

• 56:31Speaker 00

  What you're focused on at the moment is telling me about how the latest generation adds capabilities that previous generations didn't have. But every previous generation had that same conversation where they focused on the new capabilities their new generation had that the ones before it didn't have. What the conversation you're participating in is continuing the past trend. But the fundamental question is, when will AIs be able to do what fraction of the tasks that we have in the human economy? If they can't do a large fraction of them, no matter how impressive they are at the tasks they can do, we will see this economic decline as the population declines.

• 57:13Speaker 00

  They need to be able to do pretty much all the tasks in order to prevent the economic decline and then the halting of innovation. I did this study of innovation in the United States over 20 years, from 1999 to 2019. And that was a period that encompassed what many people at times said was enormous AI progress. And many people in the period were talking about how there was this revolution in AI that was causing, that was about to cause a revolution in society in this period from 1999 to 2019. So we did a study, a co-author and I, Keller Scholl, who looked at all jobs in the US, basically

• 57:55Speaker 00

  roughly 900 different kinds of jobs. And over that 20 period, year period, we had measures of how automated was each job in each year. And then we could do statistics to say, when jobs got more automated, did they get, the wages go up or down? Did the number of workers in those jobs go up or down? And we could say, what about jobs predicts how automated they are? And did the things that determine which jobs are how automated change over that 20-year period?

• 58:29Speaker 00

  That is, if there had been some revolution in the nature of automation, then the things that predicted which jobs would be more automated would have changed over time. What we found was that when jobs got more or less automated, that had no effect on average on wages or number of workers. And that the predictors of automation didn't change at all over that 20-year period. And they remain to be very simple-minded predictors that you might expect about automation from long ago. The nature of automation hasn't changed in the aggregate in the economy. Main predictors of automation are whether the job has nice, clear measures of how well you've done it, whether it's in a clean environment with fewer disruptions, and whether tasks nearby have been automated.

• 59:10Speaker 00

  So there's a way that which task automation spreads through the network of nearby tasks. So that study suggested at least up until 2019, there had been no change in the nature of automation. And basically, there's a Gaussian distribution of how automated jobs are. And the median automation had moved roughly a third of a standard deviation through that distribution. So jobs had gotten more automated substantially in that 20-year period. But still, most jobs aren't that automated. And that would be my rough prediction for the next 20 years is to say the pattern in the last 20 years will continue. That is, I will slowly get more jobs more automated, but most automation will be very, you know, basic stuff.

• 59:53Speaker 00

  So far, we just haven't seen much at all of advanced AI kinds of automation making a dent in the larger economy.

• 1:00:01Speaker 02

  So what do you make of things? I'm sure you're familiar with like the MMLU benchmark or the Big Bench. Maybe not. If not, I can characterize them for you. But is this machine learning set of tests in order to benchmark performance? Yes, I believe it's massive multitask language understanding. The great Dan Hendricks and team.

• 1:00:23Speaker 00

  So basically, a bunch of language understanding benchmarks?

• 1:00:25Speaker 02

  Yeah, they basically went and took final exams from like university and early grad school courses from every domain and compiled them into this massive benchmark. There have been a couple of different efforts like this, but this is basically the gold standard on which all the language models are measured. And we now have a like high eighties to ninety percent accuracy rate across all fields from like a single model, namely GPT four, and now Google claims that its Gemini is hitting that level as well. I would agree that these have not been broadly customized to the last mile specifications that they need to like work in the context of different firms and, you know, cultural

• 1:01:10Speaker 02

  contexts and all that sort of thing. But it does seem like, the way I typically describe it is that AIs are now better at routine tasks than the average person and that they are closing in on expert performance on routine tasks. And that's measured by, you know, these medical diagnosis benchmarks, these MMLU type things, et cetera, et cetera.

• 1:01:32Speaker 00

  So let me remind you that in the nineteen sixties, say, AI researchers took chess as a paradigm of if you can make a machine that can do that, well, obviously you'll have to have solved most of the major problems in thinking because chess involves most of the major problems in thinking. So when we can finally have human level chess abilities, we will have human level AI. That was the thinking in the sixties and they could look at the rate at which AI was getting better at chess and forecast long before it happened that in the late nineteen seventies, nineteen nineties, excuse me, is exactly when chess would reach human level ability. And that's when it did happen. And that was twenty five years ago.

• 1:02:17Speaker 00

  And clearly they were just wrong about the idea that you couldn't do chess without solving all the major thinking problems. And we repeatedly have this sort of phenomenon where people look at something and they go, if you can do that, surely you can do most everything. And then we can do that and we can't do near, and we aren't near to doing most everything. So I just got to say this benchmark is just wrong. It's not true that if you can do this language benchmark, you are near to doing most everything. You are not near.

• 1:02:44Speaker 02

  Yeah, I would find my position to say, I think you're near to being able to do all the routine things that are well documented in the training data.

• 1:02:52Speaker 00

  Well, yes, but, but the question is, in the economy, all the things we need doing, how close are you to that? I'd say you're not close.

• 1:02:59Speaker 02

  I mean, we're seeing just the very beginning of sort of, I mean, again, I don't know, like.

• 1:03:06Speaker 00

  What do you think was going on in their head in the nineteen sixties when they looked at chess, right? They looked at chess and they said, it takes really smart people to do chess. Look at all these complicated things people are doing when they do chess in order to achieve in chess. They said to themselves, that's the sort of thing we should work on because if we can get a machine to that, surely we must be close to general artificial intelligence. If you could have something that could do chess.

• 1:03:30Speaker 00

  And there is a sense that when you have general intelligence, you can use all of that to do clever things about chess. But it's not true that you need to have all those general things in order to be good at chess. It turns out there's a way to be good at chess without doing all those other things. And that's repeatedly been the problem, and that could be the problem today. Turns out there's a way to do these exam answering things that doesn't require the full range of artificial general intelligence in order to achieve that task. It's hard to pick a good range of tasks that encompasses the full range of intelligence because, again, you teach through the task and you end up finding a way to solve that problem without achieving general intelligence.

• 1:04:09Speaker 02

  This does seem different, though. I mean, I would, I agree with your characterization that basically it turned out that there was an easier way or a more direct way, a narrower way to solve chess. And it's interesting that it's like rather different, you know, it involves these sort of superhuman tree search capabilities.

• 1:04:28Speaker 00

  But that wasn't just true of chess. There were another dozen sorts of really hard problems that people in the 1960s took as exemplars of, you know, things that would require general intelligence. And a great many of them have been achieved.

• 1:04:41Speaker 02

  But when I look at the current situation, I'm like, this does look a lot more like the human intelligence, and I would say that from any number of, of different directions.

• 1:04:53Speaker 00

  And that was true in every decade for the last century. Every decade has seen advances that were not the sort that previous systems could achieve.

• 1:05:03Speaker 02

  It's clear that you, or at least I think it's clear that you don't see the human brain, the human, you know, achieve, level of achievement as sort of a maximum, right?

• 1:05:13Speaker 00

  Oh, of course not, absolutely.

• 1:05:14Speaker 02

  So it's, it's like, there's got to be a finite number of breakthroughs that need to happen.

• 1:05:22Speaker 00

  We will eventually get full human level AI. I have no doubt about that. And, and that, and not soon after vastly exceeded. That will happen, and it will happen plausibly within the next thousand years.

• 1:05:33Speaker 02

  It also seems like you would probably agree that it need not be point for point. You know, the M scenario is a great one to play out and analyze, but it need not be the case.

• 1:05:45Speaker 00

  Right, so the AIs could be much better than humans in some ways and still much worse than others. That will probably actually be true for a long time. That is, it'll take a lot longer till AIs are better than humans at most everything than that they are better than humans at, say, half of things people do today. But of course, you have to realize if you looked at what humans were doing two centuries ago, we're already at the point where machines do those things much better than humans can do. That is, the most tasks that humans were doing two centuries ago are already long since automated. We've now switched our attention to the sort of tasks that people were not doing two centuries ago. And on those, we're not so good at making machines do them, but we've already dramatically achieved full automation, basically, of most things humans were doing two centuries ago.

• 1:06:29Speaker 02

  Which, for very shorthand, I would say is kind of routine, repetitive, physical tasks.

• 1:06:35Speaker 00

  Right, I mean, we managed to change the environment to make them more routine and repetitive. So, you know, a subsistence farmer on a subsistence farm two centuries ago, they were, we couldn't, our automation could not do that job that they were doing then. We managed to make the farms different, the factories different, et cetera, so that our machines could do them. And now they are producing much more than those people produce. But if you had to try to produce the way they were doing two centuries ago, our machines today could not do that.

• 1:07:03Speaker 02

  Yeah, a big theory I have also, I actually don't think this is going to be a huge, well, everything's going to be huge, but I don't think it's going to be like the dominant change that that leads to qualitatively different future. But I do think we will start to see and are beginning to see that same process happening with language models, where, you know, I consult with a few different businesses and we have kind of processes that, you know, we would like to automate. You know, a classic one would be like initial resume screening, right? We're not going to have the language model at this point make the hiring decisions. But if we get a lot of garbage resumes, you know, we can, we can definitely get language models to kind of

• 1:07:43Speaker 02

  band the resumes into a, you know, one to five and like spend our time on the fives. It does seem to me that there's a lot of kind of process and environment adaptation that is not that hard to do. Like, I personally have done it successfully across a handful of different things. Why? It seems like your analysis sort of doesn't, assumes that that's not going to happen at scale this time around with the technology we currently have.

• 1:08:11Speaker 00

  I said, you know, in the last, in 20 years, from 1999 to 2019, we moved roughly a third of the standard deviation in the distribution of automation. Okay, so what if we, in the next 60 years, move a third of a standard deviation in each of the 20-year periods? Then, over 60 years, we would basically move an entire standard deviation. That could represent a large increase in automation over the next 60 years. And that would mean a lot of things we're doing by hand today will be done by machines then. It would mean our economy is more productive, but it still would mean humans have a huge place in the world. They get paid, and most income probably still goes to pay humans

• 1:08:56Speaker 00

  to do work, even though they have much better automation at the time. If that's the situation in 60 years, then unfortunately, that level of increase in automation is just not sufficient to prevent the economy from declining as population declines. And so, we won't get much more automation than that. The well of it, in automation will dry up because innovation will stop, and we would then have a several centuries-long period where we, our technology does not improve. And in fact, we lose a lot of technologies tied to scale economies. As the world economy shrinks, we'll manage to have less variety, less large-scale production and distribution, and we would then struggle to maintain previous technologies. And AI

• 1:09:41Speaker 00

  is at risk of the sort of technology would be hard to maintain because at the moment, AI is a really large-scale concentrated sort of technology. It's not being done by my mom and pops, it's to be done by very large enterprises on very large scales.

• 1:09:53Speaker 02

  I would agree that the, the supply chain is definitely prone to disruption in AI, no doubt about that. Can you describe in more detail what, what is the standard deviation in automation? And like, how should I conceptualize that?

• 1:10:08Speaker 00

  I mean, I guess what you'd want to do is see a list of tasks and how automated each task was. And then see sort of how much on that score and it moved up. So basically, if you look on this list at the most and least automated tasks, you'll agree which are which. Like the nearly most automated task is airline pilots. Nearly the least automated task is carpet installers. Carpet installers use pretty much no automation to staple in carpets. And airline pilots are pretty much always having automation help what they're doing. And then, you know, you can see the scores in the middle and see that we've, you know, moved up a modest degree over those 20 years.

• 1:10:53Speaker 00

  That would be the way to get an intuition for it is just to see a list of particular jobs in their automation scores and and see compare that to the amount by which we've moved up.

• 1:11:03Speaker 02

  How do you reconcile or how should I understand the idea that whatever doubling time of the economy today, I think you said it was like 15 years in the book, which seemed a little fast to me, just based on like rule of 70.

• 1:11:18Speaker 00

  Right, I think it's more like, you know, 20 or something now.

• 1:11:21Speaker 02

  But still, like, it seems it seems like there's a little bit of a disconnect between a notion of, you know, over these next 60 years, we would be double, double, double, you know, essentially 10 X-ing the economy, but we'd only move at sort of a linear rate in automation, like we would only move a third of a standard deviation in each period.

• 1:11:43Speaker 00

  Let me help you understand that, then. People have often said, look, computer technology is increasing exponentially. Therefore, we should expect an exponential impact on the economy, i.e. early on hardly any impact, and then suddenly an accelerating boom such that we get this big explosion and then everything happens. But that's not what we've seen. So what we've seen over time is relatively steady effects on the economy of automation, even though the economy is growing exponentially.

• 1:12:13Speaker 00

  The way I'd help you understand that is to imagine the distribution of all tasks that you might want automated and that there the degree of computing power, both in hardware and software, required to automate that task. For each task is distributed in a log normal way with a very large variance. That is, there's this very large range of how much computing power it takes to automate a task. As computing power increases exponentially, you're basically moving through that log normal distribution in a linear manner. And in the middle of the distribution, it's pretty steady effect. You slowly chop away at tasks as you are able to automate them because you're slowly acquiring sufficient hardware to do that task.

• 1:12:57Speaker 00

  That gives you a simple model, but in which computing power grows exponentially and yet you see a relatively steady erosion of tasks through automation.

• 1:13:07Speaker 02

  It's a low hanging fruit argument.

• 1:13:09Speaker 00

  Yeah, the low hanging fruits are hanging really low. That this this is a log normal tree, basically, that you're trying to grab things from. I mean, you're growing, your ladder is growing exponentially into the tree. And every time your ladder gets taller, you get to pick more feuds, but it's a really tall tree. That means that you have a long, long way to go.

• 1:13:27Speaker 02

  How do you think about things like the progress in AI art generation or like deep fakes over the last couple of years? This is an area where I feel like if we rewind to two years ago, just two years ago, really, when I was first starting to see AI art popping up on Twitter. And it was like, not very good for the most part. You'd see the occasional thing where you're like, oh, that's really compelling. And then you'd see a lot of stuff that was like, yeah, you know, it's whatever. It's remarkable that you can do that. It's wow compared to what came before, but it, you know, it's like. It's I'm not going to be watching like feature films based on this technology and,

• 1:14:10Speaker 02

  you know, in the immediate future. I feel like we could have had a very similar discussion where you might say, well, you know, yeah, it's progress, but you know, the the real human art, the top notch stuff like that's so far away. And then early last year, my teammates at at Waymark made a like short film using nothing but Dolly three or Dolly two at that time imagery and some definite elbow grease. But like the quality of production that they were able to achieve with a half dozen people and Dolly two is on the level that like previously would have taken, you know, a crew in Antarctica, you know, to go shoot. You know, again, there's is that work all done? No. But if you look at the mid journey outputs today, you look

• 1:14:52Speaker 02

  at some of the deep fake technologies that are happening today. It's like, it does feel like we've hit certainly phonorealistic thresholds, you know, almost indistinguishable from photography with mid journey and with the deep fakes. You're not quite quite there yet, but like, watch out for 2024 to have a lot of stories of people being scammed by the kind of custom text to speech voice, you know, with a family number, family members' voice, whatever. All my voice out there, you know, people are going to be calling my parents with my voice. So I guess what I'm trying to get at there is like, it seems like even just in the last couple of years, we have these examples where we are seeing like really rapid progress that is not stopping before critical thresholds.

• 1:15:37Speaker 00

  In the 1960s, there was a US presidential commission to to address and study the question of whether most jobs were about to be automated. It reached that level of high level concern in the country and major media discussion about it. Ever since then, we continue to have periodic articles about dramatic, exciting progress in AI and what that might mean for the society and economy. And in all those articles through all those years, they don't just talk in the abstract. They usually pick out some particular examples, and they don't pick out random examples from the economy. They pick out the examples where the automation has made the most difference. That of course makes sense if you're trying to make an exciting story.

• 1:16:23Speaker 00

  And so we've always been able to pick out the things which are having the most dramatic increase lately that also seem the most salient and interesting. And now you can pick out image generation as one of the main examples lately as something that's increased a lot lately. And I'm happy to admit it has. I would put it up, you know? And that's the sort of thing that somebody writing an article today about the exciting AI progress would in fact mention and talk about graphic artists being put out of work by the availability of these things, which probably is happening. The point is just to realize how selective that process is to pick out the most dramatic impacts and to realize just how many other jobs there are and how many other tasks there are and then how far we still have to go.

• 1:17:08Speaker 00

  I'm happy to celebrate recent progress and if I were, you know, if I were a graphic artist person, I would be especially excited to figure out how to take advantage of these changes because they are among the, you know, biggest changes. If you're a say a 20 year old in the world, it makes complete sense to say, where are things most exciting and changing? I want to go there and be part of the new exciting thing happening there. If of course you're a 60 year old and you've already invested in a career, then it makes less chance sense to like try to switch your whole career over to a new thing. But a lot of people are at the beginning of their career and they should. They should look for where the most exciting changes are and try to see if they can go be part of that. Move west, young man, if west is where things are happening, right? But you still have to keep in mind, if there's a few people going out west making exciting

• 1:17:52Speaker 00

  things happening, how big a percentage of the world is the west, right? Yes, it's exciting and there's huge growth in the west, you know, there 10 years ago there was hardly anything and now there's a big town. Look how great the west is growing and that, you know, there are always times and places where right there, things are growing very fast. And newspaper writers should focus on those to tell stories and people, novelists should focus on those to sell stories. They're exciting places where exciting things are happening and I, and I want to make sure the world keeps having things like that happening because that's how we can keep growing. But you have to be honest about the fraction of the world that's involved in those exciting frontier stories.

• 1:18:30Speaker 02

  Yeah, I mean, I guess my kind of counterpoint to that would be the same.

• 1:18:37Speaker 02

  Relatively simple technology, like the transformer or like the attention mechanism, perhaps it is better, you know, pinpointed as, is driving this art creation. It's also writing today, like, short programs. Yeah, I would personally say my productivity as a programmer has been increased like several fold, not like incrementally, but like multiple with GPT-4 assistance. You know, it's the wide range, right? You could go on, but like, it's, it's also happening in metal, medical diagnosis. It's also happening in like protein, you know, novel protein structure generation and...

• 1:19:14Speaker 00

  Certainly, from an economic point of view, the biggest category you've mentioned is programming. That's a much larger industry, less relative profession than the other ones you mentioned.

• 1:19:22Speaker 02

  Well, but watch out for biotech also, I would say, for sure, but...

• 1:19:25Speaker 00

  Biotech's on a, has been shrinking for a while, so that's not an extent thing you should point to as a growing thing.

• 1:19:31Speaker 02

  I will predict growth for biotech, definitely. I mean, we, you know, it's also, it's reading brain states. Have you seen these recent things where people can read the brain state?

• 1:19:39Speaker 00

  Among the things you're talking about, at the moment, the biggest profession being affected is programming, clearly. I have a younger son, two sons, my younger one is a professional programmer, so, you know, I've had him look at, and his workplaces looked into what they can do with large language models to help them write programs and their evaluation so far is, you know, they, they don't even, they'll wait in six months to look again. It's not useful now.

• 1:20:05Speaker 02

  Can I, what, can I short that stock?

• 1:20:08Speaker 00

  Uh, well, I could tell you after we, we finish what, what that is, but basically, I think this is true, most actual professional programmers are not using large language models that much in the, you know, doing their job. Now, I gotta say that if some people are getting factors of two productivity increase, that eventually we should see some effect of that on their wages. That is, of course, you know, now if lots of programmers go out and use productivity spaces, in some sense, we're going to increase the supply of programming. And so, supply and demand would mean that maybe increasing the supply lowers the price, even if it dramatically increases the quantity.

• 1:20:50Speaker 00

  But, you know, there's such a large elastic demand for programming in the world that I actually think that effect would be relatively weak. And so, you should be expecting large increases in the wages going to programmers if you are expecting large overall increases in the productivity of programmers. Because, again, it's a large elastic demand for programming in the world. You know, for a long time, a lot of change in the world has been driven by programming and limited by the fact that there's only so many decent programmers out there. Only so many people you can get to do programming. So, clearly, if we can dramatically expand the supply of programmers, we can do a lot more programming in a lot more areas. And there's a lot of money that's willing to go to that, to do that.

• 1:21:35Speaker 00

  There's a lot of people who would be hiring more programmers if only they were cheaper. And they're about to get cheaper, in effect. And so, you should be predicting large increases in, basically, the wages and number of programmers in the world. We haven't seen that yet.

• 1:21:51Speaker 02

  I do predict large increases in number. I'm not so sure about wages. It feels like-

• 1:21:55Speaker 00

  Why not?

• 1:21:56Speaker 02

  Well, I've done a couple of episodes with the folks at a company called Replit, which is a very interesting end-to-end, at this point, software development platform. Their mission is to onboard the next one billion developers. And, you know, they have like a great mobile app. They have kids in India that are, you know, 14 years old, that are doing it all on their mobile app. And I'd say it's much harder, and maybe this reflects the kind of programming that your son is doing, but I'd say it's much harder to take the most elite frontier work and accelerate that in a meaningful way versus like commoditizing the routine application development

• 1:22:37Speaker 02

  that like the, you know, the sort of long tail of programmers mostly do.

• 1:22:42Speaker 00

  My son is definitely doing routine application development.

• 1:22:47Speaker 00

  Not at the frontier of programming at all. But again, I'm saying I don't expect this sudden large increase in programmer wages and quantity, especially wages. I mean, the less the quantity increases, the more wages would have to be increasing to compensate. And I think it'll be hard to get that many more people willing to be programmers, but you could pay them more. And I don't predict this. So this is a concrete thing we could, you know, even bet on over the next five or 10 years. Will there be a big boost in programmer wages? That would be the consequence. It's a very simple supply and demand analysis here. This isn't some subtle, you know, rocket science version of economics.

• 1:23:29Speaker 02

  Well, typically when supply increases, price drops, right? I'm expecting lots more programmers and them to be broadly cheap.

• 1:23:36Speaker 00

  Depends on the elasticity of demand.

• 1:23:39Speaker 00

  So, you know, if you think about something that there's just a very limited demand for in the world, you know, if piano tuning got a lot cheaper, you wouldn't have a lot more pianos. Because piano tuning is not one of the major costs of having a piano. You know, it's the cost of the piano itself, plus the space for it in your living room, right? And the time it takes to play on the pianos. So piano tuning is a really a small cost of piano. So that means the elasticity of demand for piano tuners by itself is pretty low. You know, there's just basically so many pianos, they all need to be tuned. And if each piano tuner could tune each piano twice as fast, say, then we basically only need half as many piano tuners, because there's just not much of elasticity for demand. So for kinds of jobs like that,

• 1:24:24Speaker 00

  productivity increases will cause a reduction in the employment. But even in that case, you might get a doubling of the wages and half the number of piano tuners, because they can each be twice as productive. But for programming, it's clear to me that programming has an enormous elastic demand. The world out there has far fewer programmers than they want. They would love all over the place to hire more programmers to do more things. There's a big demand in the world for software to do stuff. And there's a huge potential range of things that software could be doing and it's not doing now. So that means there's a pretty elastic demand for programming. That means as we increase the quantity of programming, the price doesn't come down that much. There's still people willing to buy this stuff.

• 1:25:09Speaker 00

  So that tells me that as productivity increases. Basically, the supply is expanding and the demand's not coming down much. So we should just see a much larger quantity. But then, you know, basically because each person is being more productive, each person should get paid more. So I mean, the elasticity of supply is going to be a combination of two things. Each person getting more productive and more people being willing to join that profession. And I think we've already seen that even as the wages for programming has gone way up in the last decade or so, the number of programmers hasn't gone up as fast. That is, there's just kind of a limited number of people who are decent at programming. And it's hard to get the marginal person to be a programmer.

• 1:25:55Speaker 00

  But the people who are programmers, when they're productive, they get paid a lot. I mean, as you've probably heard rumors about AI programmers and how much they're being paid lately. It's crazy high because there's just a limited supply. So I gotta say, I expect large increases in wages for programmers if in fact large language models are making programmers much more productive. But according to my son at least, and others I've heard, you know, that's not happening.

• 1:26:24Speaker 02

  I'm with you up until the very last two points. I would say I think it is happening. And I would also say, I think my estimation of the relevant elasticities is that there will be a large growth in people who can be and will choose to be programmers, but that the wages don't go up. They don't fall like dramatically necessarily either, because it has to be like an attractive thing for people to want to do it. But I think that the prevailing wages are quite high compared to what a lot of people would be excited to take if they could easily break in with language model assistance, which I think they will increasingly be able to do. Let me change gears a little bit.

• 1:27:05Speaker 02

  So we've debated, this has been really, I always appreciate a useful and thoughtful challenge to my world model. You're definitely supplying that. Let's do a couple like a little bit more speculative things that could be kind of M first, you know, a little bit of LLM. As I was going through the book, there are a number of things that I was like, hmm, this is really interesting. How would I think about this a bit differently? And, you know, and maybe suspend a little bit of your skepticism of how much impact LLMs will make. Let's, let's go in a world where, you know, scaling continues to work, context lengths get long, you know, we start to see that not total,

• 1:27:47Speaker 02

  you know, displacement of humans, but like substantial fraction of, you know, tasks being like LLM automatable. One interesting inference that you make is that there won't be that many different base ends, that essentially there will be super selective, M-ifying of really elite, really capable people, that those will become the bases, that they'll be sort of essentially turn into kind of clans where they'll, they'll highly identify with each other and they'll have like, you know, marginally different specialization, but that there will be these sort of recognizable, almost canonical personalities that are,

• 1:28:27Speaker 02

  and not that many of them, that kind of come to dominate the economy. It seems like we're kind of seeing something similar with language models already, where it's like, we have GPT-4, we have, you know, some, the new thing from Google, we have Claude, we have like a couple open source ones, and then they get like a lot of like local fine tuning and kind of adaptation. I guess my read on that was that it's an odd, you know, it's initially a very surprising vision of the future, but it does seem like we see the proto version of that in the development of large language models. Any thoughts?

• 1:29:03Speaker 00

  It's basically, how many different kinds of jobs are there is the question, tasks are there? And then how many dimensions do they vary? So I mean, there's clearly a lot of different kinds of jobs. Like I told you, the study we did looked at, you know, nine hundred of them. But once you look at nine hundred different jobs, a lot of jobs are pretty similar to each other. And they take pretty similar mental styles and and personalities to do those jobs. So when we're looking at humans, at least, it looks like a few hundred humans would be enough to do pretty much all the jobs. That's looking at the variation of humans. Now, the harder part is to say, well, large language models.

• 1:29:47Speaker 00

  Is their space of dimensional variation similar to humans? Or is it very different? That that's much harder to judge. But, yeah, I would guess that it's in this way, not that different. That is, even in large language models, there's a difference where you first you train a basic model, and that's a lot of work. And then you train variations on it. And it does look like the variations are mostly enough to encompass a pretty wide range of tasks. And so you need a small number of base approaches and then a lot more cheaper variations that are enough to do particular things. Uh, so certainly, that's, you know, a remarkable fact in some sense about large language models is the range of different tasks they can do, starting with the same system, right?

• 1:30:32Speaker 00

  And so they have a degree of generality that way. And, you know, humans in some sense have a degree of generality that way, where we are able to do, able to learn to do a pretty wide range of things. So, yeah, I would, and I don't know if it's going to be just four as opposed to forty or four hundred, that's harder to say. But in some sense, it could be one or two. I mean, even in the age of M, I was giving the few hundred as an upper limit. It could turn out to be much lower. It really depends on how much sort of, you know, quick, fast, last minute variation can actually encompass the range of differences. If differences are somewhat shallow and and surface, which not really fundamental, then, yeah, last minute variation might be enough.

• 1:31:16Speaker 02

  Another interesting assumption, this one, I think, is more of a contrast with the the language models is, and we talked about this briefly earlier, that the M's, they can be easily cloned, but they can't be easily merged. In other words, like, you know, because we don't have a great sense of how exactly it works inside and and what internal states are meaningful, we can't just like superimpose them on top of one another. Language models, it seems like we are making actually a lot more progress on that front. It's not a solved problem, but there are techniques for merging, there are techniques for like training separately and combining. There are these sort of many Q-Loras techniques.

• 1:31:54Speaker 00

  People are exploring those, but like notice that to make GPT-4, you didn't start with GPT-3 and add more training. You started with a blank network and you started from scratch. And that's consistently what we've seen in AI over decades. Every new model does not start with an old model and train it to be better. You start with a blank representation and you train it from scratch. And that's consistently how we've made new systems over time. So that's a substantial degree of not being able to merge. And that's quite different than humans. I mean, often to get a human to do a new task, you want to take a human who can do lots of previous tasks because they can more quickly learn how to do this

• 1:32:35Speaker 00

  new task. And that's just not what we're seeing. Like, you try to take, I don't know, Claude and GPT-4 and, you know, Grok and merge them. I mean, I just don't think anybody knows how to do such a merge today. There's no sensible way you could do such a merge. You could take Claude and then do all the training that you would have done on GPT-4, except do it starting from Claude. And I think people think that would be worse than starting with the blank representation as they usually do.

• 1:33:06Speaker 02

  Yeah, I think that's definitely not a solved problem today. And I wouldn't claim that you can just like drop Claude and GPT-4 on top of each other. But there are enough early results in this that it seems much more plausible. Plus, we have like the full wiring diagram, you know, and the ability to kind of X-ray internal states with, you know, perfect fidelity. It seems like there is a much more likely path. Forget about the plausibility for a second. What do you think it would mean if the AIs could be kind of divergent but also re-mergeable?

• 1:33:40Speaker 00

  I think the fundamental issue here is rot. So we see rot in software, especially with large legacy systems. We see rot in the human brain. I think we have to expect rot is happening in large language models, too. Rot is the reason why you don't start with old things and modify them. You start from scratch. So that is basically when you have a large old legacy piece of software, you could keep trying to modify it and improve it. But typically, at some point, you just throw it all away and start from scratch again. People get a lot of advantage about being able to start from scratch. And that's because old, large things rot. And my best guess is that that will continue to be true for large language models and all the kinds of AIs we develop. We will continue to struggle with rot as a general problem indefinitely.

• 1:34:25Speaker 00

  And this is actually a reason why you should doubt the image of the one super AI that lasts forever. Because the one super AI that lasts forever will rot. And in some sense, to maintain functionality and flexibility would have to replace itself with new, fresh versions periodically, which then could be substantially different. And, you know, that's in some sense how biology has worked, too. Biology could have somehow made organisms that lasted forever, but it didn't. It made organisms that rot over time and get replaced by babies that start out fresh and rot again. And that's just been the nature of how biology figures. And that's how our economy works. We could have had the same companies as we did a century ago, running the economy, just changing and adapting to new circumstances. But we don't.

• 1:35:10Speaker 00

  Old companies rot and could die away and get replaced by new companies. And I predict in the age of M that M's would in fact rot with time and therefore no longer be productive and have to retire and be replaced by young M's. And that's a key part of the age of M star that I think would generalize to the AI world. I think, in fact, rot is such a severe and irredeemable problem that AIs will have to deal with rot in roughly the same way everybody else has, i.e. make systems, let them grow, become capable, slowly rot and get replaced by new systems. And then the challenge will always be, how can the new systems learn from the old ones?

• 1:35:49Speaker 00

  How can the old ones teach the new ones what they've learned without passing on the rot? And that's a long time design problem that we're going to face in large language models even. I, I think, you know, in a few years, a company will have had a large language model. They've been building up for a while to train, you know, to talk to customers or something. And then it'll be rotting and they'll wonder, well, how can we make a new one that inherits all the things we've taught this old one? And they'll struggle with that. They can't just move the system over. They'll have to have maybe the same training sets or something. They have to collect training sets that are going to apply to the new system like the old one. But that will continue to be a problem in AI as it has been in all complicated systems so far.

• 1:36:33Speaker 02

  Yeah, interesting. I think that is a pretty compelling argument for like medium and long time scales. And I can even see that, you know, already like OpenAI supports, for example, fine tuning on a previously fine tuned model. And I don't in practice use it. And I'm not sure how many do. What I do think is still a plausibly very interesting kind of fork and merge is, you know, with these new state space models. It seems that you could, like one remarkably difficult challenge for a language model is scan through my email and find what's relevant. You know, it's like, it has a hard time doing that for a couple of different reasons.

• 1:37:16Speaker 02

  You know, finite context window, and I just have a lot of email. With the state space models, I do think you could clone, you know, or parallelize, have them each kind of process a certain amount. And literally then just potentially merge their states back together to understand, you know, in kind of a superposition sort of view. What are all the things that are relevant, even though they were processed in parallel? And so I do think that, that kind of quick forking and merging could be a really interesting capability. But at some level of divergence, it does seem like it probably just becomes unfeasible or not even desirable.

• 1:37:55Speaker 00

  I mean, so a very basic, interesting question about brain design is the scope for parallelism. So, you know, in your brain, there's a lot of parallelism going on. But then when you do high level tasks, you typically do those sequentially. And so there's just an open question in AI. Surely you can do some things in parallel at some small time of a time scale. But how long of a time scale can you do things in parallel before it becomes hard to merge things?

• 1:38:23Speaker 02

  Okay, another different topic. So in the age of M, the assumption seems to be from the beginning that because these things are, in some sense, one for one with humans, that they should get, or people will naturally be inclined to give them a sort of moral worth status.

• 1:38:45Speaker 00

  I think it's more the other way around. They would insist on it, just like you would insist that people around you dealing with you give you some substantial moral weight. If the A M's are just actually running the society, they will similarly insist on that. And if humans want to deal with them, we'll kind of have to go along.

• 1:39:06Speaker 00

  You know, unless they are, the M's are enslaved by humans, then if the M's are free to work with the humans or not. And, you know, it's just like, in general, having a modest degree of respect for your coworkers is kind of a minimum for being a coworker. If your coworkers perceive that you disrespect them enough, then they just won't want you around and you'll have to go somewhere else. So if humans are going to interact and work with M's, they'll have to, on the surface at least, when they're not in private, treat them with modest respect.

• 1:39:36Speaker 02

  Well, for the record, I always treat my language models with respect as well. Very polite to them. I never engage in the emotional manipulation techniques that some have shown to perhaps be effective, but doesn't feel quite right to me. And not because I think they're moral patients, but it's more about just the habits I want to get into. But I'm still a little confused by this on a couple of ways. One is, first of all, just by default, it seems like they will be enslaved to humans. Like the first M's that get created, they get loaded onto a machine. They're in some state. I can turn them on. I can turn them off. They can't decide when they get turned on and turned off, right? If I boot them up in a eager, ready to work sort of state, and they're like ready to do a task, they're probably not even gonna, you know, and they've got these like virtual inputs. They're probably not even going to.

• 1:40:20Speaker 02

  Be in the mindset, right, to think like I demand respect, they're just going to be in that mindset that they were kind of stored in of like, ready to work. So why, I'm still a little confused as to where that comes from. And then the flip side of that question would be, under what circumstances, if any, do you think we would start to treat our language model or successor systems as, you know, moral patients, you know, even if they're not one to one with us, but like, are there things that they might start to do or be, you know, what ways they might start to behave where you think we would feel like that's the right thing to do?

• 1:40:54Speaker 00

  We have substantial understanding of slavery in human history and where it works and where it doesn't and why. First of all, we know that when land was plentiful and people were scarce, then people would have high wages and then it might be worth owning somebody. But vice versa case where people were plentiful, land was scarce, then there really wasn't much point in having slaves because free workers would cost about the same. And why bother with enslaving? So, um, so situations where slavery made some sense is where wages were high, but then depending on the kind of task, there's some kinds of tasks where slavery can help

• 1:41:39Speaker 00

  and others where it doesn't so much. So say in the US South, you know, out in the field of picking cotton or something, if you just need people to push through their pain, then slavery can force them to do that and be, make them be more productive. But if they need to do complicated things like being a house slave or a city sort of slave at a shop, those sorts of slaves tended to not be abused and to be treated like a worker would because they just had so many ways to screw you if they were mad. Their jobs were complicated and you were trusting them to do a lot of things. And so as a practical matter, you had to treat those sorts of slaves well. Work has become far more complicated since then and employers have become far more vulnerable to employee sabotage.

• 1:42:25Speaker 00

  Uh, you know, there's not that much that a cotton picker can do to sabotage the the cotton. If they're mad at you, uh, you can just whip them and make them pick the cotton faster. But again, house slaves, uh, shop slaves, uh, city slaves, you know, they just have a lot more discretion. And you need to get sort of get them to buy in. And so again, it's that in the age of M is a world where wages are near subsistence levels. So you know, the work, how to work you can get out of a slave is about the same as you can get out of a free worker because they're both working for subsistence wages. If the free worker is more motivated, they enjoy themselves more and they feel more and owning themselves and they and that gives them a sense of pride and devotion and they're less willing to sabotage your workplace.

• 1:43:10Speaker 00

  That would be a reason to not have them be slaves. Uh, and then I think large language models, certainly they have been trained on data about human behavior wherein humans are resentful of being treated as slaves and want to be respected and need that they feel motivated and to, you know, need to feel respected to be motivated and are less likely to sabotage if they feel like they have some freedom. And all of those things would continue to be true of large language models to the extent that they were trained on human conversation and and behavior. And that's how humans are. So uh, in this vast space of possible AIs, there could be AIs that don't mind it all being enslaved. But large language models aren't going to be those.

• 1:43:53Speaker 02

  But it does seem like you sort of expect that natural selection or sort of, you know, human guided selection of these systems will trend that direction. Like the idea that M's or language models will sort of demand the leisure seems to be at odds with the other part of the vision that they will like become okay with being sort of turned on, turned off.

• 1:44:16Speaker 00

  So the need for leisure does seem to be more just a constraint on the human mind. That is people are just more productive when they get breaks. That seems to be a very robust feature of human work across a wide range of contexts, even including literal slaves. They need, you know, a five minute break every hour. They need a lunch break. They need an evening break. They need a weekend. This is just what human minds are like. They are more productive when they get periodic breaks. So maybe the breaks aren't leisure exactly. Maybe they don't write a novel in their spare time, but they do need what they see as a break.

• 1:44:47Speaker 02

  Well, I know we're just about out of time. Maybe my last question is, are there things that you are looking for or are there things that you could imagine happening in the not too distant future where you would change your expectations for the future again and begin to feel like maybe we are entering into a transition period that will lead to a qualitatively different future, like going a different direction from this sort of technology stagnation.

• 1:45:19Speaker 00

  The trends that I would be tracking are which jobs, tasks actually get automated. How much is paid for those? So if I saw, you know, big chunks of the economy where all of a sudden workers are doing, you know, a lot more automation is doing tasks instead of workers and that changing the number of workers and the wages they get and the number of firms supplying that go up, then yeah, that I start to see a lot of things happening. That that's the thing I'm looking for. And that's the thing that people haven't seen so much in the past. They tend to focus on demos or maybe the high tech companies that get a lot of reputation out of doing AI. Uh, and not so much the rest of the economy and who's actually getting paid to do stuff. You know, I mean, you know, if you think about, say, the farming revolution,

• 1:46:05Speaker 00

  where tractors went out and replaced farmers, that was really large and really visible and really clear. If you look at, say, trucks replacing horses, you saw a very large, very substantial replacement with enormous differences in who supplied them and who got paid. We have seen large changes in automation in the past. We, we, we don't have to scrape to sort of see subtleties and such things. They're often just quite out in the open and visible and very obvious. So that's what I'm waiting for. Those big, obvious sorts of displacements. And even having, you know, trucks replace horses and tractors replacing farmers didn't make AI take over everything. Even if I saw big changes, I wouldn't necessarily predict we're about to see AI take over everything,

• 1:46:49Speaker 00

  but I would at least know what I'm looking at. And that's the sort of thing to try to project forward and try to think about where that's going to go.

• 1:46:56Nathan Labenz?

  This has been an awesome conversation. I've been a fan of your work for a long time and it's been an honor to have you on The Cognitive Revolution. Robin Hanson, thank you for being part of The Cognitive Revolution.

• 1:47:08Speaker 00

  Thanks for having me.

• 1:47:09Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.