AGI and Institutional Disruption: Sam Hammond in Conversation with FLI’s Gus Docker

For the end of the year, we're going to repost a few of Nathan's favorite AI scouting episodes from other shows. Today: Samuel Hammond joins the Future of Life Institute podcast to discuss how AGI will transform economies, governments, institutions, and other power structures. If you need an ecommerce platform, check out our sponsor Shopify: https://shopify.com/cognitive for a $1/month trial period. Samuel Hammond is a Canadian-born, DC-based senior economist for the Foundation for American Innovation, a think tank focused on bridging the cultures of Silicon Valley and DC. This conversation is super wide ranging, covering the most likely default path to AGI, the economic and institutional transformations that AI will beget, the critical distinction between thinking about AI in isolation versus as part of a dynamic system, the proposal for a Manhattan or Apollo like megaproject for AI safety, and lots more. You can subscribe to Future of Life Institute here: https://futureoflife.org/project/future-of-life-institute-podcast/ You can read Samuel Hammond's blog at https://www.secondbest.ca --- We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. --- SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. TIMESTAMPS: (00:00) Intro: End of year and bonus episodes (04:24) Discussion on AI Timelines (14:08) Insights from Information Theory (15:11) Sponsors: Shopify | MasterClass (32:12) AI Progress and Hard Steps in Evolution (34:21) Sponsors: NetSuite | Omneky (39:12) Government Preparedness for Advanced AI (46:16) The Internet's Role in Mass Mobilization and Erosion of Trust (01:05:47) The Role of AI in Monitoring and Surveillance (01:27:18) The Potential Dangers of Open-Sourcing AI (01:36:24) The Influence of AI on Society and Privacy (01:42:21) The Future of Government Services in a Techno-Feudalist Society (01:53:58) The Future of AI in Robotics (02:15:25) The Role of AI in the Future of Financial Markets This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. As we've reached the end of 2023, we're still just 11 months into making The Cognitive Revolution, but I wanted to take a moment to say a big thank you to everyone who has made this show possible. Starting with our sponsors, from our first presenting sponsor, Omnikie, who agreed to support the show before hearing even a single episode, to the household names you've heard from throughout the year, including Oracle, MasterClass, Shopify, and NetSuite. We appreciate your support. I also want to say thank you to the entire Turpentine team, and in particular, our producers, Natalie Toran and Vivian May, our editor, Graham Besaloo, and of course, our superconnector

• 0:41Nathan Labenz?

  and the mastermind behind the idea for a network of expert-hosted podcasts, Erik Torenberg. We've made nearly 90 episodes of this show, and I always emphasize that AI content ages poorly. With that in mind, we've turned many episodes around in just a day or two. Now, this is not the norm in the podcasting business, and it does create a lot of odd hours for the team. But they've been here for it every step of the way, and I have been extremely privileged to be able to focus 100% of my time and energy on the content itself, knowing that they have my back on all the logistics. Of course, I also want to thank our many incredible guests. They are an amazingly diverse group, hailing from at least a dozen different countries

• 1:24Nathan Labenz?

  around the world, with all sorts of different academic and professional backgrounds, but all united by an appreciation for the transformative, dare I say, revolutionary, potential of AI. Collectively, they have helped me develop a far deeper understanding of AI technology than I had coming into the year, and I hope the same is true for you. Finally, I want to thank all of you, the listeners, for making this project not just worthwhile, but a success well beyond my expectations. Coming into the year, I had never created original content before, and I had no platform or audience anywhere. And to be honest, I wasn't really sure if anyone would want to listen to the deeply in the weeds, hopefully anti-polarizing content that I wanted to create. On that

• 2:08Nathan Labenz?

  front, I'm glad to say that I have been consistently pleasantly surprised. Thousands of you have become regular listeners, and some of our most challenging episodes, including my recent two and a half hour monologue about state space models, have been among our very best received. We end the year as the 20th ranked technology podcast, according to Spotify, 31st according to Apple, and I take that as a sign that something is definitely working. Nevertheless, when a friend recently asked what metrics we use to track performance, I responded that for me, the best indicator is the number and the quality of people who reach out to share feedback with me directly. So with that in mind, if you have any topic or guest suggestions, or just general thoughts about how we can make the show more valuable,

• 2:52Nathan Labenz?

  we would love to hear them. We have a ton planned for the cognitive revolution in 2024, including episodes about the rise of AI automated scientific exploration, the intersection of AI and biology, the energy demands of AI, the future of robotics, and much, much more. So while we're working on all that, as a bit of bonus content, we're going to repost a few of my favorite AI scouting episodes from other shows, episodes that I was jealous of when I first heard them, starting today with an episode from the Future of Life Institute podcast, in which host Gus Docker interviews economist Samuel Hammond. This conversation is super wide-ranging, covering the most likely default path to AGI, the economic and institutional transformations that AI will to get, the critical distinction

• 3:36Nathan Labenz?

  between thinking about AI in isolation versus as part of a dynamic system, the proposal for a Manhattan or Apollo-like mega project for AI safety, and lots more along the way. I hope you find this and the next couple of episodes to be helpful complements to my own AI scouting work, and I encourage you to subscribe to the Future of Life Institute podcast feed, where you can hear lots more about AI and other issues of global importance. With that, Happy New Year from the Cognitive Revolution, and here's Gus Docker and Samuel Hammond.

• 4:09Gus Docker?

  Welcome to the Future of Life Institute podcast. My name is Gus Docker and I'm here with Samuel Hammond, who is a senior economist at the Foundation for American Innovation. Samuel, welcome to the podcast.

• 4:22Samuel Hammond?

  Hi Gus, thanks for having me.

• 4:23Gus Docker?

  Fantastic. All right, I have so much I want to talk to you about, but I think a natural place to start here would be with your timelines to AGI. Why is it that you expect AGI to get here before most people?

• 4:38Samuel Hammond?

  Well, I don't really know what most people think. I think there's, the world divides into people who are paying attention and people who are, you know, basically normies. And in my day job, I work on Capitol Hill and in Washington DC, talking to folks about AI. And if you think about sort of what people's implicit timelines are, right? You can sort of read out people's implicit timelines by their behavior, right? You know, I know Paul Cristiano has short timelines because he puts, he's doubled up into the stock market, right? He's sort of practicing what he preaches. But then when you have, you know, Sam Altman testifying to Congress, I like to say people are taking him seriously, but not literally, right? He's saying, "We're going to develop something like AGI,

• 5:20Samuel Hammond?

  potentially this decade and superintelligence thereafter." And then, you know, you have folks like Senator Marsha Blackburn being like, "What will this mean for music royalties?" And, you know, when the focus of policymakers is things like music royalties or, you know, impact on copyright, it's not that those are invalid issues. It's that they belie relatively longer timelines. And then we also have this definitional confusion where folks like John L. Coon would say AGI is probably decades away because he is using AGI to mean something that learns, like a human learns, in the sense that it's born as a relative blank slate and sort of can acquire language with, you know, very few examples. So people have these moving goalposts of what they mean.

• 6:05Samuel Hammond?

  For me, You know, I think we can sort of avoid those definitional conflicts if we just talk about, you know, human level intelligence and, and, you know, humans are quite general or generally intelligent. That's what separates us from animals in a lot of respects. And when you look at how machine learning models are being trained today, like large language models and now multimodal models, they're being trained on human data and they're being trained to reproduce the kinds of behaviors and tasks and outputs that humans output. And so they're kind of like an indirect way of emulating human intelligence. And then so, and so if you, if you benchmark AI progress to that, then you can sort of put information theoretic

• 6:50Samuel Hammond?

  bounds on, you know, what, what's the likely timeline to basically an ideal human emulator, something that can extract the sort of base representations, the internal representations of our brain through, you know, the indirect path of the data that are ranging rates.

• 7:08Gus Docker?

  Yeah, you have an interesting sentence where you write that AI can advance by emulating the generator of human generated data, which is simply the brain. Do you think this paradigm holds all the way to AGI?

• 7:20Samuel Hammond?

  I think it holds this decade to systems that in principle can in context learn anything humans do. Again, this is a semantic question. Do you want to call that AGI or not? I think there are still outstanding issues around, you know, the limits of other aggressive models for autonomy and, you know, the question of sort of real time learning, the way we train these models, we sort of are freezing a crystal in place and, and humans are continuously learning. So there's still our genuine, you know, potential architectural gaps. But, you know, from the practical point of view, from the economic point of view, we don't need to debate whether something is conscious or whether something learns strictly the way

• 8:04Samuel Hammond?

  humans learn if, if it demonstrably can do the things humans do, right? And that, and that goes to the original insight of the Turing test, right? You know, it's sometimes presented as a thought experiment, but what Alan Turing was getting at was if you can't distinguish between the human and a computer, In some ways, indistinguishability implies competence, right? And we can broaden that from just language, because arguably, we've we've surpassed the Turing test, at least a weaker version of it, to think to human performance on tasks in general, right? If we have a a system that can output a scientific manuscript that experts in the field can't distinguish from a human, then debating whether the whether this is real

• 8:48Samuel Hammond?

  AGI or not is, I feel, academic.

• 8:52Gus Docker?

  It is surprising in a sense that when you interact with GPT-4, for example, and it it can do all kinds of amazing things and organize information, present information to you, but then it can't, or at least at some point, it couldn't answer questions about the world after September 2021 or a date like that. That would be surprising if you if you presented that fact to to an AI scientist 20 years ago. For how long do you think we'll remain in this paradigm of training a foundational model and then deploying that model?

• 9:24Samuel Hammond?

  I mean, it's worse than that. I think it would surprise people five years ago. Progress is sort of moving along two tracks. There's the industry track and the pure research academic track. And and they're obviously having feedback with one another. The pure industry track is just looking to create tools that are, you know, have practical value and can improve products and so forth. And so, you know, Meta has their own GPU cluster and their training models, so they could they could they can have fun chatbots in their in their messenger. And so, you know, those kinds of things are going to progress, I think, well within current the current paradigm, because we know the paradigm works, you know, basically deep learning and transformers.

• 10:05Samuel Hammond?

  And, you know, there's lots of merchandilly on the side, but that basic framework seems to be quite effective. And just scaling that up, because we haven't sort of hit the range of irreducible loss and what and what transformers can do. Meanwhile, there's also this parallel pure research track where people on seemingly a weekly basis are finding, you know, better ways of specifying the loss function, ways of of improving upon power law scaling and all these all these different. Sometimes they're new architectures, but often they're just like bags of tricks. And and those bags of tricks then, you know, to the extent that they they comport with the paradigm industries running with, they can be reintegrated and and end up accelerating

• 10:50Samuel Hammond?

  progress in industry as well.

• 10:53Gus Docker?

  So do you think scale of compute is the main barrier to getting to to human level AI?

• 11:00Samuel Hammond?

  Yes, right. I mean, it's not all we need, but it's it's the main unlock, right?

• 11:06Gus Docker?

  To what extent can can more compute be used to to trade off for a lower quality data or for lower quality algorithms? Can you just throw more compute and solve the other the other factors in that equation?

• 11:21Samuel Hammond?

  It depends on on the thing you're trying to solve for. In in principle, if if you're talking about, you know, mapping inputs to outputs, then, you know, transformers are known to be universal function approximators. And and so and so the answer is yes, you know, that doesn't mean that they're necessarily efficient at approximating everything we want them to approximate. And and sometimes universal function, you know, approximation theorems can be kind of trivial because they'll be like, OK, if your neural network has infinite width, then yes, we can approximate everything. The the key fact is both that they're universal approximators and and also that they're relatively sample efficient, at least relative to things we've found in the past. And so that to me suggests that yes, they can compensate for things that they're bad at. On the other hand, the way

• 12:06Samuel Hammond?

  research is trending is towards these mixed models, ensembles of different kinds of architectures, things like the recent Q transformer, and that's from Google DeepMind, which is sort of uses a combination of transformers and Q learning to have sort of the associational memory and and and sample efficiency of transformers with the ability to assign policies to do tasks that you get from reinforcement learning. So I imagine that there's going to be all kinds of mixing and matching. The key, the key point is that

• 12:37Samuel Hammond?

  In that space of architectures, it's relatively sort of finite search space, right? And as an economist, you know, economists believe that supply is long run elastic, right? And so, you know, there's this famous bet from between Paul Ehrlich and Julian Simon vis a vis the population bomb and whether population growth would lead to a sort of a Methusian purge. And Julian, Julian, Julian Simon, being the economist, recognized that, you know, if prices rise for these core commodities, then that will spur research and development into extracting new resources, right? So he didn't, he didn't have to know that fracking would be a technology. He, he understood that if oil prices went too high, people would find

• 13:20Samuel Hammond?

  new oil reserves. And I think there's a, I have an analogous instinct when it comes to progress in deep learning, meaning you can become too anchored to sort of the current state of the literature. But over a 10 year horizon, you can say, well, there's, there's a huge search on a huge gold rush to find the right way of blending these architectures. And I don't need to know in advance which is the right way to do that, to have high confidence that someone will find it.

• 13:51Gus Docker?

  Yeah, we can sometimes, if we're too deep in the, in the literature, we might lose the focus on, on the forest for, for the trees in a sense. And if we zoom out, we can just see that there's more investment, there's more talent pouring into AI. And so we can predict that something is going to come out, come out of that. You have lots of interesting insights about information theory and how this can help us predict AI. What's the most important lessons from information theory?

• 14:18Samuel Hammond?

  The reason I, I start there is because it's sort of, and within the conceptual realm, it's sort of like the most general thing that bounds everything else. And when you look back at the record of, say, you know, Ray Kurzweil, you know, I first read The Age of Spiritual Machines when I was a kid. And, and you know, in there, he makes a prediction that we'll have AIs that pass the Turing test by 2029 or so.

• 14:43Gus Docker?

  And when was this book written?

• 14:45Samuel Hammond?

  1999.

• 14:47Gus Docker?

  Yeah, that's pretty good.

• 14:48Samuel Hammond?

  Right. And so, you know, and you got, and people will complain that he got things wrong because he said, well, I'll be wearing AR glasses by 2019. Uh, you know, when in fact, like Google Glass came out in 2013 and, and now we have Meta Glasses five years later. Uh, so he, he was like wrong on the exact timing, but sort of right where the technology was wrong, where the minimum viable product was.

• 15:11Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:15Samuel Hammond?

  But nonetheless, like if you look at his track record, it's, it's quite good for a methodology as, as relatively stupid as just staring at Moore's Law. And extrapolating it out. And I think that reveals the, the power of these information theoretic methodologies to forecasting because they, they set bounds on what will be possible. The team at Epoch AI have a, a forecast called the direct approach where it's sort of, you can think of it sort of like a way of putting bounds on when we'll have AIs that can emulate human performance through an information theoretic lens. Where they're looking at sort of how much entropy does the brain sort of process and how much compute will we have over time?

• 16:00Samuel Hammond?

  And what, what's implied by AI scaling laws and you sort of put those three things together and you can sort of set bounds on when we'll basically be able to brute force human level, um, intelligence. And, and of course that's an upper bound because we're going to do better than brute force. We're going to also have insights from uh, cognitive science and neuroscience and also, uh, you know, ways of distilling neural networks and so forth and better ways of curating data. So their modal estimate for human level AI is, is 2029. Um, and their meeting is like 2036 and I've talked to the authors and they lean, lean towards the 2029, 2030 for their own um, personal forecasts. And, you know, so going back to, you know,

• 16:41Samuel Hammond?

  is, is this an outlier? Am I, am I out on a limb here? I think among our circles, probably not, but among Congress and among the broader public, I think people are seeing sort of, they, they think everything's an asymptote. Right? They, they're imagining, okay, we have these chatbots and they're not seeing the next step. You know, I see a very smooth path from here to systems that can basically in context, learn. Any arbitrary human task. And so what does that look like? It looks like systems that can basically sit over your shoulder or can monitor your desktop, your operating system as you work and watch you for an hour or two and then take over.

• 17:16Gus Docker?

  And that'll be key to overcoming lack of training data? Or why is it important that they can learn in context?

• 17:24Samuel Hammond?

  Well, in context, learning is sort of the secret source of the power of transformer models. They learn these inductive biases and induction heads and so forth that let them, if you shot, learn different tasks. So, you know, GPT-4 is very good at zero-shot learning on a variety of different things, but it's incredibly good. If you shot learning, if you give it a few examples, it can kind of pick up where you left off. You know, when I think about myself, when I want to learn a new recipe, right, I can go read a recipe book, but often what I prefer to do is to go on YouTube and watch someone make that recipe. Right, and just by watching that person put together the stir fry, I have enough of a world model and enough of knowledge of how to cook in general that I can sort of in context, learn how to pick up from there and do that recipe myself.

• 18:10Samuel Hammond?

  LLMs do that already. Multimodal models are increasingly doing that. Some of the recent progress in robotics, like I mentioned, the Q-Transformer paper, it shows that you can basically build robots with a basic world model and then have it learn new tasks with fewer than 100 examples of a human demonstration. So the human sort of demonstrates the task and the robot can pick it up and take it from there. And why that's important is both for understanding the trajectory of AI, but also its economic implementation. Because we're sort of used to automation being this thing where you get a contract from IBM and you spend many millions of dollars with consultants and they build you some bespoke thing that doesn't really work very well and requires lots of maintenance.

• 18:55Samuel Hammond?

  And so people have this sort of prior that AI, even if it's near, will be rate limited by the real world because of all the complexity of implementation. But the point is, if you have things that can in context, learn and perform sort of as humans perform, then you don't need to change the process. You can take human-designed processes and have the AI just fill in for the human. And so it leads to this paradox where we're probably going to have AGI before we get rid of the last fax machine.

• 19:26Gus Docker?

  Yeah, when we think of, say, old IT systems in large institutions, we might think of moving from analog storage of information to the cloud. That's still going on in some institutions. That transformation has taken over a decade now. And so what exactly is it that makes AI different here? It is that AI plugs in directly where the human worker would be?

• 19:52Samuel Hammond?

  Yeah, precisely. You don't need to redesign an existing process to sort of plug into the automation. And that applies both for sort of the structure of tasks, right? Like, so much of mechanical automation takes something like this sort of artisanal work of a shoemaker, and has to translate it into something repetitive that a machine or an automatic seamstress can do over and over and over again. Our older school kind of automation requires sort of collapsing a task into a lower dimension so that simple automations can handle it. But when you have AGI,

• 20:31Samuel Hammond?

  the whole point is generality. It's a flexible intelligence that can map to existing kinds of processes. So that's sort of why I think this will catch people by surprise, because it's not just the AGI could be this decade, but that when it arrives and sort of crosses some thresholds of reliability, the implementation frictions could be very low.

• 20:53Gus Docker?

  And do you expect, would AI have to get all the way there in order to substitute for a human worker? I mean, I would expect it to be a bit more gradual than that, taking over, say, 20% of tasks before 40% of tasks, 60% of tasks, and so on. But here we're imagining that the AI kind of plugs in for the human worker for all tasks? Or what do you have in mind?

• 21:16Samuel Hammond?

  These things are, yeah, you're right, much more continuous. It's not an on or off switch, in part because the requisite threshold of reliability varies by the type of task. You know, arguably, self-driving cars like Waymo or Tesla have matched human performance, but regulators want them to be 100x better than human before they're let loose on the road because of safety. You know, codex and coding models are arguably still much worse today than elite programmers, but everyone is using them because...

• 21:50Samuel Hammond?

  Even if it generates fifty percent of your code and you have to go back in and debug, it's still a huge productivity boost. So I think it will vary by occupation, by task category, sort of modulo the risks and stakes involved in those tasks.

• 22:06Gus Docker?

  Yeah, I guess then the question is, how many of our jobs fall into the is more like self-driving cars and how many of our jobs is more like programming?

• 22:15Samuel Hammond?

  Right, I mean, I've been in a manager position before and I've had research assistants and interns and I know that they're like a very lossy compression of the thing I want to do and so they require oversight and sort of co-piloting. We're sort of in that stage now with AIs and a variety of different kinds of tasks. You know, I recently read a paper evaluating the use of GPT-4 for peer review in Science and it found that GPT-4 would write reviews of work that bore some striking correlations with the points raised by human reviewers, but also left some things out.

• 22:53Samuel Hammond?

  And so it concluded by saying GPT-4 could be an invaluable tool for scientific review, but it's not about to replace people. And that's just a case of like, okay, give it five years.

• 23:09Gus Docker?

  Yeah, this is a phenomenon you often see with some AI models out there and it has some capabilities but lacks other capabilities and then people might kind of over-anchor on the present capabilities and not foresee the way the development is going. I think people are continually surprised at the advancement of AI.

• 23:31Samuel Hammond?

  Yeah, absolutely. Ramaz Nam, the sci-fi author and futurist and energy investor, he gives this talk on solar energy and other renewables and he has this famous graph where he shows the International Energy Agency, the IEA, every year they put out this projection of solar build-out and every year it's like a flat line, but it's like a flat line on an exponential, like the real curve is like going vertical and every year their projection is that it's just going to plateau. And I feel like people make that same mistake and it sort of has this sort of ironic lesson to the extent that. We're drawing sort of parallels with the way our brain works and the way these models work. It seems like humans have a very strong autoregressive bias.

• 24:17Gus Docker?

  So what's going on there? Is it, is it an institutional problem or is it, is it a psychological problem? Why is it that we can't project correctly in many cases?

• 24:28Samuel Hammond?

  Well, to what I just said, I think it's, it's probably both, but largely psychological. Right? Our brains are evolved for, you know, hunter gatherer societies that didn't really change over millennia. And, uh, you know, even the last 40, 50 years have been a period of relative stagnation where we have a lot of sort of pseudo innovation. And so I think people are just a bit, uh, sort of disabused.

• 24:51Gus Docker?

  Okay. You, you have some super interesting points about the comparing the human brain, how the human brain works to how neural networks learn. What is universality in, uh, in the context of, of brain learning and neural network learning?

• 25:07Samuel Hammond?

  So universality is a term of ours that sort of refers to the fact that different neural networks independently trained, you know, even on different data, uh, will often converge on very similar representations in their embedding space of that data. And you can extend that to striking parallels or isomorphisms between the representations that neural, artificial neural networks learn and that our brain appears to learn. Probably the, the area the brain has studied the most is the visual cortex. And it seems to me as like a layperson that the, the broad consensus in neuroscience is that the visual cortex is very similar to a deep convolutional neural network.

• 25:48Samuel Hammond?

  It's basically isomorphic to our, our, uh, artificial deep convolutional neural networks. And you train a CCN on image data and our brain is trained on.

• 26:01Samuel Hammond?

  Our sensory data and it turns out they end up learning strikingly similar representations. And there are a few reasons for that, right? So, you know, one is sort of hierarchies of abstraction. It makes sense that early layers in a neural network will learn things like edges and and simple shapes and only later in the only deeper in the network will you learn more subtle features. So there's that sort of sequencing part of it. And then there's also just the energy constraint, you know, gradient descent isn't costless, right? It requires energy, it requires a lot of energy. That's, you know, these data centers suck up a lot of energy. The same is true of our brain. You know, our brain consumes a lot of energy, like 25% of our calories.

• 26:43Samuel Hammond?

  And especially when it's in when we're young, there's a very strong metabolic cost associated with neural plasticity. Our brain being something shaped by evolution was obviously very energy conscious. And so those energy constraints greatly shrink the landscape of possible representations from sort of this infinite landscape of all the ways you could represent certain data to a much more manageable set of representations. And that doesn't guarantee that will converge on the same representations. At least suggestive of a of a weak universality where even when we don't have the exact same representations, there are often a coordinate transformation away from each other.

• 27:21Gus Docker?

  It's actually a bit surprising as you mentioned, when we train neural networks, we don't have the same energy constraints as the as the brain had during our evolution. And I would expect again from evolution, the human brain to have many more inbuilt biases and heuristics. But if we then compare the the representations in a in a neural network to to the those in a human brain, we found that they are quite similar. Isn't that the whole point of of universality? So does the neural network have the same heuristics and biases that we have or what's going on here?

• 27:57Samuel Hammond?

  Well, one of the primary, you know, biases in stochastic gradient descent is sometimes called a simplicity preference, basically an inductive bias for more parsimonious representations.

• 28:11Samuel Hammond?

  Person monious in the sense of Occam's razor, right? And that's, that's a byproduct of this information theoretic concept of Kolmogroff complexity, where Kolmogroff complexity means is measured by, you know, is there a short program that can reproduce this longer sequence? And if you can find a short program, that's sort of a more compact or more compressed way of representing it. And when you're under energy constraints, you're looking for those more compressed representations. And so that simplicity bias seems to be also the origin of generalization of our ability to, to go beyond merely memorizing data, overfitting our parameters to finding a simpler way of representing those parameters, right?

• 28:55Samuel Hammond?

  Where we go from sort of fitting a bunch of data points to recognizing, oh, these data points are being generated by a sign function. So I can replace all these data points by a simple circuit for that sign function or something like that.

• 29:07Gus Docker?

  What can we learn about AI progress when we consider the hard steps that humans and our ancestors have gone through in, in evolution?

• 29:17Samuel Hammond?

  It's beyond evolution. This is often comes up in the discussion of the Fermi paradox. Life on Earth to exist at all, let alone intelligent life, had to pass through many hard steps, right? We had to have a planet in a habitable zone. We had to have, you know, the, the right mix of organic chemicals in the Earth's crust and so forth. We had to have the conditions for abiogenesis, the emergence of the very earliest sort of non-living replicators, probably, you know, some kind of polymer type of crystal structure. Then we had to have, you know, the transition from single cell to multi-cellular organisms, to transition through the Cambrian explosion, right?

• 29:59Samuel Hammond?

  All, all, every one of these steps, you could think of as a very unlikely improbable thing, all the way up to, you know, the development of warm-blooded mammals and sort of social animals that were, were heavily selected for, for brain size to, to then the socio-cultural hard steps of like moving from small group primates to, to sort of settled technological cultures, then, you know, technological hard steps, like the discovery of the printing press or the discovery of the transistor.

• 30:33Samuel Hammond?

  You put those all together and life seems just incredibly unlikely. And you know, this often goes to the point of view that, you know, creationists or intelligent designers would put forward. But then you zoom out and then you recognize, oh, wait, there are, you know, trillions of galaxies, each with trillions, you know, hundreds of billions of stars and hundreds, hundreds of billions of trillions of planets. There's a, there's an awful lot of potential variation out there. And then meanwhile, every one of these hard steps seems characterized by a search problem that is very hard. But then, once you find the correct thing, like the earliest self replicator, things kind of take off, right?

• 31:16Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:20Gus Docker?

  So you imagine that before the earliest self replicator, there were millions or billions of attempts to self replicate like that didn't succeed?

• 31:30Samuel Hammond?

  Yeah, it's just a huge search problem, right? And, you know, maybe there are more gradual intermediate stages where you have sort of, you know, everything in biology ends up looking way more gradual, the more you learn about it. But there are these phase transitions where you tip over and you get the Cambrian explosion or you get the printing press and the printing revolution. And so those hard steps end up looking relatively, they look more easy in retrospect, because even though the search was hard, once you, once you tripped over the correct solution, there's sort of an autocatalytic self reinforcing loop that pulls you into a new regime. And indeed, when you look at the emergence of life on Earth relative to the age of the

• 32:13Samuel Hammond?

  universe, and Avi Loeb with some co-authors have done this, life on Earth is incredibly early. Like, you know, the universe is 13.7 billion years old, but life couldn't emerge really much sooner. The reason being, the, you know, the universe started out as hot and dense, had to cool down, stars had to form, the stars had to supernovate so they could produce the heavy, heavy elements that are essential to life. And then those solar systems had to then take shape and then, and then had to further cool so the solar system wasn't being, you know, irradiated constantly. And when you put all those factors together, human life emerged basically as soon as it was possible for life to emerge anywhere.

• 32:58Samuel Hammond?

  And so this, this is one way to answer the Fermi paradox is that we're just in the first cohort. Right, but it also should give you strong priors that passing through those hard steps isn't as hard as it looks.

• 33:08Gus Docker?

  And what's the lesson for AI here?

• 33:11Samuel Hammond?

  Developing AGI is is sort of a hard step.

• 33:15Samuel Hammond?

  We're doing this kind of gradient search for the right algorithms, for the right what have you, and we seem to be now in a slow takeoff where we've figured out the core ingredients and there is now an autocatalytic process that's pulling us into a new phase.

• 33:32Gus Docker?

  And what do you mean by autocatalytic?

• 33:34Samuel Hammond?

  Self-reinforcing. You know, once it gets started, it it sort of pulls itself, it sort of has an as if teleology, right? You see this in nature, but you also see this in, you know, in capitalism.

• 33:47Gus Docker?

  And you would expect us to get to advanced AI basically as soon as it's computationally possible?

• 33:55Samuel Hammond?

  It it basically seemed that way, right? Like, you know, there was a kind of tacit collusion between Google and and other players in the space to, you know, they had transformer models since 2017, but really, you know, there's some of the precursors to transformers go back to the early 90s. But, you know, once you have this sort of profit opportunity that's that's in the background, it's hard in the competitive environment to stop an open AI from being like, oh, let's let's chase those profits. And then once that ball gets rolling, it's basically impossible to stop. This is this is why, you know, whatever the merits of the pause letter, it's virtually impossible to to really have a pause in AI development because because everything is

• 34:38Samuel Hammond?

  is sort of structured by these game theoretic incentives to just keep going faster. Once you've stumbled on the gold reserve, it's hard to keep the the prospectors from running there.

• 34:48Gus Docker?

  Samuel, is the US government prepared for advanced AI?

• 34:53Samuel Hammond?

  No.

• 34:56Samuel Hammond?

  No, I mean, where do I start? I mean, the US government's in, if you think of it from a firmware level, you know, many countries have national IDs. The US doesn't have a national ID. We have social security numbers. There are these like nine-digit numbers that date back to 1935. We have

• 35:15Samuel Hammond?

  the core administrative laws date back to the early 40s. You know, much of our sort of technical infrastructure, like the the system the IRS runs on,

• 35:27Samuel Hammond?

  date back to the Kennedy administration and are written in assembly code. There's also been this general decline in what you could call state capacity, sort of the ability for US government to execute on things. And you hear about this all the time. You hear about how, you know, the the Golden Gate Bridge was built in four years or something like that. And now it takes like 10 years to build an access road. One of the reasons for that goes to what the the legal scholar Nicholas Bagley has called the procedural fetish. Really, since the 70s, the machinery of the US government has shifted towards a reliance on explicit process. Right? And proceduralism

• 36:09Samuel Hammond?

  has pluses and minuses. If you have a clear process, government can kind of run on autopilot to an extent. But it also means you limit the room for discretion. And you limit the flexibility of government to move quickly. And moreover, in our adversarial legal system, you also open up avenues for sort of continuous judicial review and legal challenge where, you know, famously, New York has taken over three years to approve congestion pricing on one of their bridges because because that's to undergo environmental review and people who don't want to pay the congestion price keep suing.

• 36:45Gus Docker?

  Do you think having more procedures would make it easier for AI to interface with with the with government?

• 36:52Samuel Hammond?

  I would say having fewer procedures would make it easier for government to adapt.

• 36:55Gus Docker?

  My assumption would be that that having something written down, having a procedure for something would make it easier for AI to plug AI into that procedure. If it's if it's less opaque and and more kind of a almost like an algorithm step by step.

• 37:10Samuel Hammond?

  Yes, but the you know, the analogy I would give is to, you know, the Manhattan Project, right? The Manhattan, the the original Manhattan Project was run like a startup. You had Oppenheimer and General Leslie Grove sort of being the the technical founder and the the, you know, type A get things done founder. And, you know, they broke all the rules, they they, you know, pushed pushed as hard as they could. They're managing at at its peak, like a hundred thousand people in secret. And they built the nuclear bomb in in three years, right? And so the way we would do that today, under procedural fetish framework, would be to put out a bunch of request for proposals. And have some kind of a competitive bid.

• 37:55Samuel Hammond?

  And and then we'd probably get like the lowest cost bid. And it would be like Lockheed Martin and they would, you know, build half an atom bomb. And it would take 20 years and five times the budget. And so so that's sort of what I'm getting at. It's not, it's not about process versus discretion per se. It's about the way process hobbles and straightjackets our ability to adapt and sort of represents a kind of sclerosis, a kind of a sort of like crystallized intelligence. We we lay down the things that worked in the past as process and sort of freeze those processes in place, ossifying a particular modality.

• 38:37Samuel Hammond?

  Right? And when the when the motor production shifts and you need to completely tear up that process, root and branch is is very difficult because often. There's no process for changing the process.

• 38:48Gus Docker?

  Yeah, I I I wonder if if there are lessons for how government will respond to AI in in thinking about how governments responded to to, say, historical, technical innovations of of a similar magnitude, uh, like the Industrial Revolution or the the printing press, or maybe the Internet computer. Do you think we can draw general lessons, or is it is it, uh, so specific that we can't really extract information about the future from them?

• 39:16Samuel Hammond?

  I think there are very powerful general lessons. You know, I think one of the first general lessons is that every major technological transformation in human history has has preceded a institutional transformation. You know, whether it's the shift from nomadic to to settle city-states with the agricultural revolution, or, um, you know, the rise of modern nation-states, or the the end of feudalism with with the printing press, to, you know, in the New Deal era, the sort of transition with industrialization from the kind of laissez-faire, classical liberal phase of, um, 18th century America to to an America with, um, a robust welfare state and administrative bureaucracies, um, and really an all an all-new constitutional order, right? And so there's sort of better and worse ways for this transition to happen.

• 40:01Samuel Hammond?

  There's there's sort of the internal regime change model, and you can think of, you know, Abraham Lincoln or or FDR as inaugurating a new republic, a new American republic. There or there's a scenario where we don't change because we're too crystallized and sort of like an innovator's dilemma, get displaced by some new upstart. And there are different countries have different abilities and different sort of capacities for that that internal adaptation. Um, as a Canadian, I'm a big fan of like Westminster-style parliamentary systems, and one of the reasons is because it's very easy for parliamentary systems to shut down ministries, open up new ministries, to reorganize the civil service.

• 40:41Samuel Hammond?

  Because it's it's sort of vertically integrated under the prime minister's office or what have you. In the US, it's much worse because given the separation of powers, you know, Congress and the executive are often not working well together. Just to put it just to as an understatement. But then moreover, the different federal agencies have a sort of a life of their own. Often they're, you know, self-funded and all these other things that make it very difficult to reform.

• 41:09Gus Docker?

  Do you think Canada responded better to to the rise of the internet than the US, for example? Isn't there something wrong with the story? Because the US kind of birthed the internet and Canada, you know, adopted the internet from from the US.

• 41:22Samuel Hammond?

  Let's compare, first of all, the the impact of the internet on weaker states. Because, you know, Canada and the US are similar or sort of in in in one quadrant. They have differences, but the differences are small compared to other countries. If you think about like internet safety discussions that would have been taking place in early 2000s, people would have been talking about, you know, identity theft, credit card theft, child exploitation, these kind of like direct first order potential harms from the internet. They didn't foresee that concurrent with the rise of of mobile and social media, that the internet would enable tools for mass mobilization simultaneous with a kind of legitimacy crisis where the sort of new transparency

• 42:05Samuel Hammond?

  and information access that the internet provided eroded trust in government and trust in other institutions. So you have these two two forces interacting. The internet exposing government and exposing corruption and leading to a decline in trust while also creating a platform for people to rise up and mobilize against against that corruption. Right. And it's something that that kind of rhymes with the printing press and the printing revolution, where you had these sort of dormant suppressed minority groups like the Puritans or the Presbyterians, the nonconformists. And with the with the collapse of the censorship printing licensing regime, they actually had a licensing regime

• 42:45Samuel Hammond?

  in in in the UK Parliament back circa 1630. That licensing regime collapsed, I think, 1634, something around there. And that that was like five years before the English Civil War. And you see something like this in the Arab Spring, where internet Quite directly led to mass mobilization in Cairo and Tunisia and elsewhere and led to actual regime change, in some cases, or temporary state collapse. And that's because those were weaker states that hadn't democratized, that hadn't sort of had their own information revolution earlier in their history the way we did, right? In some ways, like, the American Republic is sort of a founder country built on the backbone of the printing revolution.

• 43:31Samuel Hammond?

  So we were, we were a little bit more robust to that because it's sort of part of our ethos to have this open, disagreeable society. But clearly, it's also, the internet has also affected the legitimacy of Western democracies. I think it's clear, clearly, one of the major inputs in sort of rising populism, the mass mobilizations that we see, whether in the US context, the 2020, you know, racial awakening or, or the January 6th, sort of peasant rebellion, right? These, these look, these, these sort of look like the kind of color revolutions that we see abroad. And, you know, some people want to ascribe conspiracy theories to that. I, I think there's a simpler explanation, which is, which is that people will self-organize

• 44:17Samuel Hammond?

  with the right tools. Our state hasn't collapsed yet, but it's clearly, it, there's clearly a lot of, a lot of cracks in the foundation, if you will.

• 44:25Gus Docker?

  Is it, would it be fair to say that the main lesson for you from history is that technological change brings institutional change?

• 44:32Samuel Hammond?

  Yeah. You know, not, not necessarily one for one, I'm not, I'm not kind of a vulgar Marxist on this, but, but yes, and the, and the reason for that is because institutions themselves exist due to a certain cost structure. And if you have general purpose technologies that, that dramatically change that, the nature of that cost structure, then institutional change will follow.

• 44:51Gus Docker?

  Yeah. And I think we want to get to that, but before we do, I think we should discuss AI's impact on the, the broader economy. So, so not just the government, but, but the economy in general. Economists have this fallacy, they, they, they point out often, the lumber labor fallacy. Maybe you could explain that?

• 45:08Samuel Hammond?

  The lumber labor fallacy is essentially the idea that there's a fixed amount of work to be done. If you were...

• 45:15Samuel Hammond?

  Thinking about the Industrial Revolution and what would happen to the 50% of people who are in agriculture, you couldn't imagine the new jobs that would be created, but new jobs were created. And the reason is because human wants are infinite. And so, you know, demand will always fill supply. The second reason is because there's a kind of circular flow in the economy where one person's cost is another person's income. Society would collapse if we had true technological unemployment, because there'd be things being produced, but no one to pay for them. And so that ends up kind of bootstrapping new industries and new sources of production. There's still this open question, is this time different?

• 45:54Gus Docker?

  Yeah, that's what I, that is, that's exactly what I want to know. Because, I mean, for me, it's, in retrospect, let's say, it's easy to see how workers could move from fields to factories into offices. But if we have truly general AI, it's difficult for me to see where workers would move, especially if we have also functional robots and perhaps AIs that are better at taking care of people than than than other people are. I'm not asking you to predict specific jobs, but I'm asking whether whether you think this historical trend will hold with the advent of advanced AI.

• 46:33Samuel Hammond?

  You know, the first first thing to say is, you know, when Keynes wrote Economic Possibilities for Our Grandchildren, a famous text where he predicted that technological progress would lead to the growth of a leisure society.

• 46:49Gus Docker?

  And this was in the 1930s?

• 46:52Samuel Hammond?

  Yeah. You know, people have dismissed him as being wrong. But actually, you look at time use data and employment data, and people are working less. You know, it's not, it didn't match his, the optimism of his projection, right? Because it turns out, you know, maybe if we fixed living standards at what he expected, people, people want more and people will work more for more. But overall, people are working less. People do have more leisure. We've sort of moved to a de facto four-day work week. So there is one world where rapid technological progress continues that trend, and we all work less. It's sort of a technological unemployment that's spread

• 47:34Samuel Hammond?

  across people, and is enabled in part because in a world of AGI, maybe you only have to work a few hours a day to make a hundred thousand dollars a year. There's another possibility, which is that while AGI could in principle be a perfect emulation of humans on specific tasks, it can't emulate the historical formation of that person, right? So what I mean by that is if you had a perfect atom-by-atom replication of the Mona Lisa, it wouldn't sell at auction,

• 48:11Samuel Hammond?

  right? Because people aren't just buying the physical substrate, they're also buying the kind of world line of that thing. And that's clearly the case in humans as well. There are certain talking heads that I go and enjoy, not because they are the smartest or what have you, because I'm interested in what that person thinks on this, because they have a particular personality, a particular world line. The third factor is sort of artificial scarcity, right? And so even in a world with abundance and supply in services and goods, there are still things that will be intrinsically scarce, real estate being

• 48:55Samuel Hammond?

  probably the canonical thing, but also energy and commodities and so forth. And the reason real estate is intrinsically scarce is because people want to live near other people, and people want to live in particular areas of a city. They want to live in the posh part of town, right? And those are positional goods. We can't all live in the trendy loft. So that builds in a kind of artificial scarcity. And so people will still be competing over those things. This is sort of related to artificial scarcity, but there's also sort of break it out into a fourth possibility, which are tournaments and things that are structured as tournaments. Having chessbots that are strictly better than humans at chess hasn't killed.

• 49:36Samuel Hammond?

  You know, people playing chess, if anything, uh, more people play chess today than they've had in human history.

• 49:41Gus Docker?

  Yeah, it's more popular than ever.

• 49:43Samuel Hammond?

  Yeah, and the reason is because people like to watch other humans playing. And also it's they're structured as as sort of zero sum tournaments where there can only be the best human. You look at other things that have been created just in the last fifteen, twenty years, like the X Games, right? You know, I I think people will still want to watch other people do the Olympics or or do motocross and all these other things. And so maybe more of our life shifts into, you know, both um, maybe greater leisure on the one hand, more competition over uh positional goods and more um, more production that is structured as a tournament.

• 50:19Gus Docker?

  Yeah, yeah, I I can see many of those points. I'm just thinking, again, with with fully general AI, you would be able to to generate a a much more interesting person uh playing chess, or at least a simulation of a of a very charismatic and interesting uh human chess player. Why wouldn't people watch that chess player uh as opposed to to the most, to the to the best human?

• 50:43Samuel Hammond?

  Um, maybe they will. It's hard to know. Uh, the question is who's producing that that video stream? Because uh you still need the human behind it that's that had the idea, right? And um.

• 50:58Gus Docker?

  You could imagine people being dishonest about the history of of this chess player. That this this simulated chess player could could be uh fully fully digital, fully fully fictional, so to speak, and just pretending to be human.

• 51:13Samuel Hammond?

  Right, so i- they could fool people. Um, that's that's the case, too. No, I I can't rule that out, but I would just say that um, however that person is monetizing, their uh deep fake chess player, they're making money which they're then spending back into the economy, and so they'll produce jobs somewhere.

• 51:28Gus Docker?

  Do you think more people will move into, say, people focused industries like nursing and and teaching? Is that is that a possible way for for for us to to maintain jobs?

• 51:40Samuel Hammond?

  Maybe nursing, at least in the short run.

• 51:44Samuel Hammond?

  I'm not very long on education being labor intensive for much longer.

• 51:49Gus Docker?

  But you don't think education, at least say grade school education, is that really about teaching people or conveying knowledge, or to what extent is it about conveying knowledge? And to what extent is it about the social interaction and specializing your teaching to the individual student?

• 52:10Samuel Hammond?

  Well, AIs are very good at customization and sort of mastery to degree. Education is a bundle of things. And for younger ages, it's also daycare. It is socialization, like you said. At the very least, it suggests a reorganization of the division of labor. Those types of teachers that you would select higher for may differ if the education component of that bundle is being done by AI. And maybe you select for people who are maybe don't have any subject matter expertise, but are just highly conscientious and good around kids. Or maybe you one bundle from public education altogether and it rebundles around a jujitsu school or chess academy.

• 52:53Samuel Hammond?

  Because you'll have the AI tutor that will teach you math, but you'll still want to grapple with the human.

• 52:59Gus Docker?

  Yeah, yeah. What about industries with occupational licensings like law or medicine? Will they be able to keep up their quite high wages in the face of AI being able to be a pretty good doctor and a pretty good lawyer?

• 53:16Samuel Hammond?

  It's easy to solve for the far, you know, long-term equilibrium. With the rise of the internet, you can do a comparison of the wage distribution for lawyers pre and post-internet. And circa the early 90s, lawyer incomes were normally distributed around $60,000 a year. You know, after in the 2000s, they become bimodal. And so you have one mode that's still around that $60,000 range, those are like the family lawyers. And then you have this other mode that's into the six figures and those are like big law, right? It's the emergence of these law firms where you have a few partners on top and maybe hundreds of. Of associates who are doing kind of grunt work using Westlaw and LexisNexis and these

• 53:56Samuel Hammond?

  other legal search engines to accelerate drafting and legal analysis. So if that pattern repeats, I could imagine these various, you know, high skill knowledge sectors to also become bimodal, where in the short run, AI serves as a co-pilot, sort of like Westlaw or LexisNexis was for legal research, and enables the kind of 100x lawyer. And so there's a kind of averages over dynamic. Longer run, you know, you start to see the possibility of doing an end run around existing

• 54:37Samuel Hammond?

  accreditation and licensing monopolies, where, you know, obviously, the medical, American Medical Association and medical boards will be highly resistant to an AI doctor. You know, I tend to think that they'll probably end up self-cannibalizing because the value prop is so great, even, even, you know, for doctors to, you know, do simple things like automate insurance paperwork and stuff like that. But to the extent that there is a resistance, to the extent that, you know, in 10 years, there's still a requirement that you must, you know, have the doctor prescribe the treatment or refer you to a specialist, even if, even though the AI is doing all the work and they're just sort of like the elevator person that's like actually just pushing the button for you.

• 55:19Samuel Hammond?

  It'll be very easy to end run that because AI is both transforming the task itself, but also transforming its, the means of distribution. And if you can go to GPT-4 and ask for, put in your blood work and get a diagnosis, but no regulator is going to stop that, right? And so, you know, GPT-4 becomes sort of the ultimate doctor of up orders.

• 55:40Gus Docker?

  You write a lot about transaction costs and how changes in transaction costs change institutional structures. First of all, what are transaction costs and how do you think they'll be affected by AI?

• 55:54Samuel Hammond?

  So transaction cost is sort of an umbrella term for different kinds of costs associated with market exchange. And this goes back to Ronald Koest's famous paper on the theory of the firm. Where he asked the question, you know, why do we have corporations in the first place? If free markets are so great, why don't we just go and spot contract for everything? And the answer is, well, market exchange itself has a cost. There's the cost of monitoring, you know, if you hire a contractor, you don't know exactly what they're doing. There's the cost of bargaining, you know, having to haggle with a taxi cab driver is a friction. And there's the cost of association of search and information. So taking those three things together, they're not all that companies do, but they structure the boundary of the corporation.

• 56:38Samuel Hammond?

  They explain why some things are done in-house and some things are done through contracts. If there's high monitoring costs, you want to pull that part of the production into the company so that you can monitor and manage the people doing the production.

• 56:51Gus Docker?

  And some of the same effects go for the existence of governments, right?

• 56:56Samuel Hammond?

  Yes, because governments, you know, with a certain gestalt, governments and corporations aren't that different. There are kinds of institutional structures that pull certain things in-house and certain things are left for contracting or outsourced. And, you know, you even see sort of different kinds of governments having different parallels with different kinds of corporate governance, right? Relatively egalitarian democratic societies like Denmark are kind of like mutual insurers. Whereas more hierarchical authoritarian countries are more like, you know, like Singapore, say, is more of a joint stock corporation. And indeed, you know, Singapore was founded as an entrepot for the East India Company.

• 57:39Samuel Hammond?

  So there are very deep parallels. And it's also essential, transaction costs are essential to understand why governments do certain things and not other things. All Western developed governments guarantee some amount of basic health care, right? But Most, you know, outside of, say, the National Health Service in the UK, most of these countries guarantee the insurance. They don't necessarily nationalize the actual providers, right? And the reason goes to transaction costs and sort of an analysis of the market failure in insurance. Likewise with roads, you know, it's possible to build roads through purely private means.

• 58:20Samuel Hammond?

  And indeed, you know, countries like Sweden, a lot of the roads are run by private associations. But if you have lots of different boundaries, different micro jurisdictions and so forth, there can be huge transaction costs to negotiating up to a interstate highway system. And those transaction costs then necessitate public infrastructure projects.

• 58:44Gus Docker?

  So the transaction cost in this case would be being a private road provider. You have to go negotiate with 500 different landowners about building a highway, whereas a government can do some appropriation and simply build the road much, much faster or with less transaction costs at least.

• 59:03Samuel Hammond?

  Yeah, precisely. And we're seeing this dynamic in the US with permitting for grid infrastructure and transmission. You know, we're building all the solar and renewable energy, but to build the actual transmission infrastructure to get the electrons from where it's sunny to where it's cold requires building, you know, high voltage lines across state lines, across different grid regions. And there are all kinds of NIMBs and negotiation costs involved, holdouts and so forth. And so that, the more those kinds of costs exist, the more it militates towards a kind of larger scale intervention that, you know, federalizes that process.

• 59:44Gus Docker?

  Yeah, the big question then is how will AI change these transaction costs? What will the effects be here?

• 59:52Samuel Hammond?

  It's easy to say that they will be affected. And you know, obviously, the internet affected them to an extent. And we were talking, we talk about sort of the ease of mobilizing protest movements, or the kind of the sunlight that was put on government corruption. Those are, those are reflecting declines in the cost associated of information and coordination. I think AI takes us to another level. And I think it's, it's important to, to think through, in part because right now, the AI safety debate, at least in the United States, is very polarized between people who, who are like, everything's going to be great, and people who are like, this is like a Terminator scenario, or an AI kill us all existential risk.

• 1:00:35Samuel Hammond?

  You know, even if we accept the, you know, existential risk framing, there's still going to be many intermediate stages of AI before we flip on the superintelligence. And those intermediate stages have enormous implications for the structure of the very institutions that will need to respond to superintelligence or what have you. The ways we can see this is because all these information and, and monitoring and, and bargaining costs are directly implicated by commoditized intelligence. You know, start with the principal agent problem. You know, there is no principal agent problem if your agent does exactly as you ask and works 24/7, doesn't steal from the till, right?

• 1:01:19Samuel Hammond?

  And so AI agents dramatically collapse agency cost, monitoring. Now that we have multimodal models, in principle, we could have cameras in every house that are just being prompted to say, you know, is someone committing a crime right now? Whether we want to go that direction or not, it gives you a sense that, of how, you know, the costs of monitoring have basically plummeted over the last two years and are going to go way lower. And so you're starting to see this rolled out in the private sector with, you know, Activision has announced that they're going to be using language models for moderating voice chat in Call of Duty, right? And, and this, this is a more robust form of monitoring because in the past, you would have to like ban certain words, like certain swear words or, or.

• 1:02:04Samuel Hammond?

  Things associated with sexual violence. But then people could always get around those by using euphemisms. Right, like on YouTube, you know, the algorithm will ding you if you talk about coronavirus, or if you talk about murder or suicide, these things that throw off our flags. So what people have taken doing is saying they were unalived rather than murdered. Right, and that doesn't fool a language model. If you ask a language model, if you prompt it in a way to look for sort of broad semantic categories, not just a narrow word, it's much more robust. And so what that means, you know, you already start to see it, like I said, with Activision and the use of LLMs and content moderation, you're going to see it in the use of multimodal

• 1:02:49Samuel Hammond?

  models for productivity management and tracking. You know, Microsoft is unveiling their 365 Copilot, where you're going to have GPT-4 in Word and Excel and Teams and Outlook. But at the same time, you're also going to have a manager who is going to be able to say, you know, to prompt the model, tell me who is the most productive this week. Right, something as vague as that. And so you see this diffusion in the private sector. The question is, does it diffuse in the public sector? There's obvious ways that it would be a huge boom, right? You know, Inspector General GPT could tell you exactly, you know, how the civil service is working, whether there's corruption, whether there's a deep state conspiracy or something like that, right?

• 1:03:31Samuel Hammond?

  And at first blush, a lot of what government does is kind of a fleshy API. Bureaucracies are nodes that apply a degree of context between printing out PDF and scanning it back into the computer. It varies, there's degrees of human judgment that are required. But on first order, government bureaucracies seem incredibly exposed to this technology in a way that could diffuse really rapidly because, you know, going back to Microsoft 365 Copilot, Microsoft is the biggest IT vendor in US government. Right, and so you can imagine, once everyone has this pre-installed on their computer, that the person at the Bureau of Labor Statistics, who's in charge of doing the monthly employment

• 1:04:14Samuel Hammond?

  situation report, the jobs report, you know, at some point, he's going to be walking into work and hitting a button, right, that, you know, asking Excel to find the five most interesting trends and generate charts and the report is done. And in the private sector, that person will be reallocated and maybe doing things that the computer's not good at yet. But these positions are much stickier in government. To the extent that diffusion is inhibited on the public sector side, I worry about the kind of disruption and displacement of government services by a private sector that's adopting the technology really fast.

• 1:04:47Gus Docker?

  This is something we'll talk about in a moment. Before that, I just want to get to your complaints about isolated thinking about AI. You've sketched out some complaint about people thinking about AI only applying to one domain and then not really seeing the bigger picture. What are some examples here? Why do you worry about isolated thinking?

• 1:05:10Samuel Hammond?

  A few dimensions to this. One is what I've called the horse's carriage fallacy, right? The kind of view that, you know, what automobiles were was just a carriage with the horse, right? And so that anchors you to the older paradigm and it's like you're changing one thing and everything else stays the same. And you neglect all the second order ways that the development, the automobile, you know, enabled the build out of highway systems, the total reconfiguration of sort of the economic geography, right? And then implications for institutions like the state where, you know, once you have road networks or telegraph networks or any of these, these kind of networks, it suddenly becomes easier to monitor agents of the state and other parts of the country. And so you can, you

• 1:05:54Samuel Hammond?

  know, build out more of a federal bureaucracy. And so all these things were second order and were kind of neglected if you just were too focused on, on the first order effects of displacing the horses.

• 1:06:04Gus Docker?

  And in a sense, the second order effects turned out to be much more consequential in the end.

• 1:06:10Samuel Hammond?

  Yes, it seemed to always be. And likewise with the internet and sort of the I think this comes up a lot in how to think about AI use and misuse. There's lots of valid discussions there, but they're always very first order. And when you think about the way the internet has disrupted legacy institutions, yes, there's disinformation. But often the thing that's disrupting is not fake news. It's real news that's being repeated with misleading frequency, right? That's like throwing off our availability heuristic. Or it's valid things that, you know, valid complaints, whether, you know, the protests in Iran, right? The protests in Iran have this like striking parallel to the protests following the George Floyd protest.

• 1:06:55Samuel Hammond?

  And protesting in other countries where they even have like a three word chant, right? Or the case of the Arab Spring in Tunisia that started with a person self-immolating. Right? There's sort of like the structure that repeats where you have like a martyr or like some shocking event. And because of the way social media is organized, it synchronizes people around the event. In a way that's kind of stochastic, like it's like lightning striking. You don't know what event is going to strike on, but once we're synchronized. Then we start, you know, moving back and forth in a way that like causes the bridge debacle. Nothing about that is a misuse, right? Those are all valid uses, but their use is under collective action.

• 1:07:40Samuel Hammond?

  It's sort of solving not just for the partial equilibrium, but the general equilibrium. When everyone is doing this, and I think the person who wrote the best on this sort of conceptually was Thomas Schelling. And one of his little books, Micro Motives, Macro Behavior, had been influenced by me as a kid where he, you know, he talks about all these sort of like toy models where you're at a hockey game or basketball game. And something is happening, something exciting is happening on the, in the arena. And so people, the people in front of you stand up to get a better view. And then you have to stand up to get a better view of them, over them and so on. And so it cascades and suddenly everyone went from sitting to everyone went to standing and no one's view has improved. Right? And so these sort of.

• 1:08:21Samuel Hammond?

  General equilibrium, where you sort of solve for everyone's micro incentives and the kind of new Nash equilibrium that emerges. That ends up being the thing that drives the kind of multiple equilibrium shift from one regime to another. And and throughout, there may be no actual examples of misuse involved. It may just be people following their individual incentives.

• 1:08:43Gus Docker?

  I think it's it's worth stressing this point you make about the effects of earlier AI systems on our institutions. They might have effects that deteriorate our institutions such that we can't handle later and more advanced AI. And this ignoring this would be an example of isolated thinking and ignoring the second order effects, right?

• 1:09:06Samuel Hammond?

  Yeah, and they also it also changes the the sort of agenda, right? The AI safety agenda shouldn't just be about the first order of things or in alignment, you know, very important. But, you know, it's led to a discussion of do we need a new federal agency? And if so, what kind of agency? Whereas it may be more appropriate to think not what new agency do we need? But how do all the agencies change? Right? And and how do we sort of like brace for impact and enable a degree of co-evolution rather than displacement?

• 1:09:43Gus Docker?

  I don't know whether the question whether the question of how to get our institutions to respond appropriately is more difficult or less difficult than the problem of aligning AI. But it certainly seems very difficult to me. So is there are we making it harder on ourselves if we focus on the on the effects on the second order effects on institutions?

• 1:10:05Samuel Hammond?

  I mean, it's unavoidable. I mean, we can't pick and choose what kind of problems, but.

• 1:10:12Samuel Hammond?

  You know, the alignment problem, the hard version, is yet to be solved. But we have many examples of governments building state capacity and having kind of, you know, shifting from very clientalistic, sticky, corrupt governments to sort of modernized governments where state capacity is built and then that government can sort of break out of the middle and come trap and become rich.

• 1:10:36Gus Docker?

  You mentioned Estonia as an example of a country that's pretty advanced on the IT front, on the technology side. Maybe you could talk a bit about Estonia?

• 1:10:46Samuel Hammond?

  Yeah, I would just say, in general, it's hard for any organization to reform itself from within when there is path dependency, but I would say, at least we have examples of it being done, where we don't have examples of alignment being solved yet.

• 1:11:01Samuel Hammond?

  When it comes to Estonia, Estonia is an interesting case. It's sort of an exceptional case because after the fall of the Soviet Union and the breakup of the peripheral former Soviet states, they kind of had a blank slate, right? They also had a very young population and people who had a kind of hacker ethic within their civil service. And so with that blank slate and with that hacker ethic, they were very early to adopt and to foresee the way the internet was going to shape governments through a variety of e-government reforms. So early in the, you know, in the late 90s and into the 2000s, they were some of the earliest to digitize their banking system, like e-banking, to build this system called

• 1:11:44Samuel Hammond?

  X-Road, which is kind of like a cryptographically secured data exchange layer. It resembles a blockchain, but it was about a decade before blockchain was invented.

• 1:11:53Gus Docker?

  For exchanging information between different government entities. Your medical information could be uploaded to this, to the system and then be available to all systems that have the right to see that information.

• 1:12:06Samuel Hammond?

  Exactly, in a way that's cryptographically secured and distributed. So if a missile hit the Department of Education, you don't lose your education records because it's distributed. And that also enabled, you know, an enormous amount of automation where, for instance, this is my understanding, a child born in Estonia, once you file that birth record, it more or less...

• 1:12:29Samuel Hammond?

  initiates a clock in the system that will then then enroll your child in school when they turn four or five, like automatically, because it knows that your child has aged and unless it had a death record to cancel that out. But it also means you can, you know, do taxes and transfers, much simpler, you get your benefit within a week. It can integrate across different parts of public infrastructure, like, you know, use the same card to ride the bus as you do to to launch a new business. It also serves as a kind of platform for the private sector to sort of to do government by API, right, to build new services on top of

• 1:13:11Samuel Hammond?

  government as a platform and integrate with government databases.

• 1:13:15Gus Docker?

  Yeah. And so the point here for us is that institutional reform is possible, modernizing government is possible, at least under certain circumstances. We have proofs of concepts of this happening.

• 1:13:27Samuel Hammond?

  The hard thing is the path dependency. There is always a strong instinct to want to start from scratch, and it's normally not advisable because it's too hard. And so this is why it's hard in the US. Like, this is why you have African countries that leap progress in payment systems and so forth. The challenge of this decade or century is, how do we solve that path dependency problem? And how do we get to Estonia? It used to be get to Denmark. Now let's get to Estonia and find that sort of that pathway up Mount Improbable.

• 1:14:01Gus Docker?

  Great. Let's get to your wonderful series of blog posts on AI and Leviathan. In this context, what do we mean by Leviathan?

• 1:14:10Samuel Hammond?

  Well, this all interrelates. So Leviathan was the book Hobbes, Thomas Hobbes, wrote at the start of the interregnum after the English Civil War. And it was his, basically, his early political science, early defense of absolutist monarchy as a way to restore peace and order after a decade of infighting.

• 1:14:34Samuel Hammond?

  And Hobbes kind of hit on some basic sort of structural game theoretic properties of, you know, why we have governments at all, right? He talked about life being nasty, brutish and short in the state of nature, war of all against all. And peace is only restored when people, when people who don't trust each other offload enforcement and policing responsibilities to, to a higher power that can then restore a degree of peace and order. AI and Leviathan is talking about, you know, how does AI change the story? Does it reinforce the Leviathan? Does it lead to a digital police state à la China? Or is it something that we impose on ourselves?

• 1:15:18Samuel Hammond?

  And we talked about how multimodal models could in principle be used to put a camera in everyone's house and have it just continuously monitoring for people doing any kind of crime. That's something that North Korea might do. In, in the US context, it's something that we're very liable to just voluntarily do to ourselves. Because we, we want to have ring cameras and, and Alexa assistants and so forth. And so that leads to a kind of bottom up Leviathan that is potentially no less oppressive and maybe even more oppressive because, because it's, it's, there's no one that we can appeal to, to change the rules.

• 1:15:54Gus Docker?

  Yeah, so Leviathan is one way to respond to technological change. But you mentioned two other ways we, we could alternatively respond.

• 1:16:02Samuel Hammond?

  Right, so, so basically anytime a technology greatly empowers the individual, it, it creates, it creates a potential negative externality, right? Hobbes called these are natural liberties. In the state of nature, I have a natural liberty to, to kill you or to strong arm you. And governments exist to revoke those natural liberties, right? But, but for a higher form of freedom, right? And so there are sort of, anytime a technology greatly increases human capabilities vis a vis other humans, the three canonical ways we can adjust are, you know, ceding more authority to that higher power, the Leviathan option. And then the other two options are, you know, adaptation and mitigation and, and normative evolution.

• 1:16:45Samuel Hammond?

  So the example I give is, is, you know, if, if suddenly we all had X-ray glasses and could see through walls and see through clothing, you know, one option, we, we have a draconian totalitarian crackdown that tries to seize all those X-ray glasses. Another option is we Adjust normatively culturally that we, our privacy norms wither away and we stop caring about nudity. And then the other option is adaptation mitigation, where we put in mesh into our walls and wear leaded shirts and pants.

• 1:17:20Gus Docker?

  Yeah, I guess continuing that analogy a bit between the smart glasses and AI. You have this amazing write-up of ways in which AI can increase the informational resolution of the universe. So you give some examples that are, I'm thinking specifically of AI identifying people by gate, for example.

• 1:17:44Samuel Hammond?

  Right, so gate recognition is nothing new. China has had advanced forms of gate recognition for a while now. So even if you cover your face, it turns out we're constantly throwing off sort of ambient information about ourselves, about everything. And the way you walk, the particular gate that you have, is a unique identifier. Another example is galaxy surveys. We've had from Hubble telescope to now the JWST, tons of astronomical surveys of distant galaxies and so forth. And all of a sudden, all that old data, it's like that same data set is now more useful.

• 1:18:26Samuel Hammond?

  Because applying more modern deep learning techniques, we can extract entropy that was in that data set, but we didn't have the tools to extract it and discover that there are new galaxies or other phenomena that we missed.

• 1:18:40Gus Docker?

  Another example you give is listening for keystrokes on a keyboard and extracting information about a password being typed in, for example. Which is something that of course humans can't do, but we can do with AI models.

• 1:18:56Samuel Hammond?

  Yeah, so that was a paper showing that you can reconstruct keystrokes from an audio recording, including a Zoom conversation. So I hope you haven't typed in your password because people in the future. And so this goes to the fact that it's sort of retroactive, that even if the technology wasn't diffused yet, any Zoom conversation, any recording where someone typed their password in the future will be like those galaxy surveys, where someone will go backwards in time and. And you know, turn up the information resolution of that data.

• 1:19:23Gus Docker?

  Yeah, this is pure speculation, but I wonder if, I mean, imagine anonymized people in interviews, say, 10 years ago, whether they will be able to stay anonymous or whether AI will be able to extract the data about their face or their voice that wasn't technically possible when the interview aired.

• 1:19:42Samuel Hammond?

  Yeah, exactly. There are already systems for like deep pixelating. You probably do something similar for the voice modulation. And then also sort of, you know, again, going back to like this ambient information we're always shedding, identifiers in the way we write, you know, the kind, where we place a comma, the kinds of adverbs we like to use and so forth. People just dramatically underrate, you know, how much information we're shedding, in part because we're blind to it.

• 1:20:10Gus Docker?

  Some people who are taking great efforts to stay anonymous online, people in the cryptography space, for example, will put their writings through Google Translate to French and then back to English to erase subtle clues to how they, that could identify them personally. Why is AI so much better at tasks like the ones we just mentioned compared to humans?

• 1:20:35Samuel Hammond?

  Well, it goes back to what we were talking about with sort of putting information theoretic bounds on AGI. When you minimize the loss function in a machine learning model, you're trying to minimize the cross entropy loss. The cross entropy is how many bits does it take to distinguish between two data streams? And if it takes a lot of bits to distinguish between the two, that means they're relatively indistinguishable. So this going again to the Turing test, like if we have a Turing test where I can tell right away that the AI is different from the human, that suggests a high cross entropy. But if I had to talk to it for days and do all kinds of adversarial questioning, I might still be able to, in the end, tell the difference between the two, but we've minimized that cross entropy loss. And so when you have any arbitrary data distribution that you're trying to predict,

• 1:21:21Samuel Hammond?

  whether it's, you know, trying to predict. Galaxies and astronomical data or passwords from fingerprint data on a phone screen, you know, all these things embed a kind of physical memory of the thing in question and can often be reconstructed through this kind of loss minimization where you have a system that asymptotically extracts the entropy that was latent in the data. And this can be done in a way that is often quite striking where, you know, we can with stable diffusion make fairly accurate predictions of what people are imagining in their mind using FMRI data. And FMRI data is like blood flow data in the brain is a very lossy

• 1:22:06Samuel Hammond?

  representation of whatever's happening in the brain. But there's still enough latent entropy in there that we can kind of reverse engineer or decompress it into a fuller picture.

• 1:22:16Gus Docker?

  And this could turn into a form of lie detection?

• 1:22:19Samuel Hammond?

  Yeah, I think it already basically has. If you have, you know, FMRI data or, you know, EEGs or other kinds of like direct brain data is probably, you know, is probably a lot easier. But we already have systems that are over 95% accurate at detecting deception from just visual video recordings.

• 1:22:42Gus Docker?

  We can see how all of this information that we are continually shedding gives rise to the possibility of a Leviathan either of the private or of the government's kind. I wonder what world do you see open sourcing AI models playing here? What are the trade offs and risks in open sourcing AI?

• 1:23:03Samuel Hammond?

  Among the people who are most bullish to open source, there's often a kind of libertarian ethic undergirding it, right? Regardless of whether that's a good idea or not, one of the things I'm trying to communicate to that to that group is to say that be careful what you wish for because of these kind of paradoxical Hobbesian dynamics, you know, the fact that in America, you never know if someone has a gun or not, you know, on the one hand, the Second Amendment enhances our freedom. On another hand, you know, you don't get the sort of like everyone's doors unlocked and people are, you know, like the police in England don't even have guns, like there's a certain freedom that derives from us not all being, you know, heavily armed.

• 1:23:50Samuel Hammond?

  And likewise with with open sourcing powerful AI capabilities empowers you as an individual, but in general equilibrium, once we all have the capabilities, the world could look much more oppressive, either because we're all spying on each other all the time and we can all see through each other's walls, or because there's a backlash and the introduction of sort of Leviathan type solutions to restrict our ability to spy on each other all the time. And, you know, my my general sense is that we can only delay, and we can't really prevent things from being open source over the long run, because there's a sort of trickle down of compute requirements. But in the interim, you know, there are definitely things that are valuable to open source, you know, having

• 1:24:34Samuel Hammond?

  70 billion parameter language models, not a not a threat. In fact, I think it's probably useful for alignment research for something like that to be open source. But if you are a researcher and you've developed a emotional recognition model that can tell if, you know, with like 90 99% accuracy, whether someone is lying or or not lying, and whether your girlfriend loves you or not, like these these things, or the ability to see through walls using like I talk about the use of Wi Fi displacement, there are people have built pose recognition models using displacement of the electromagnetic frequency of your Wi Fi, and they can see they can, and it was while penetrating. So you can see through

• 1:25:19Samuel Hammond?

  walls, like, what's the rush to put that on hugging face? And to like, make it as democratized as quickly as possible? I would say that if we if we value the adaptation and mitigation pathway, as opposed to the Leviathan pathway, then there's a there's a value in, you know, slow rolling some of these things.

• 1:25:43Gus Docker?

  How do you think a government power will be or relative government power will be affected by by AI? So you you write somewhere in this long series of blog posts that AI will cause a net weakening of governments relative to to the private sector? Why is that?

• 1:26:00Samuel Hammond?

  Yeah, specifically, Western liberal liberal governments under constitutional constraints.

• 1:26:07Samuel Hammond?

  So if you imagine society being on this kind of knife edge, I talk about this in the context of Darren S. Mogul's book, The Narrow Corridor, where he describes liberal democracy as sort of being in this corridor between despotism on the one hand and anarchy on the other, and we sort of stay in this saddle path where society and the state are kept in balance. If you veer off that path, you can, on the one hand, you know, the state could become all-powerful, and that's the sort of China model, more authoritarian digital surveillance state. And indeed, you know, China built up their digital surveillance state and their internet firewalls and so forth, after watching the Arab Spring and seeing how the internet was destabilizing to weaker governments. And so I fully expect

• 1:26:52Samuel Hammond?

  that AI will be very empowering and self-reinforcing of the power of the Chinese government. Indeed, there are draft regulations for large language models stipulate that you can't use the model to undermine national unity or challenge the government, and so they're baking that in. In liberal democracies, we think of ourselves as open societies, and the issue is that we're only open at the meta level. There's the public sphere, right? There's freedom of information laws. We have freedom of speech. I don't have freedom of speech if I walk into a Walmart. Wait, right? The Walmart is private property. In open societies, it's not that we don't have social credit scores and forms of thicker forms of social

• 1:27:34Samuel Hammond?

  regulation. It's just that we offload those functions onto private, competing private actors, whether it's a church that has very strict doctrines to be a member, or, you know, other kinds of social clubs. The fact that, you know, these days, if you want to go to a comedy club, they'll often confiscate your phone at the door because they don't want you recording the comedian's set and putting it online. My anticipation is that because of those constitutional constraints that limit the ability of liberal democracies to go the China route, right, because of our civil laws or bills of rights and so forth, and also because of a lot of these procedural constraints, this will naturally shift into the private sector. And

• 1:28:18Samuel Hammond?

  We see that already with the use of AI for monitoring and employment, for policing speech in ways that would be illegal if done by the state, but are fine if done by Facebook. To the extent that the AI continues to increase these kind of negative externalities and therefore puts more value on having a sort of vertically integrated experience, a walled garden that can strip out the negative forms of AI and reinstate a degree of harmony between people. That more and more of our social life will be mediated through these sort of private organizations rather than through a kind of open public sphere.

• 1:28:59Gus Docker?

  You're imagining that government services will be gradually replaced by private services that are better able to respond. Want governments fight to uphold individual rights? In Walmart or on Facebook, you are regulated in ways that the government couldn't regulate you. But you still have the choice to go to Target instead of Walmart or to go to, or X instead of Facebook. Isn't that the fundamental thing? So the fundamental thing is the choice between services. And won't governments uphold citizens' rights to make those kinds of choices?

• 1:29:36Samuel Hammond?

  Yeah, no, I agree. And so this would be the, the defense of the liberal model is that we allow thicker forms of social regulation because it's moderated by choice and competition. And the issue with Chinese Confucian integralism isn't the fact that it's super oppressive. It's the fact that you only have one choice and you don't have voice or exit. So, yeah, but it's obviously a matter of degree, right?

• 1:30:10Samuel Hammond?

  When ride hailing first first arose, you know, I remember back in 2013, 2014, it wasn't that long ago, you know, I think Uber was founded in 2009, but it really only started taking off in the in the early 2010s. You know, people thought it was crazy to ride a car with a stranger. And then within five years, it was the dominant mode of ride hailing. And in that five year period, essentially, we saw a kind of regime change in micro, where taxis went from being something that was regulated by the state through these commissions that were granted, you know, legal monopolies, and use licensing and exams

• 1:30:53Samuel Hammond?

  and other other sort of brute force ways of ensuring quality, to competing private platforms where you have, you know, Lyft or Uber to choose from, and they replaced the explicit governance of legal mandates with the competing governance of reputation mechanisms, of dispute resolution systems, of structured marketplaces that that that collapsed the bargaining frictions, right? You never have to haggle with an Uber driver, you just sort of get in. And that was obviously a much better way of doing, you know, ride hailing. So even though there was sort of a violent resistance early on, you know, literally, like in France, they were throwing rocks off of bridges and cab drivers in New York were killing themselves. So for the people affected, it was a very dramatic sort

• 1:31:37Samuel Hammond?

  of regime change, but for everyone else, it was a huge positive improvement. And yet, it's only made possible because Uber has a social credit score. If your Uber rating goes too low, you'll get kicked off the platform. And so we're fine with social credit scores. It's when you only have one and don't have an option, and it can follow you across all these different verticals that it becomes a problem.

• 1:31:59Gus Docker?

  Do you imagine that because of of rising danger in the world, you talk about the the externalities from the widespread implementation of AI all across society, because of those dangers, those externalities, you know, you will either use Uber or whatever service, or you're you can't participate in society. Do you imagine increased pressure in that direction?

• 1:32:25Samuel Hammond?

  It does seem to be a longer term trend. I don't know if AI will, you know, accelerate it. I have another series of essays that I call Separation Anxiety.

• 1:32:36Samuel Hammond?

  And it's a reference to the fact that in insurance markets, there's kind of two equilibrium. There's the pooling equilibrium where we're pooled together into one risk pool. And then there's a separating equilibrium where the insurance pool unravels and we break up into, you know, the great power insurance for like senior citizens who never had an accident and stuff like that. And it turns out that insurance markets are competitively unstable, that without government regulation or social insurance, that insurance markets will naturally tend to unravel because of adverse selection into, you know, the the high-risk people being in one pool and the lowest people being in another pool. And it turns out you can sort of use that as a mental model to look at other kinds of implicit pooling equilibrium.

• 1:33:21Samuel Hammond?

  Right? So in, in, within company wage distributions, often there is, you know, 20% of the workers who are doing 80% of the work, but they're, they're pooled together under one wage structure. And that was sort of the dominant structure of, of the period of wage compression in the United States in the 50s and 60s. And once we had better monitoring technologies and were able to tell who, who were the 20% that were doing 80% of the work, it suddenly became possible to, to differentiate pay structure. And a lot of the rise in inequality in the United States is actually between firm. So what, you know, what happens is Ezra Klein is like the most productive whiz kid at the Washington Post. And he realizes, why don't I just go start my own website? Right?

• 1:34:06Samuel Hammond?

  And so that, that dynamic sort of played out across a variety of domains leads to a world that, you know, to the extent that these features are correlated, that does separate. Right? Where you have, you know, the one-star Uber riders driving the one-star Uber drivers, the drivers driving the riders. And, you know, people who have, you know, the five-star Uber ratings and the perfect credit scores self-sort into communities with other people with perfect driving records and perfect credit scores. And that, and, you know, we see that to an extent already with the. You know, enclaves of rich zip codes with private schools and everyone is sort of self-selected. AI could, it seems to me that AI would would exacerbate that. I mean, at first blush, just because going back to the point about signal distraction,

• 1:34:51Samuel Hammond?

  it can find all these different ways. You're a high-risk type and I'm a low-risk type and so forth, that are probably latent in all kinds of data that we don't even need to give permission to the insurance company. They'll just like, the same way that they use like smoking or going to a gym as a proxy, there's all kinds of proxies they could use and likewise for employers and how they pay people.

• 1:35:12Gus Docker?

  Society kind of runs on us not being entirely open and entirely honest all the time, otherwise you wouldn't be able to have kind of smooth social interactions and so on. Won't these norms be inherited by the way we use AI?

• 1:35:27Samuel Hammond?

  Yeah, I think this is a really big issue. I'm a big fan of Robin Hanson and a lot of his writing on social status and signaling is is sort of presenting humans as basically hypocrites. Like, we're constantly deceiving other people and we often deceive ourselves, so it's better to deceive others, as the evolutionary biologist Robert Trivers once pointed out. So, you know, all the kinds of polite lies that we tell are, I think, critical lubricants to the social interaction and actually like, it's it's good that there's a gap between

• 1:36:12Samuel Hammond?

  our stated and revealed preference. You know, I think a world where we all lived our stated preference could be hellish because we don't actually mean it. And AI, you know, has a direct implication on that because if I can have a pair of AR glasses on that will tell me if you're interested, if you're bored, if you're if you're on a date and, you know, are you really attracted to me? You know, all that sort of polite veneer, that social veil could be lifted in a way that we'll probably want to coordinate to not do, right? But again, it's this Nash equilibrium where, you know, it's in my interest to know whether you're interested or bored. And so I'll want to have the glasses on and my ideal world is where only I have the glasses and you don't.

• 1:36:59Samuel Hammond?

  And and the other way that our hypocrisy is being exposed and challenged is is the need to, you know, explicate the utility function that we want these models to to work under. You know, we need to formalize human values if we want to align these models. And so then we have to be honest and open about the fact that about the fact that our stated preferences probably aren't our true preferences. And that's that's a very challenging thing because it goes it cuts right to the nature of the human condition and involves topics that are intrinsically things that we lie to ourselves about.

• 1:37:35Gus Docker?

  You have what you call a timeline of a techno feudal feudalist future, which I found quite interesting. Yeah, it's it's great writing and it's very detailed. We don't have to go through it in all of its detail, but maybe you could you could tell the story of what happens in what you call the default scenario. This is the scenario in which Western liberal democracies are too slow to adapt to AI. And so we get something like a replacement of government services with more private services. What happens in the techno feudalist future?

• 1:38:06Samuel Hammond?

  Right, and and this sort of piggybacks and everything you've just been discussing, right? And I I don't want techno feudalist to carry too much of a pejorative. I'm sorry, using it descriptively. And certainly, some people would prefer this this this world. So the the example of Uber and Lyft displacing taxi cabs is sort of a version of this in micro, where we go from this regulated taxi commission to competing private platforms that use various forms of of artificial intelligence and information technology to replace the thing that was being done by explicit regulation.

• 1:38:44Samuel Hammond?

  And as AI progresses and both creates a variety of new negative externalities, whether it's like, you know, suicide drones or or the ability to spy on each other.

• 1:38:58Samuel Hammond?

  There's going to be a demand for new forms of security and also kinds of like opt-in jurisdictions that like tie our hands in the same way that we give up our phone at the before we go into the comedy club. And so I think this leads to a kind of development of clubs, the kind of club structure, maybe at the city level, as the vertically integrated walled garden that will that will police and build defensive technologies around the misuse of AI, and at the same time, provide a variety of like new AI native public goods that are only possible once AI unlocks them. And it's easy to see how this could very quickly displace and eat

• 1:39:43Samuel Hammond?

  away at at formal government services, both because we saw it already with Uber, but also if you map that that model to other areas of regulatory life, you know, does it make sense to have a USDA farm inspector, a human person has to go to a commercial farm and you know, maybe only goes to that farm once every few years because there's so many firms and only so many people. And, you know, does a little checklist and says, oh, you're you're not abusing the animals and like you got all the process in place and you get the USDA stamp of approval. Or does it make more sense to have, you know, multimodal cameras on in the in the farm 24/7 that are continuously generating reports that throw up a red flag

• 1:40:27Samuel Hammond?

  anytime someone sneezes on the conveyor belt? And to the extent that government is going to be slow at adopting that, will there be a push for the kind of Uber model of governance as a platform where you have the kind of AI underwriter, the consumer reports that sells these firms, the camera technology and the monitoring technology and builds their own set of compliance standards. And then you want to go to those firms or what have you that have the stamp of approval of the underwriter because it's much, much higher trust, it's sort of it's sort of like the end of asymmetric information. And you can map that from, you know,

• 1:41:07Samuel Hammond?

  Food safety to product safety to OSHA and workplace safety. You know, there's other parts of government that maybe just rendered completely obsolete. Right. Like, once we have self-driving cars that are 1,000x more safe than humans, do we need a National Highway Traffic Safety Administration?

• 1:41:27Samuel Hammond?

  You know, once we have sensors that are privately owned everywhere and can model weather patterns better than the National Oceanic Administration, do we need a National Weather Service or could we bootstrap that ourselves? And then, and then, once we have, you know, AI accelerated drug discovery, do we want to rely on the FDA to be a kind of choke point to do these sort of frequentist clinical trials that are inherently slow and don't capture the kind of videosyncras- synchrises and heterogeneity that could be unlocked by personalized medicine? Or do we move to an alternative drug approval process that is maybe non-governmental,

• 1:42:10Samuel Hammond?

  but much more rapid and much more personalized?

• 1:42:13Gus Docker?

  That's, that's the overall picture. I'll just run through the timeline here, picking up on some, some, some of your comments that I thought were especially interesting. You write, and this is, this is in 2024 to 2027, you write that the internet will become Balkanized and you will have, it will become more secure and more private, in a sense. Why, why does that happen?

• 1:42:38Samuel Hammond?

  We're already starting to see this a little bit, right? Once people realize that the data that's being generated on Stack Overflow or Reddit or whatever is valuable for training these models, suddenly everyone's closing their API, you know, and consequently, Google searching, Google search and the Google index have sort of started to degrade already. So I think that will continue for the kind of like privatization of data reasons. Then when you also think about how websites are going to handle sort of the growth of bots and catfishes and catfish attacks and, and cyber attacks and so forth, it makes sense that we're going to move from a sort of open-

• 1:43:19Samuel Hammond?

  You know, everything goes kind of Twitter-esque platform to things that are much more closed because they require human verification and identity verification to sort of build the trust that you're talking to other people and not deepfakes. And then medium term, again, over this sort of 2024 to 2027 horizon. You could also start to see the emergence of, you know, intelligent malware, sort of modern AI native cyber attacks that could be devastating to legacy cybersecurity infrastructure in a way that, you know, I talk about could harken back to the famous Moore's worm that

• 1:44:02Samuel Hammond?

  in the late 80s, basically shut down the early internet. Like, they literally had to partition the internet and turn it off so they could rid the network of the worm. So for all those reasons, I think you start to see the internet balkanize. And then particularly at the international level, we're already starting to see sort of the semiconductor supply chain become a critical part of national security. The growth of, you know, the Chinese firewall. The European Union is going to have to have their own quasi firewall and they kind of already do with GDPR and the EU AI Act. And so the kind of nationalization of compute and telecommunications infrastructure that will take off once people understand both the security risks and the value prop of owning the infrastructure for the AI revolution.

• 1:44:46Gus Docker?

  Yeah, in 2028 to 2031, you write about alignment turning out to be easier than we thought with the increasing scale of the model. That was somewhat surprising to me. Why does alignment turn out to be easier?

• 1:45:01Samuel Hammond?

  And part of this is, you know, imagining a scenario where alignment is easy. So we can talk about what happens if alignment is easy. But I think there are reasons to think that the classic alignment problem will be easier than people think. You know, I think that some of the early intuitions about the hardness of the alignment problem were rooted in a view of, you know, maybe AI turns out to be a very simple algorithm rather than like a deep neural network that achieves its generality because of its depth. You know, clearly, the kind of Value, I forget what Eliezer Yurkiewicz called it, but the, the, there's like a value alignment

• 1:45:43Samuel Hammond?

  problem where, how do we teach the model our values? That that part of the alignment problem seems trivial now because our large language models aren't like autistic savants. They're, they're actually incredibly sensitive to soft human concepts of value and context. They're not going to have a, the paperclip maximizer sort of monkey paw kind of threat models don't really make sense in that world.

• 1:46:11Gus Docker?

  But, but there's a difference between the output of the model and the weights or the, the, what the model has learned. And so the, just because a model can say, you know, it can say the right words that we want it to say, but what has it actually learned? We are not entirely sure. And so it has learned to satisfy human values to some extent, but has it learned to, has learned to want to comply with human value out of distributions or, yeah, in other domains. And in a deep sense, I'm not, I'm not sure about that.

• 1:46:43Samuel Hammond?

  No, I agree. So, so I'm sort of just laying some of my groundwork for, to explain my priors on this. Now, I agree, like, you know, reinforcement learning from human feedback is not alignment. In the same way that, you know, you could argue that like the co-evolution of cats and dogs with humans led to a kind of reinforcement learning from human feedback in, in their, in their short run evolution that, you know, made them look, appear as if they, you know, they experience guilt and shame and these human emotions when in fact they're, they're just sort of a simulacra of those emotions because it means that we'll give them a treat.

• 1:47:19Gus Docker?

  But I, I've done plenty of episodes on deception in, in, in these models and so on. We, we don't have to go through that, but I just wanted to point out that, yeah, maybe there's some complexities there.

• 1:47:29Samuel Hammond?

  So the first, my first prior is that these models aren't autistic savants the way they, the way they might've been. The second is going back to universality. Well, well, it is true that you, there are, that, you know, it's possible through reinforcement learning from human feedback, for example, that you, you're, you're, you're not selecting for honesty, you're selecting for. A deep fake of honesty, but in the bigger picture, the intuition that these models are converging or convergence with human representations should give you some confidence that they're not going to be as alien as we as we think they will be. It's also useful input for thinking about interpretability.

• 1:48:11Samuel Hammond?

  You know, some recent work showing that discussing sort of representation interpretability, where where instead of trying to interpret individual neurons, you interpret sort of collections of neurons and and circuitry and through sort of human interpretable representations. And one of the one of the lessons of universality is that like some of these high level human concepts like happiness or or anxiety, like these seem like vague psychological abstractions that there's no way that can correspond to like the micro foundations of the way our brain works. But in fact, they may actually be very efficient, low dimensional ways of talking about this, what's happening in our brain. And then the third thing is, I think that I just have seen, you know, my sense is that

• 1:48:55Samuel Hammond?

  the work on interpretability is actually making some some good progress. You know, whether it can scale is another question, but I think we'll get there. In my timeline, I talk about sort of AGI level models within the human emulator plus domain. I do later on talk about like super intelligence emerging maybe in the 2040s. And that's another story, right? And so I think some of this stuff maybe goes out the window if we have, you know, models that are bigger than all the brains combined and have like strong situational awareness. But I don't think that happens this decade. Certainly, certainly not with the current way we're building these models. With the way we're currently building these models, I think it comes much closer to a simulacra of the human brain.

• 1:49:38Gus Docker?

  Got it. In in in 2036 to 2039, you talk about robotics being solved to the same extent, or maybe even in the same way as as we are now solving language. That was, I found that super interesting. Explain to me, why? Why would robotics suddenly or quite relatively suddenly become much easier? Roboticsists have been fighting for decades to get these these models to to walk relatively unencumbered and it's it's been an uphill battle. Yeah, why can we solve robotics in the in the 2030s?

• 1:50:15Samuel Hammond?

  This may end up happening sooner than than I project, but.

• 1:50:20Samuel Hammond?

  I mean, if you look at LLMs, one of the stylized sort of trends with large language models is that natural language processing went from being this study of how to make machines understand language, went from being a dozen different sub-disciplines. People working on parsing, people working on syntax, people working on semantics, people working on summarization and classification, and these are all different research directions. And then along comes transformer models, and it supplants everything, and LLMs can do it all. And I think robotics is still in that ancient regime, where a lot of what Boston Dynamics

• 1:51:01Samuel Hammond?

  does is ad hoc control models, analytically solvable differential equations, different kinds of object recognition modules, and control action loops and so forth. And so it's still in that early NLP phase where they have 12 different sub-disciplines, and they're sort of mashing them together. And of course, you get something that's not very robust. I think we're already starting to see that paradigm shift to end-to-end neural network trained models, like Tesla, for instance. I think one of the reasons why Tesla cars had a sort of temporary decline in performance

• 1:51:41Samuel Hammond?

  was because they were undergoing the transition from these ad hoc lane detectors and stop sign detectors and stuff like that to a fully end-to-end neural network transformer-based model. And that tends to be a much more robust way to train the model, because stop signs look different in different countries, and maybe a stop sign isn't the thing you care about really, so on and so forth. And so I think the transformer-scale deep learning revolution is only now coming to robotics, and people in that field are a little bit cynical because they're used to relatively small RL models, thinking that the fit with actuators and some of the hardware

• 1:52:24Samuel Hammond?

  is a really challenging problem, and also believing that we don't have the data sets for it. But then you look at this recent robo-dog that you may have seen on Twitter, a fully open-source robot model for a Boston Dynamics-style dog. It was trained on H100s, 10,000

• 1:52:46Samuel Hammond?

  Human years of training and simulation and then some fine tuning on real world data. And they have a very robust robot control model that you could plug into all kinds of different form factors and have something that can, you know, hop gaps and climb stairs and do all the things that Boston Dynamics robots don't do very well outside of their distribution.

• 1:53:10Gus Docker?

  Do you think we'll have a general purpose algorithm that we can plug into basically arbitrarily shaped robots that can then navigate the navigate our apartments or our construction sites or maybe our highways? That's, that's, that's an interesting vision. Why, why is it that we achieve this level of generality?

• 1:53:29Samuel Hammond?

  If you look at humans, you know, humans are very good at, you know, if we've suffered an amputation or you have to go through physical therapy and it's not easy necessarily, but, but humans are able to adapt to different kinds of physical layouts of our body. And I think there will be a trend towards unified robotic control models that aren't like super tailored to, you know, two legs and two arms and so on and so forth. You know, once you've installed it through a little bit of in context learning or, or fine tuning or reinforcement learning, adapt to that particular form factor.

• 1:54:09Samuel Hammond?

  And this will parallel the kind of pre-trained foundation model paradigm that is currently taking place in LLMs where you have like the really big foundation model that can sort of do everything reasonably well and then you can fine tune it beyond that.

• 1:54:23Gus Docker?

  If we get to the 2040s in your timeline, you talk about massive amounts of compute being available. You talk about post scarcity in everything except for land and capital. And then you also talk about the development potentially of, of superintelligence at that point. What, what happens there? Who is in control of the superintelligence, if anyone?

• 1:54:47Samuel Hammond?

  Yeah, this is sort of where I start to get a little bit tongue in cheek, but. You know, I, first of all, I talk about how, you know, I tend to think that, you know, once we have exascale computing, and I think DOE just built their first exascale computer, maybe it was private company, but, you know, we have like one exascale computer in the world, by the 2040s, they'll be, they'll be commonplace. And if we ever worried about sort of, you know, controlling the supply of GPUs, you know, I don't, I don't, I don't know, I don't know exactly how much compute will be on our smartphones, but it will definitely be possible to train a, you know, GP5 model from your home computer. And so

• 1:55:31Samuel Hammond?

  any kind of AIC regime that we build today, that doesn't take into account that falling cost of compute will probably break down. And therefore, you know, amid this broader sort of fragmentation of the machinery of government, the state, I expect more and more government functions to be offloaded into, you know, basically private cities, HOAs, gated communities. And likewise with the internet, I expect more and more of our sort of permissioning regime for new AI models and deployment to shift to the infrastructure layer where, you know, telecommunications providers will be monitoring network traffic for, for unvetted AI models and so forth. And we'll have like Chinese style firewalls,

• 1:56:16Samuel Hammond?

  but that are specific to a particular local area network. And at that point, the world looks, the United States where this takes place, looks more like an archipelago of micro jurisdictions of, I tend to think that like a post scarcity political economy is, looks, looks a lot like the Gulf states, Gulf state monarchies, right? Because Gulf state monarchies are basically living post scarcity, right? They have a spigot of oil they can turn on, and then they can go build mega projects in the desert. And they have like infinite labor because they can just import guest workers. And so you end up with like this, this, uh, but if, if we can't have a Gulf state monarchy in the United States, instead we have a bunch of micro monarchies, um, dotting the country.

• 1:57:02Samuel Hammond?

  So I, I sort of jokingly say, you know, um, who's going to stop the free city of California? That's like, uh, home to all the trillionaire ML engineers and tech founders from the decade prior from, um, plugging in their, their, uh, humanity sized supercomputer into a fusion reactor and turning it on.

• 1:57:21Gus Docker?

  Yeah. And this is really your, the kind of end point of, of the discussion or your, your main point of, of institutions being eroded and then afterwards, uh, being unable to respond, uh, to, to strong AI.

• 1:57:36Samuel Hammond?

  Yeah. And leading up to this, it sounds like a scary dystopian type of thing. It doesn't have to be, right? Um, Uber is not dystopian. Airbnb is not dystopian. Private airports in other countries are, are way better than the public airports in the United States. So privatization and the sort of techno feudalist paradigm doesn't have to be bad, but what it, what it is, is more adversarial, right? And, you know, people have, um, sometimes speculated, you know, what, what do we have? The, the, the crumbling of the Roman Empire, was that kind of prerequisite to a renaissance, right? Because it allowed for these principalities to sort of compete and, uh, to get the Florentine,

• 1:58:17Samuel Hammond?

  you know, creativity and so forth. Uh, I think, you know, the next couple of decades could similarly be a renaissance for science and technology, uh, and for understanding the world. But, but it's partly a renaissance because we'll be moving into a much more competitive adversarial world where, you know, these city states and so forth will be, will be hard to coordinate. And, and so to the extent that there are still these like meta risks where we would value some, um, large scale intra and international.

• 1:58:49Samuel Hammond?

  Coordination like peace treaties and so forth. The disintegration of the United States where this revolution is occurring would be would be bad for that.

• 1:58:59Gus Docker?

  You talk about or you hint at an alternative path. What we've been talking about, your timeline here is the default path. You hint at a path where we have something you call constrained Leviathan. What is constrained Leviathan?

• 1:59:15Samuel Hammond?

  It's limited government, right? So this is, this is a Derron Nelson Mogul's word for it from the narrow corridor. And if you, if you trace the rise of sort of, of what we associate with liberal democracy, it is part of a particular technological equilibrium, in particular, an equilibrium that, that favored centralized governments with impersonal rule of law and impersonal tax administration and so on and so forth. So we associate today with libertarians with like being anti-government, but the, the basic idea of liberalism is actually associated with strong government, a strong impersonal government that can impose the rule of law.

• 1:59:55Samuel Hammond?

  And so if we want to maintain that kind of equilibrium in a world where AI is diffusing on the society level faster than it is on the, on the, on the state and elite level, then we want to accelerate the diffusion of AI within government. And there's obviously lots of low hanging fruit. We talked about how bureaucracies are basically fleshy APIs. You know, even today, you know, I have, I have a friend, a friend at the FTC, the Federal Trade Commission, they have like a 30 person team that, that is part of the healthcare division and they're, they're in charge of, you know, policing the entire pharmaceutical industry in the United States for competition. You know, his day job right now looks like manually reading through 40,000 emails that they subpoenaed from a pharmacy.

• 2:00:40Samuel Hammond?

  Right? And today, you could take that, those emails and put them into, you know, Claude 2 or something like it with a big context window and ask, you know, "Find me the five most egregious examples of misconduct." And it would do that. You know, it might not be perfect, but it's a hell of a lot more efficient than reading through them manually. And obviously, big law is going to be doing that. And the pharma CEO and his, his personal attorneys will be doing that conversely. You know, to maintain our state capacity in the face of, of AI is to, you know, run in this arms race. And, and you can kind of liken it to what in evolutionary biology they called the Red Queen dynamic, which comes from Alice in Wonderland, where the

• 2:01:22Samuel Hammond?

  Red Queen tells Alice that sometimes you need to run just to stay in place. And so, you know, I think our, our government needs to be adopting this technology as rapidly as possible, so that they can basically tread water. And that means both diffusing it in existing institutions, but also being open to, you know, radical reconfigurations of the machinery, machinery of government, and addressing some of those firmware level constraints that we talked about, whether it's the lack of a national identification system, or the outdated information technology infrastructure, or the accumulation of, of old procedural kinds of methods of governance.

• 2:02:00Gus Docker?

  A focused way of doing this is, is what you've, you've called for in a Politico article, which is a Manhattan Project for AI Safety. First question here, would it be better to call it an Apollo project as opposed to a Manhattan Project? I mean, the Manhattan Project created some pretty dangerous weapons, whereas the Apollo project might have been more benign.

• 2:02:22Samuel Hammond?

  I mean, what, what the Apollo Project and the Manhattan Project have in common is that they came from an era of US government where we still built things, where we still had competent state capacity, where we still had, you know, a lot of in-house expertise, and we weren't saddled with all these constraints. So we, you know, today, we couldn't go to the moon in 10 years. You know, NASA couldn't, you know, SpaceX can. And so, our modern Apollo projects are being done by the private sector through, through competitive contracts. And so, you know, one of the messages of my piece on the Manhattan Project is to say, the reason I make this analogy is not just because AI is a Oppenheimer-like technology, but also because responding to it will require a throwback to that, those kind of institutional forms where we gave

• 2:03:06Samuel Hammond?

  The people at the top, a lot of discretion and sort of gave them an outcome and let them solve for that outcome without having much of prescriptive rules about how to solve for that outcome. And then the second reason to make the analogy is, you know, OpenAI and Anthropic, they both have, you know, contingency plans for developing AGI and having like, you know, a runaway market power, right? You know, in the case of OpenAI, it's their nonprofit structure. In the case of Anthropic, it's their public benefit trust where they both are envisioning a world where they could potentially be the first to build AGI and become basically trillionaires. And so at that point, they need to become basically,

• 2:03:47Samuel Hammond?

  you know, governed by a nonprofit board. You know, at that, at that point, that's not where progress ends, obviously. Like, there's going to be continued research. It would make sense for the US government to step in and say, let's do this as a joint venture. We're not, we're no longer competing. In fact, the basic structures of capitalism and market competition are starting to break down. Let's just pull this together into a joint venture, study the things that require, you know, huge amounts of capital that the private sector doesn't have, but the government can. You know, the government, the government, US government spent $26 billion on the Manhattan Project in today's dollars. When you think about the financial resources of nation state actors to put behind scaling, you know, it's nothing like what Microsoft or Google have, you know, what, what, what's

• 2:04:32Samuel Hammond?

  our first $200 billion training run, right? What kind of things, what kind of things can come out of that? I think that's something that you want to do with the Defense Department's involvement and working with these companies in a joint way through, you know, secured data centers and doing kind of like gain-of-function style research that really is dangerous and, and more like, you know, more Manhattan Project than, than Apollo Project.

• 2:04:59Gus Docker?

  What would be the advantages here? We would be able to spend more, to slow down capabilities research and spend more of our, of the resources on, say, mechanistic interpretability or evaluations or alignment in, in general, because now the, the top AI corporations have kind of combined their efforts under one government roof.

• 2:05:22Samuel Hammond?

  Yeah, yeah. And, and, and my, in my vision, they're still allowed to pursue their Commercial verticals and I have an extended version of the proposal where I talk about, you know, needing sort of biosafety style categories for high risk, medium risk and low risk styles of AI that very closely parallels what Anthropic recently put out with their recommendations for sort of a BSL categorization of AI research. So I'm really talking about that like BSL four lab and beyond style stuff and some of that stuff, some of it will be to accelerate alignment and to ability research to sort of do versions of like the OpenAI

• 2:06:02Samuel Hammond?

  super alignment project where they're dedicating 20% of their compute to study alignment. Another part of it will be to forestall competitive race to the bottom dynamics. So, so that they can coordinate and not, you know, violate antitrust laws. And then the, the third thing is sort of the, the gain of function stuff that we really only want to be doing with like very, very strict oversight compartmentalization, kind of pooling of talent and, and resources so we can, you know, share knowledge on alignment and safety. But then also because government has this huge spending power relative to the private sector, you know, anytime you build a supercomputer, you're basically borrowing from the future. You're trying to see what

• 2:06:47Samuel Hammond?

  like the smartphones 20 years from now are, will be capable, capable of. And so if we want to sort of get ahead of the curve and see where scaling is leading, then I think governments are really the only actor that can waste a bunch of money basically scaling up a system and seeing, seeing what comes out of it.

• 2:07:06Gus Docker?

  Yeah. When we talk about gain of function research in, in AI, it's, it's an analogy to, to the gain of function research that's done on viruses in, in bio labs, but done for AI models. And, and this could, this could be experimenting with creating more agent-like models or inducing deception in a model and planting it in a simulated environment, see what, seeing what it does, or, or, you know, enticing it to acquire more resources. But again, perhaps in a, in a safely, if, if this is even possible, in a, in a safely constrained simulated environment. And, and this is the type of research that we could do in, in this Manhattan project, this, this government lab, because we would have excellent cybersecurity

• 2:07:51Gus Docker?

  and secure data centers and the combined efforts of. Of the most capable people in AI research.

• 2:07:58Samuel Hammond?

  If you've watched Oppenheimer, the movie, you know, a lot of that revolved around suspicions of coming as spies and so on. And, you know, we really don't have great insight into the operational security of the major AI labs. And that's something that, you know, bringing in-house of the Defense Department, they would necessarily have to, you know, disclose everything they're doing, but also hopefully beef up their operational security.

• 2:08:26Gus Docker?

  Yeah, they're kind of stuck with a startup mindset, but they're not developing a startup product. They're developing something that, in my opinion, it could be more dangerous than the average startup.

• 2:08:39Samuel Hammond?

  Yeah, and Dari Amade has said as much, that we should just assume that there are Chinese spies at all the major AI companies and at Microsoft and Google.

• 2:08:47Gus Docker?

  When we think about gain-of-function research in AI, how do you think about the value of gaining information about what the models can do and what the models can't do versus the risk we're running? It would be a tragic and ironic death for humanity if we experimented with dangerous AI models to see whether they would destroy us and then we hadn't constrained them properly and they actually destroyed us. So how do you, how do you think of that trade-off between gaining information and avoiding lab leaks?

• 2:09:23Samuel Hammond?

  Yeah, hopefully lab leaks are less likely than in the biology context where, you know, getting a little bit of blood or urine on your, on your shoes as you walk at the door. You know, it's a difficult thing to talk about in part because we just went through a pandemic that very probably was caused by a BSL4 lab leak. And so, you know, one saving grace is that AI models don't get caught in your respiratory system.

• 2:09:54Samuel Hammond?

  And so hopefully, there's forms of compartmentalization that are much, much more robust than in the biology context. And to the extent that this research is going to be done anyway, you know, it would be much better to move it offsite and hopefully in a way that, you know, facilities are air gapped and so forth, rather than, you know, what, you know, Microsoft is doing right now. They, Microsoft just recently announced their Autogen AI, which are sort of agent-based models, very similar to like AutogPT, but like, that work?

• 2:10:26Samuel Hammond?

  And they're doing this through Creative Commons, totally open source framework. All this, all these capabilities work as gain-of-function research, where we draw the line between doing things that are intentionally dangerous, or doing things that are dangerous, but we're, we're kind of pretending that they're not, is hard. I do think there's, and Paul Cristiano's also agreed with this, sort of threat models that would be valuable to be running in virtual machines, to see, you know, if the AI comes, develops awareness, situational awareness, and tries to escape, if it escapes into a simulated world that we built for it.

• 2:11:01Gus Docker?

  Okay, let's end by talking about a recent critique of expecting AGI to arrive pretty, in a short time. This revolves around interest rates. And I guess the, the basic argument is, or the basic question is, if AGI is imminent, why are real interest rates low? I can explain it, but you're the economist, so maybe you can explain the reasoning here?

• 2:11:28Samuel Hammond?

  So it's, it's really a question of how efficient are markets and how much foresight do markets have? You know, we're coming out of a world of very low interest rates, of ultra-low interest rates, near-zero interest rates. And what we, when we think about that is there's a surplus of savings relative to investment. And so one of the reasons interest rates have been in secular decline is because populations are aging, and so old people have a huge amount of savings built up. And meanwhile, we're going through this sort of technological stagnation. So the amount of savings relative to the amount of profitable investments was out of whack, and so that pushes interest rates down. In a world where AI takes off, It's a world where we have enormous investment opportunities, where we'll be

• 2:12:10Samuel Hammond?

  building data centers left and right and can't do it fast enough, where there's new products, new commercial opportunities left and right. And so you would expect in that world where the singularity is near, so to speak, to be one where the markets begin forecasting rapidly rising interest rates because the savings to investment balance is starting to shift. And in addition, there's a long run stylized fact that the interest rates, real interest rates, track growth rates. And so if GDP growth, GDP growth takes off, you'd also expect at least nominal rates to also take off. And so some have argued that looking at current interest rate data, like the, you know, the five-year, 10-year, 30-year treasury bonds, that the markets are not predicting AGI.

• 2:12:55Samuel Hammond?

  You know, the two responses to that are, one, first of all, interest rates are up quite a bit. Nothing's monocausal, there's lots of confounding factors. Is this, to some extent, the markets anticipating an investment boom? You know, maybe they're not anticipating full AGI, but they're seeing the way LLMs are going to impact, you know, enterprise and sort of baking some of that in. And then the second piece would be, okay, to the extent that they're not pricing in AGI, how much foresight do markets have anyway?

• 2:13:27Gus Docker?

  Before we discuss market efficiency, I just want to just give a couple of intuitions here. If AGI was imminent and it was unalliance, say, and it would destroy the world in five years, well, then it doesn't make a lot of sense to save money. Similarly, if AGI is about to explode growth rates, well, then a lot of money will be available in the future. You're about to become very rich, so it doesn't make sense to save a lot now. And the pool of available savings determine what's available for lending, which determines interest rates.

• 2:14:06Gus Docker?

  But let's, let's discuss, yeah, whether markets then are efficient on this issue or to what extent they're efficient.

• 2:14:14Samuel Hammond?

  Right, so this is the efficient market hypothesis, which, which comes in strong and weak forms. So the strong form of the efficient market hypothesis would say that markets aggregate all of available information and are our best sort of point estimate of anything we care about. The weaker form, which I think is more defensible, is that markets can be wrong, but they can be wrong longer than you can be solvent, right? And so you can try to short a company that, like Herbalife, famously, there was a big short position on that because Herbalife sort of looks like it's a multi-level marketing Ponzi scheme, but yet the hedge fund that did that lost several billions of dollars before they ended their position because the markets stayed irrational longer

• 2:14:58Samuel Hammond?

  than they could stay solvent. The second factor is the weaker versions of the efficient market hypothesis are sort of based on a no arbitrage condition, right? They say markets are efficient only insofar as you can arbitrage an inefficiency, right? And so you look at some prediction markets, for example, they predict it. They'll often have very clear inconsistencies across markets that look like they're irrational. But then you realize, oh, I can only make like $7,000 total on the website and there are transaction fees and there's work involved.

• 2:15:39Samuel Hammond?

  And so if the market isn't very deep or liquid, there may be inefficiencies that exist, not because the market's inefficient, but it's as efficient as it can be under the circumstances. And when it comes to AI, you know, how do you arbitrage? I've been thinking for a while now that Shutterstock, their market cap should be collapsing, right? Because we have image generation that is proliferating. And yes, people will make the argument that Shutterstock has all this image data, they could build a better image model. You know, maybe it seems like it's cannibalizing their business, it's sort of turning a moat into a commodity. And yet Shutterstock's market cap has basically held constant throughout this recent rebirth

• 2:16:22Samuel Hammond?

  of image, birth of image generation models.

• 2:16:25Gus Docker?

  What if you borrow a lot of money cheaply and then put it into an index of semiconductor stocks or just IT companies in general, even just the general S and P 500, say? Would that be a way of arbitraging this AGI forecast?

• 2:16:41Samuel Hammond?

  Yeah, I would say if you have short timelines, you should be putting a lot of money into equities.

• 2:16:47Gus Docker?

  This is not financial advice, would you say?

• 2:16:50Samuel Hammond?

  Right, and that's, and I mentioned earlier that Paul Cristiano has said in interviews that he's twice levered into the stock market. He basically owns a bunch of AI-exposed companies and he's borrowed enough money to double his investments. So that's putting your money where your mouth is. When you look at market behavior over the long stretch of time, markets didn't anticipate the internet very well. There was a short-run bubble that led to a boom and bust of dot-com stocks. But in terms of the real economy, the internet just kept chugging along and kept being built out. And eventually, a lot of those investments ended up paying off, even if you rode through the bubble. Markets are made of people.

• 2:17:34Samuel Hammond?

  Some of the biggest capital holders in the markets are institutional investors, pension funds, life insurance companies, governments, like Saudi Arabia or the Norwegian Pension Fund. And often, these are making safe bets. They're not taking very heterodox views on markets. And so as a result, markets can be a little bit autoregressive, they're a little bit biased to the past, you know, past is prologue, and prone to kind of multiple equilibrium. Where there's two prices that the shutter stock can be. The shutter stock could be a $50 stock or it could be a $0 stock.

• 2:18:18Samuel Hammond?

  And at some point, the market will update and will undergo through the great repricing and all these asset prices will flip in relatively short order. The efficient market hypothesis has to be false, or else we wouldn't have Silicon Valley. Right? We wouldn't have founders, we wouldn't have Elon Musk. Right? So I would just say the markets are wrong. And partly, they're wrong because to be right would require having a bunch of relatively...

• 2:18:45Samuel Hammond?

  Bespoke and kind of esoteric priors about the direction of the technology that are only now just sort of percolating into the mainstream.

• 2:18:53Gus Docker?

  Yeah, and that the big capital allocators can't really respond to because they're risk averse.

• 2:18:59Samuel Hammond?

  Exactly. No, that doesn't mean Renaissance technologies won't respond to it, but they're not going to move the market.

• 2:19:05Gus Docker?

  Samuel, thanks for this conversation and I've learned a lot.

• 2:19:08Samuel Hammond?

  Thank you.

• 2:19:09Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.