Success without Dignity? Nathan finds Hope Amidst Chaos, from The Intelligence Horizon Podcast

This special cross-post from The Intelligence Horizon features Nathan Labenz in a wide-ranging conversation on compressed AI timelines, expert disagreement, and why he believes the singularity is near. They discuss interpretability, RL scaling, and the balance between extraordinary upside, like curing major diseases, and serious existential risks. Nathan explains his evolving p(doom), why he’s slightly more optimistic about robustly good AI, and how defense-in-depth strategies might keep society on track. The episode also explores US-China rivalry, AI governance, and why human cooperation may matter more than technical control alone. Google: Keep up with AI research on the go with NotebookLM, Google's steerable research and thinking partner. Try it at https://notebooklm.google.com/. Sponsors: Tasklet: Build your own Cognitive Revolution monitoring agent in one click. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai VCX: VCX, by Fundrise, is the public ticker for private tech, giving everyday investors access to high-growth private companies in AI, space, defense tech, and more. Learn how to invest at https://getvcx.com Claude: Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (03:27) Special Sponsor (05:12) Opening and AGI framing (12:08) Scaling RL and paradigms (Part 1) (21:31) Sponsors: Tasklet | VCX (24:24) Scaling RL and paradigms (Part 2) (28:56) Verifiability and long horizons (41:13) LLMs and world models (Part 1) (41:19) Sponsor: Claude (43:32) LLMs and world models (Part 2) (54:17) Energy, hardware, and chips (01:00:42) Alignment risks and bottlenecks (01:10:18) AI values and agency (01:20:31) Defense in depth alignment (01:30:48) US-China AI cooperation (01:41:05) Episode Outro (01:45:42) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm sharing a special crosspost from my recent appearance on the Intelligence Horizon podcast with hosts Owen Jeong and Will Sanoc to follow. Owen and Will will soon be graduating from Yale College. And as you'll hear, they've clearly spent much of their senior year thinking deeply about the current state of AI, where we're headed, and what it means for all of us. And I was really impressed, not only with the quality of their questions, but their ability to challenge me with follow-ups that effectively steelman the most relevant counterarguments. We start with the fact that while AI timelines have compressed dramatically over the last five years, genuine experts still disagree radically on critical questions.

• 0:44Nathan Labenz?

  Having established what I hope is appropriate epistemic humility, I then go on to call it how I see it. In short, the singularity is near. Interpretability science proves that AIs are developing increasingly sophisticated world models. And with reinforcement learning scaling now clearly working, AIs are no longer simply imitating humans and likely won't be limited by what we know for much longer. The potential upside of this is, of course, incredible. The value that I've got from using AI to navigate what humans have discovered about how cancer works and how to treat it has been invaluable. And the prospect that we might cure the majority of human diseases in just the next decade or so is obviously extremely exciting.

• 1:26Nathan Labenz?

  That said, the risks are also very real, and they will remain serious for as long as we lack a solid understanding of how AIs work and why they do what they do. My P doom remains somewhere in the 10% to 90% range. And yet, at the same time, I've become at least a little bit more optimistic that we might actually build robustly good AIs. Because scaling laws at least seem to imply that powerful AIs can only be created with massive resources. The three companies competing at the frontier today are at least reasonably responsible actors, and our best alignment techniques are working better than I had expected.

• 2:04Nathan Labenz?

  Given these fundamentals, it seems at least plausible that a defense-in-depth strategy, which combines techniques like Goodfire's intentional design, Redwood's AI control, improved cybersecurity through formal verification of software, and various forms of pandemic preparedness, could collectively be enough to keep society on the rails. We touch on a number of other topics as well, including the US-China rivalry, and why, especially in the context of the Department of War's recent attack on Anthropic, which, I'm sad to say, has us looking more and more like China all the time. I would rather bet on figuring out a way to cooperate with our fellow humans than bet everything on AI researchers' ability to steer AI advances in a way that will ultimately work for us humans.

• 2:47Nathan Labenz?

  I appreciate Owen and Will for allowing me to cross-post this conversation, and I definitely encourage you to subscribe to The Intelligence Horizon. Their recent conversation with former OpenAI researcher Zoe Hitzig covered the evolving ways that people are using ChatGPT, variations on universal basic income, AI governance models that emphasize a decision-making process over specific principles, and why she believes that these kinds of structures will probably have to come from outside the frontier companies. For now, I hope you enjoy my conversation with Owen Zhang and Will Sanak do follow from The Intelligence Horizon.

• 3:28Nathan Labenz?

  The Cognitive Revolution is brought to you in part by Google, makers of the Gemini family of models and much more. One of my big AI goals for 2026 is to find ways to spend less time at my desk and more time exercising and outside. The challenge is that I'm genuinely so obsessed with keeping up with everything that's happening in AI that it's hard to pull myself away from the screen. Google Notebook LM gives me the best of both worlds. By creating podcast-style deep dives about whatever I'm curious about on any given day, it helps me keep learning, even on the go. If you haven't tried Notebook LM for a while, you should know that it's become a much more steerable research and thinking partner.

• 4:09Nathan Labenz?

  These days, you can select short, medium, or long for audio length, and you also get a free text field to steer the direction of the conversation. For AI research, I always ask for rigorous, literal, technical explanations with no analogies.

• 4:25Nathan Labenz?

  Recently, I used Notebook LM to study a paper by Google DeepMind researcher Rohan Shaw, which attempts to set upper bounds on how much reasoning different kinds of models can do without needing to externalize their thinking in a chain of thought. The hope is that by establishing these limits, we can better calibrate how much confidence we should have in chain of thought monitoring. And standard transformers, in fact, are much more limited than alternatives like state space models, a notable upside to transformers that we shouldn't take for granted. If you're trying to keep up with the pace of AI research while staying physically fit, or you're just a natural audio learner, give Notebook LM a try at notebooklm.google.com. Thank you to Google for supporting the cognitive revolution. And now, on with the show.

• 5:10Nathan Labenz?

  When it comes down to the kind of core question I think a lot of people are getting at is like, is this AI thing gonna kind of fizzle out before it really becomes a big deal? Or is it gonna be a huge world-altering deal? I'm very much confidently, clearly in the camp of it's gonna be a huge, huge deal. And the details are where I think the discussion or the debate remains now, not for me at least, whether or not the overall trajectory of AI is gonna take us to something that is powerful enough to be transformative. One of the strangest things in the world today, full stop, is the fact that the disagreement among very plugged in, very informed, very smart people has not really been reduced

• 5:54Nathan Labenz?

  much at all, even as we've gained a ton of information over the last couple of years about the trajectory of AI. I think that is super strange, and I'm honestly pretty confused by it. The one thing everybody seems to agree on is like, the timeline on which we should expect this has come in. In today's world, if you say you don't think you're gonna see AI until 2035, you're like an AI bear. But only five years ago, that was considered to be quite aggressive. And most people were more like, I don't know, 2050, maybe not in my lifetime. So there's been this massive compression of timeline, there's obviously been this huge jump in capability, and yet, on these fundamental questions of like, what's gonna happen? There's still total disagreement.

• 6:42Speaker 01

  Today is Nathan Leventz, host of the Cognitive Revolution Podcast. Before switching to full-time podcasting, Nathan founded Waymark, an automated marketing platform for local businesses that pioneered the use of generative AI to produce video ad campaigns. After leading the company for six years, he stepped back to focus full-time on understanding and communicating the trajectory of AI. As host of the Cognitive Revolution, he has conducted hundreds of in-depth interview views with AI researchers, founders, policymakers, and investors, and has become a go-to source for people trying to keep up with what's happening at the frontier, including us. Nathan was also a member of OpenAI's Red Team, where he was among the first outside users

• 7:26Speaker 01

  to interact with GPT-4 before its public release. Welcome, Nathan. Great to have you on today.

• 7:32Nathan Labenz?

  Thanks for having me. I'm excited for this conversation, guys.

• 7:35Speaker 01

  So first question, are we on the cusp of AGI?

• 7:39Nathan Labenz?

  Starting with the easy ones.

• 7:43Nathan Labenz?

  Well, okay, first of all, as I'm sure you guys are very well aware, what exactly do we mean by AGI is a slippery question, at least, that has people talking past each other quite a bit. So I don't think there's any, you know, like, super privileged definition. What I do think we're pretty clearly on the cusp of is powerful AI that is better than the vast majority of people, although perhaps not the, you know, very few top, top experts in a given domain at pretty much all cognitive work. That seems like pretty clearly on the horizon, and I think it is absolutely going to be enough to be transformative to the economy, to daily life, potentially even, you know,

• 8:28Nathan Labenz?

  bigger things than that, like the very nature and status of the human species. And I think that we will have that kind of regardless of whether or not there are some niche areas where humans retain an advantage, which my guess is probably will be the case. Certainly, one of the big things that has become clearer over time as AI systems have gotten better is that they are jagged in this weird way, where they have certain things they just do amazingly well at, other things they kind of weirdly struggle at. They're not very adversarially robust. For example, you know, they're Easier than humans, I would say, still to trick. So, you know, there's, there's going to be weirdness. And I think, you know, throughout the conversation today, probably a big theme will be to expect weird things to happen.

• 9:10Nathan Labenz?

  But, you know, when it comes down to the kind of core question, I think a lot of people are getting at is like, is this AI thing going to kind of fizzle out before it really becomes a big deal? Or is it going to be a huge world altering deal? I'm very much confidently, clearly in the camp of it's going to be a huge, huge deal. And the details are, you know, where I think the discussion or the debate remains now, not for me at least, whether or not the, the overall trajectory of AI is going to take us to something that is powerful enough to be transformative.

• 9:42Speaker 02

  Okay. So it sounds like what you're saying is that there's no doubt that we're going to get massively powerful and transformative AI across domains. Maybe like there'll be some like, you know, a tiny little bit where human experts still have an advantage in specific domains. Maybe there'll also be like comparative advantage. And we'll get more in depth on like what specifically those concerns are. But I guess just to hone in, are these more so uncertainties as to whether AI can fully generalize? Or are you like pretty confident that for like the foreseeable future, there will be these gaps between like human experts in specific domains and like problems like adversary or robustness? So, so that just to clarify what your, your position, is this like a prediction or is there like this is where your uncertainty lies? Your uncertainty doesn't lie in AI being transformative?

• 10:25Nathan Labenz?

  Yeah, I think the latter. I wouldn't be shocked if there are additional unlocks that allow AI systems to truly undeniably surpass what humans are capable of. I think you look at the natural world and clearly humans did that to every other animal species that existed before us, right? So there is a historical precedent for some new kind of mind showing up on the scene and blowing away all the other minds that came before us. I don't think we have any reason to believe that that couldn't happen to us in some, you know, law of physics guarantee sort of way. So I definitely think that's possible. But in terms of what I could confidently foresee, I don't think it is clear that that's

• 11:09Nathan Labenz?

  going to happen in the next few years. But what I do think is still, again, clear is that we are going to have systems that are powerful enough to be transformative across almost all the questions that we care about in terms of what is society going to look like? How are we going to organize? Are we going to need a new social contract? Like, all those things seem to me pretty clear. And then, you know, people quibble often around the edges of, could, I mean, you get into, there's some pretty esoteric jobs out there, right? So like, could, will an AI system ever be able to be as good of a sommelier as the best human sommeliers? Well, I don't know, you know? Is anybody going to be motivated to train one to try to do that? You know, we don't really have a lot of taste AI at this point in time. And, you know, maybe tasting will be the, forever the domain of humans.

• 11:53Nathan Labenz?

  But trying to identify these little niches where we may have some really longstanding durable advantage, I think too often distracts from the big question of like, is this going to change just about everything that matters to us? And there I clearly come down that the answer will be yes.

• 12:08Speaker 01

  Right. So let's talk a little bit about how maybe we get to this transformative or extremely powerful AI. Do you think current paradigms? I think the one that comes to mind the most and which is discussed the most is scaling RL is going to be sufficient to get to that transformative or powerful AI that we discussed.

• 12:25Nathan Labenz?

  Yes, I mean, I think it probably is. And I also think if it's not, we may never quite answer that question in the sense that I do expect that we will continue to see new conceptual advances in AI research. You know, the field is growing. Everything, one of my sayings is everything is going exponential. So that is the number of people that are working in AI research. It's the number of papers. It's the number of experiments being run. It's the amount of compute that people have to run those experiments on. It's the data sets that have been collected. It's the RL environments that are being built out over time. And I do think all those things are going to probably give us some new conceptual unlocks such that we will never probably answer the question, and this already kind of happened with pre-training, right?

• 13:09Nathan Labenz?

  You rewind two to three years. There was a time when people were like, well, we're kind of running out of data. You know, can we really scale this all the way to AGI? And nobody's talking now about can we scale pre-training all the way to AGI anymore because there's been a new thing. And so now the new thing is on top. And it's like, well, clearly that's going to be part of the mix. Will this, you know, exact pattern or this, this sort of shape be enough to get all the way to AGI? My prediction is that. Again, what does AGI mean? But my prediction is it probably is enough to get us to systems that are transformative. But if we fast forward to 2028 and, you know, look back at this conversation, we'll probably say, well, nobody's really asking that anymore because we do have a couple of new things that have come online.

• 13:54Nathan Labenz?

  And so now we have a little richer sense of like, what the shape of it's going to be. And, you know, we're either getting there or we're not. We maybe still are missing a little something at that point. But I would guess that we would look back at the current thing and say, yeah, clearly a couple of things have been added and they were a big deal. And so it was, you know, the question of like, is exactly what we have in February 2026 enough? Kind of ends up being beside the point in the final analysis.

• 14:18Speaker 01

  Interesting. So what you're trying to say here is that in the same way that there were questions about whether pre-training would take us to this like extremely transformative AI. Now that we have these new unlocks such as RL, for example, like we discussed, who knows in maybe two to three years or, you know, whatever time scales that we're talking about to reach transformative AI. There might be other conceptual progressions that are also made that build upon things like pre-training RL that will get there. So I guess my question specifically was, do you think there will be more of these conceptual progressions that need to be made to get to this transformative AI that we're talking about? Or do you think the RL paradigm is going to get us there?

• 14:54Nathan Labenz?

  I think RL probably would be enough to get to AI systems that can like do most of the cognitive work in the economy, for example. I mean, it seems like honestly, we're already reasonably close to that and it doesn't seem like we're anywhere close to done. And by the way, if you listen to the lab leaders, the frontier model developer leaders, they are still saying, by the way, too, that pre-training is still working. You know, it was never really the case that pre-training stopped working. As far as I know, those scaling laws basically have held. I think what happened at one point in time was the next step on the pre-training frontier was becoming really expensive. And then they found another kind of, maybe not fully orthogonal, but you know, pretty different direction to go

• 15:39Nathan Labenz?

  with scaling post-training with the RL paradigm that we have now. And that was just much bigger bang for their buck, at least at that moment in time, right? Like they had already gone pretty far up the pre-training curve. They hadn't gone very far up at all, the RL curve. Now presumably those things were gonna kind of even out, you know, a general kind of economic theory would be like, people should be investing in one until the marginal return decreases to the level of the other, and then they could maybe invest in both. So if you find some new path that's like, oh, this is really giving us huge ROI, you go hard at that path for a while. But then that kind of hits some diminishing returns. Now you're kind of back to, okay, well, maybe we need to do all these things more at the same time. So we'll do a little more pre-training, we'll do a little more RL, and we'll do, you

• 16:24Nathan Labenz?

  know, maybe more of that mystery third thing, all advancing in tandem. Right now, I think we're advancing, I don't have insider information on, you know, exactly what these ratios look like, but I'd say it's probably roughly the case that we're close to, if not at the point where additional compute going into RL is roughly giving the same kind of returns as additional compute going into pre-training. And so both are gonna be, you know, places where frontier companies can invest for the time being. You know, and I guess another way to think about this is like, people also ask like about generalization in RL. And I think again, it's kind of worth unpacking, like, what does it mean to generalize? Like, what are we talking about generalizing? One way to think about the question is, if we

• 17:09Nathan Labenz?

  do a bunch of RL on a model, does the model generalize to all sorts of new things? And there I would say, again, there's probably another way to break that down, which is like, there are domain-specific skills, and then there are more kind of cognitive or even like meta-cognitive skills that work across domains. So one of the biggest, like, I guess we'll call it the aha moment for me and for DeepSeek researchers and for the R1 model that they were training, was from the R1 paper, this is January 2025. They reported this, what they call the aha moment, which is, in their process of doing RL on, you know, an already pretty capable base model, of course, right? They found that these

• 17:55Nathan Labenz?

  Previously unobserved, higher order cognitive behaviors started to come online. The aha moment in particular was in the reasoning trace, the R1 model gets to a point and it says, oh, wait, this is an aha moment. Like, I can come at this from a totally different direction. And this is something that hadn't been observed much. I mean, it's out there in the pre-training data. There's at least some examples, right, of people kind of documenting their own chain of thought and getting to these aha moments and realizing, oh, I was coming at it the wrong way. Now I can come at it this other way. But that hadn't been observed too much in AI systems. Reinforcement learning seems to be bringing, you know, clearly has like brought that sort of thing out. And now we have these long reasoning traces where the model will kind of come at the same problem from a bunch of different

• 18:39Nathan Labenz?

  directions. And so what generalizes, what doesn't generalize, I think you probably can't take a model that has never been trained on a particular domain and expect it to go into that domain and be successful. But you can't expect some of these like meta traits to generalize from one area to another. And then if you zoom out the farthest and just say like, does RL as a process that companies or, you know, organizations apply, does that generalize? And there, I think the answer is like, definitely yes. It's just a question of getting the reward signal dialed in to the point where it actually works. And that's definitely easier in some areas than in others. And so we do see like in areas where

• 19:20Nathan Labenz?

  it's easy, like math and programming, we see fast progress there relative to things where it's harder to get a clear reward signal. But still, I think we are quite obviously making that work. An episode of the podcast that's coming out, I think today as we're talking is with the head of health at OpenAI. And, you know, I can tell you from personal experience, latest models are absolutely on the level of attending physicians. My son unfortunately has had cancer over the last few months. He seems to be very much on track to be cured and be all better, which is fantastic. And I probably wouldn't be here talking to you if that wasn't the case.

• 20:01Nathan Labenz?

  But I've had occasion to really use the latest models intensively in a medical context. And they're absolutely on the level of the attending physicians. They know a lot more than the residents and they really are step for step with the most senior doctors at the hospital. How is that happening? Well, they've worked with 250 plus human doctors closely at OpenAI to create training data, to grade, et cetera, et cetera, et cetera. But now their latest models are also outperforming their human doctors when it comes to the task of evaluating AI outputs. So there are these kind of thresholds that they're crossing where it's like, you know, it was really hard. I sometimes think of this as like spinning a big wheel, like your first pushes on this big wheel, don't move it much, right?

• 20:46Nathan Labenz?

  But as you build up this momentum, as the flywheel really starts to turn, you start to hit these thresholds where it's like, well, now we have a model that is outperforming our humans. Think of all the work that they had to put in thousands and thousands of hours and millions and millions, hundreds of millions of dollars potentially to hire hundreds of doctors to do all this work. Now they've got a model that is beating the doctors at evaluating outputs in the medical domain. And so that totally changes the game. Those thresholds will be crossed at different times for different domains, but I think it's like safe to say that in most domains, if there's any sort of objective ground truth or even like a high level of agreement among professionals, you can get there. It just takes more time.

• 21:26Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:30Speaker 02

  Okay. So just to like go back a little bit, it sounds like, okay, a lot of your confidence that we'll have transformative AI very soon, it hinges, I'm not sure to what extent it hinges on like your confidence that we'll find some other paradigm if a new paradigm is required on top of pre-training RL. You also sound very bullish on RL, but maybe to steel man the other view that's like, okay, if RL doesn't work, then we actually might not find another paradigm. Is that like, okay, so RL is a very general machine learning principle, right? It's just that like you reward the Model for doing the correct thing on a task that you can like define a reward signal for, which is like kind of like a maximally general machine learning principle. And it's also something that existed for like a long time.

• 22:15Speaker 02

  It wasn't something that was like just recently discovered after we got LLMs. It was like a major, like one of the foundational sort of paradigms in machine learning even prior to LLMs. And so I guess the skepticism is like, maybe there's like, it's actually not that easy to find some other paradigm. Like maybe, you know, pre-training was like this, the pre-training LLMs on text data was this novel thing. And then we just like applied this thing that had been in machine learning for a while, namely RL. But then like the next leap is like, there's no precedent for it. We don't see any like big promising thing. What do you think about that skepticism?

• 22:45Nathan Labenz?

  I think it's probably not going to play out that way. And if it did, I still don't really think it matters that much. It probably shifts the timeline a little bit.

• 22:57Nathan Labenz?

  But the RL will continue to scale. The amount of compute coming online is, again, that's exponential, right? So there's just tremendous amount of additional resources to be thrown into this. One thing we're doing a bit, but not a ton is just like using the signal from the world. You know, Elon's got a plan with XAI to, and I think he's got a real advantage here to just have the AIs solve the same super hard problems that the engineers at Tesla and SpaceX and Neuralink are solving. And so he's just going to give them a computer, say, here's all the professional software. Here's the problem. You got to solve it, right? There's like a never ending supply of those problems.

• 23:38Nathan Labenz?

  And I wouldn't be surprised to see Grok get to like Tesla, SpaceX, Neuralink engineer level just based on the fact that those problems are there to be solved. And, you know, they've got the compute to keep trying. And what's really the fundamental barrier there? I don't really see it. I honestly think that the, the new paradigm is probably more about usability. Then it is about capability. By which I mean, like, people have a lot of complaints when they use models because they're like, "Oh, it didn't really do what I wanted it to do," or "I sort of, you know, didn't really understand me," or what have you. And clearly, one thing that they're not great at doing is going into a new environment, kind of scoping the situation out, getting the vibe, you know, getting,

• 24:22Nathan Labenz?

  picking up that subtle feedback that people give each other, and gradually kind of figuring it out and like becoming a useful contributor. That's like how a lot of people go from day one to like effective employee in their jobs. And AIs don't really do that in the same way, obviously, right? They don't have continual learning. They are able to manage their own memory somewhat, but they're not that awesome at that. That's another one of these like cognitive skills that does generalize, though. Like, once you're good at managing your memory, you'll be able to apply that in new domains. But they're not that great at that yet. And I think that prevents a lot of people from getting value. More often, I think that, not that the model can't fundamentally do the thing, but that it doesn't have the context. The person trying to get it to do the thing doesn't know how to assemble the context,

• 25:06Nathan Labenz?

  or doesn't believe that it could do the thing such that they're willing to invest the time and energy to give it the context. And so that unlock might just be like, now you don't have to do that anymore. It will kind of figure it out on its own. And this is maybe a way to expose, in a way, similar to instruction following, right? Like, in the, if you were really good at prompting GPT-3, you could get it to do a lot of things. But it was a weird art to prompt GPT-3. And to a much lesser degree, but still somewhat, you know, getting value from the current models is still kind of a weird art. You kind of have an intuition for them. You kind of need to know how to assemble context and give effective instructions. And people aren't that great at that.

• 25:47Nathan Labenz?

  So that next version might not even be so much about allowing them to do qualitatively new things, but just making the barrier to use much lower so that people can just be like, "Hey, AI coworker, welcome to Slack," you know? And then over a week or whatever, they just kind of ramp up and gradually get it.

• 26:04Speaker 01

  Yeah, interesting. I think the pushback there is the verifiability problem becomes a little more apparent in these specific long horizon, excuse me, situations. I think when we talk about verifiability right now, we clearly see that with model capabilities, like you said with Claude code, that they work great for math, they work great for code, and they work great in these settings where you can clearly check the answers and whether or not they're right or wrong, right? That's how you kind of create this reward signal. But when we talk about these domains where there's no clear verifier, for example, when it comes to writing a good essay, what does that mean? Give good therapy in the healthcare settings.

• 26:43Speaker 01

  And then we emphasize that even further by placing it in a longer horizon context, such as the one that you're talking about with, instead of having a human in the loop at each step prompting it in a specific direction, giving it a broader task and having it go in a direction for a longer time horizon. It gets even harder to solve this verifiability problem, right? Like, how do you make sure that you get high-quality signals? How do you make sure to tune these systems in a way where they can sustain autonomous work for days or weeks at a time? And so I wonder if you have any thoughts in terms of how we kind of go in that direction

• 27:26Speaker 01

  and solve this seemingly very large verifiability problem that hangs over our head when we talk about areas that have long time horizons?

• 27:36Nathan Labenz?

  Well, again, I'll take one beat to say I think the recent trajectory shows that this problem is being solved. The, I'm sure you are very familiar with the meter graph, which is everywhere these days. It's basically going vertical at this point. It's to the point where the meter people are like, we are really struggling to have tasks long enough to even be able to evaluate these things on. And, you know, it's also kind of worth noting, too, that that's a bit of a challenge with humans, right? Like, we hire people on much less than, you know, a week's worth of work, and it's not like always super easy for people to agree, like, did somebody do a good job on that month-long project or not? You know, you can usually tell if they, like, totally crushed it or, like, totally sucked. There's often a lot of disagreement in organizations about, like, well, you know, maybe it was

• 28:20Nathan Labenz?

  actually harder than we thought, or, you know, they didn't have, the ingredients for success weren't really there. I mean, there's often a lot of fuzziness in this stuff, even in human context. I mean, there's a bunch of techniques that I think are being used. One is Rubrik rewards. So Elon likes to talk about things like, does the rocket fly? The ultimate ground truth is if I can send this thing into space and I can come land it down on a pad, standing up on its tail again, then clearly that worked. And yet, the reward for that, that's an expensive experiment to run. You couldn't just launch a million rockets and have all of them, most of them crash to find the ones that worked.

• 29:04Nathan Labenz?

  So there is a challenge there in terms of sparse reward and the cost of the experiments. I think what is happening a lot is that people are defining rubrics of things that they want to make sure the AI does well. They're probably working with AIs to develop those rubrics. Again, in the OpenAI Health context, they created a benchmark called Health Bench, where there are 49,000 evaluation criteria. So all these different tasks, puzzles that the AIs have to figure out. And it's not just like, did you get it right? Or did you not get it right? But it's a painstaking effort to really flesh out all the different things that would matter, that would make for a complete, awesome, best-in-class answer.

• 29:48Nathan Labenz?

  And then the AIs are not scored zero and one. They're scored on some sort of scale that's like, you may have got zero out of 25 things that you could have got on this question. You may have got five, 10, 15. But that gives you enough of a signal that you can kind of climb that hill. And it really does seem to be working. I think that is going to work pretty well in domains where there is a professional consensus. Because I think that's how people evaluate each other, too. Multiple choice tests. But there are also like, you've got to kind of show in a medical context, you're a student, you go through these in-person training processes. They say, watch one, do one, teach one. And so as a medical student, you watch and then later you start to do.

• 30:31Nathan Labenz?

  And then you get a lot of feedback and eventually the next thing you know, you're the one teaching. I think the AIs will pick up those signals. And then there's other things where it's taste. And I think that'll be a little bit different probably. But if I had to guess... You know, what is a good novel? Well, first of all, there is no consensus on that, right? Even you can find somebody who hates even the, you know, most universally critically acclaimed novels, and you can find somebody who loves something that everybody else thinks is trash. I think what we'll see there is kind of taste-based communities coming together to shape models for their own tastes. So in other words, you might start with a base model, and you might want it to write romance novels, or you might want it to write anime, or you might want it to write hard sci-fi.

• 31:15Nathan Labenz?

  And what it means to be good in those different genres is like quite different, but what you do have is fans of those genres that can engage with outputs, give their scores, and shape models based on what they like.

• 31:29Speaker 02

  Okay.

• 31:30Nathan Labenz?

  And I think that can work.

• 31:31Speaker 02

  Yeah, so I just want to go back to the, even within task, but there is a consensus, right? Like, I see what you're saying, that like, you know, there's ways to build like minor consensuses within like fiction communities or something like that. But, so like, even the question of, does the rocket fly? I think there's an argument to be made that this is like categorically different than a chatbot putting, giving out like useful medical advice, right? Like, when a chatbot's just giving out useful medical advice, it's like, not very, it's not a long context task, it's like easily transcribed into text, right? It's like sort of what LLMs are like clearly going to be good at, they're like good at, they're LLMs, so they're like good, they're native to text. But like, something like, does the rocket fly?

• 32:12Speaker 02

  It's like, okay, even if there is, like, obviously a consensus on what it means to make a rocket fly, it seems like there's an argument to be made that like, the signals on how to perform that task are very noisy, and like, you need a ton of them, right? Like, you need the like, you know, like, let's say you're like a project manager for the rocket flight team, right? Like, you need to understand what's feasible within engineering, which like, okay, maybe LLMs are better than that, at that, but then you also need to like, know how to manage humans and like, or not, maybe AI agents in this case, but like, you need to know how to like, manage people and to allocate tasks, and how to like, what is going to be like feasible with the amount of like, money you have, and how can you raise money from investors, and stuff like that? And so it's like,

• 32:54Speaker 02

  The longer time horizon, the more agentic a task, the more like little itty bitty signals are required on your day to day work to be able to complete the task. So I think, and going back to what you said about like this sort of flywheel thing, we're like, okay, like if we have like, we work really hard and we get a lot of medical experts to provide this high quality data, then we get LLMs that are like pretty good at giving medical advice and then all of a sudden they can also, they're actually better than human doctors at verifying the output of medical advice in the first place. Then we get this flywheel effect and we have, we have like a quick and robust verifier. But maybe like the argument is we just never get over this flywheel hump for long-term, highly agentic tasks precisely because it's like, it's not even a question of whether

• 33:38Speaker 02

  or not there's consensus on what constitutes a good outcome. It's like, can we define enough reward signals and like do enough rollouts to get, to get to this point?

• 33:47Nathan Labenz?

  One thing I find helpful sometimes is just to try to reflect on like what I do, what is my experience like? And you really try to interrogate it for like, is there something really magical going on or not? And I think, I have to say, really long horizon agency is pretty rare among humans, right? I mean, you do have these people who, like Elon, who keeps going up, say, you know, my goal is to make humanity a multi-planetary species and if it takes 30 years, you know, that so be it, right? So I would say, you know, skepticism around like, will we get AIs to be, you know, multi-decade

• 34:28Nathan Labenz?

  horizon planners that can sort of show up with the fierceness and determination required on a day in and day out basis to really make that happen, remains like pretty valid skepticism in my opinion. At the same time, you look at like what most people do and what drives the economy and it's like, Much shorter term than that. A lot of it is like, I kind of keep showing up one day to the next. And every day I kind of start fresh. You know, I went to sleep and I kind of shut down. You know, there's like, there's sort of a discontinuity of consciousness between workdays, which, you know, I don't want to, I'm not a big analogy guy, but if you squint at it, you can sort of make an analogy between like different context windows. And there's kind of a, you know, a somewhat similar resetting type of moment.

• 35:13Nathan Labenz?

  And what do I have to do? Like, I have to kind of reboot myself and be like, what was I doing yesterday? Where did I leave off? And what did I accomplish? What was still to be accomplished? And most of that stuff, you know, I could write down if I, if I had like a worse, if my integrated memory was worse, I could compensate for that pretty well. You know, at least on sort of a week long, or maybe a month long, or maybe a quarter long basis. If I got really good at the end of the day, at the end of each day of saying like, here's everything I did today. Here's what went well. Here's what failed. Here's what's next. Here's the feedback I got. And here's what I think I should do in the next session. And then wiped all that out and came back and looked at that scratch pad, you know, again, to start my day tomorrow. I think over time I could get like pretty effective at that.

• 35:56Nathan Labenz?

  So I don't, and I think this does connect back to also to the certain kinds of skills do generalize. And I do think this is one of them, right? Managing your own memory, writing notes for yourself to kind of document what happened and what's supposed to happen next. Giving yourself a sense of like where you are on the overall story that is unfolding. I think that, you know, they're not great at that yet, but they're getting decent. And, you know, when my Claude code crashes these days, a lot of times, you know, just kill that tab, resume in the next one. And then I'll just be like, resume, you know, or like, sorry, we got cut off. Like my internet went out, you know, please pick up where you left off. And they're able to do it. You know, again, they're not awesome yet, but they're way better than they were even three months ago.

• 36:40Nathan Labenz?

  So I think it is hard for me to see how that doesn't extend, especially because it is a relatively new thing, right? And, and most of these things that pop up, they have at least a few generations before they kind of level off. It's hard for me to see how we don't see a, again, a pretty steep, you know, meter curve is currently vertical. It can hardly get steeper than it is. But I think we've got at least a few generations of like very steep progress there. And then maybe there's some other kind of thing where it's like, okay, sure, you can do a quarter worth of work when somebody gives you a project. Can you like figure out what the next big civilizational advance could be? Like some of the true human visionaries? Maybe not. Maybe that's a different kind of thing. In a way, I kind of hope that we stop there.

• 37:24Nathan Labenz?

  Sometimes I, because I talk up a lot, what I think AI is capable of, will be capable of. It's easy to mistake me as like a booster. I'm actually kind of afraid of that. I think it would be great in some ways if we did find certain fundamental barriers where it's like, hey, I can delegate a month or a quarter's worth of work to this thing. And it might be able to do it in a day for a couple hundred dollars. Like, wow, amazing. If we could kind of park it there, you know, and not have it go to the point where it's doing multi-decade level planning, that might be a really good thing. That might be the sort of the sweet spot where we get a lot of the advances that we want and the better quality of life and, you know, all the abundance that people dream about. And we don't have so much risk of losing control.

• 38:09Nathan Labenz?

  So I don't, you know, I don't think every advance, by the way, is a, is a good thing. But I just don't see fundamental barriers on the horizon, at least.

• 38:21Speaker 01

  Okay, great.

• 38:22Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 38:27Speaker 01

  So let's talk about LLMs more generally and how we feel about that architecture overall towards reaching this transformative or powerful AI. What are your thoughts on the general idea that LLMs fundamentally cannot be the core of this journey towards reaching this transformative AI? I think Yannick Kuhn is the biggest person behind this hypothesis where

• 38:50Speaker 01

  This next token prediction is the wrong objective entirely and we need something completely different, something like world models or energy-based models that have a real understanding of the input that's coming in and the output that it creates. I think this is important because there's a lot of intuitive thought processes behind it in which, you know, the way you're learning statistical correlations or languages isn't necessarily how the world works and that you need something different to really achieve that understanding to create like powerful AI. Do you think that's the case or do you think LMs do take us to this next frontier that we talk about?

• 39:26Nathan Labenz?

  Yeah, I think there are a couple of different levels that I would want to use to address that. One is, I do agree with the Jan Leckun thesis in the sense that I feel like we are running right now a depth-first search in AI space where we are all jamming as hard as we can on a particular architecture and scaling it as much as we can. And, you know, people are now, of course, like even building chips that literally embody the architecture of the model in the chip itself. I don't really like that. I kind of wish that we were doing a little bit more of a breadth-first search where we would like explore different kinds of architectures and find their relative

• 40:07Nathan Labenz?

  strengths and weaknesses and, you know, hopefully bring, because we're not one thing, right? And we have a lot of modules in our brains. And so it's just fundamentally weird on some level. And you would expect it to be kind of brittle in some sense to take one, you know, relatively simple thing and just stack layers of that. You know, that's like certainly our, the solution that nature found in humans is like a lot more complicated. And it feels like if you want something to be robust in various ways, you would probably want to have different modules. So I do kind of agree that like, It would be nice if we were doing a little bit more breadth of explanation or breadth of exploration rather than just like trying to jam this one thing as hard as we can,

• 40:50Nathan Labenz?

  you know, until we can all retire or whatever exactly the, you know, the dream is supposed to be. At the same time, I think where I would disagree with the Lacun School pretty strongly, and I honestly think this is like kind of a closed question, although he still, he still disputes it and you can find people who will. For one thing, the AIs are not trained anymore on next token prediction in the way that they were, right? So RL is not next token prediction in a fundamental sense, right? The task that the model is given is not, here's a bunch of text, can you predict what comes next? The, what it's trained on now, the signal it's getting now is, did you get the right answer? And the right answer could be a fully verifiable mathematical proof or, you know, numerical

• 41:31Nathan Labenz?

  answer to a question, or it could be one of these things that is, you know, 49,000 evaluation criteria on a huge medical corpus, but it's not anymore that it was supposed to be this token and you gave it this token. It's now qualitatively or quantifiably, as the case may be, did you get the right answer? And then that signal is translated into a gradient update through like a, there's obviously a lot of different mechanisms, but GRPO, group relative policy optimization, is basically comparing for a given model, you know, here's a bunch of attempts that it made, some were right, some were wrong, let's use that to create a direction in weight space and move in the direction that would make the answer that was right be more likely next time.

• 42:15Nathan Labenz?

  So it's, I think something people should all update on at this point, that like, we're not just doing next token prediction anymore. That's still part of the process, but it's, you know, it's not the whole story. And then the other thing is, I think it's also very clear at this point that the AIs do have world models. We can look at the internals, obviously not anywhere as much as we would like to understand about what's going on inside them, but we do have enough of an understanding at this point to create things like the Golden Gate Claude experiment, which I'm sure you guys have seen, where they train what's known as a sparse autoencoder. The huge problem in terms of figuring out what's going on inside a neural network is, it's very dense, right? The, the width of

• 43:02Nathan Labenz?

  A model might be, depending on how big it is, they usually go like powers of two, right? So it might be 4,000 or 8,000 or 16,000 wide vector of activations that sit between the layers. After each layer, there's like this sort of bottleneck of, okay, we've got, let's say, 16,000 numbers that are all, you know, some precision floating point number, whatever, and that represents the state of play. Well, there's obviously way more than 16,000 concepts, right? So that means we can't just have one space for each concept. Instead, we've got to have what is known as superposition, which means like, you know, if just point one is lit up, that might mean something. If points one and two are lit

• 43:45Nathan Labenz?

  up, that means something else. If points one and three are lit up, that means something else. One and four, one, two and three, one, two and four, out to, you know, the vast space of combinatorial possibility. So sparse autoencoder basically tries to untangle all that stuff and say, can we get to a representation where we can look at a sparse number of, and these sparse autoencoders, they're computationally expensive on their, in their own right, and there are millions, I think like tens of millions of activations wide, but they sort of branch this very dense, superimposed concept mess out into this sparse space. And believe it or not, it works. And you can now look and say, okay,

• 44:26Nathan Labenz?

  here's the 10 concepts that are like most active in this network at this time. And then, you know, that's, okay, maybe you're kind of fooling yourself, but the proof is in the pudding when they are able to then say, okay, now that I know what pattern of activation corresponds to this concept, I can intervene on it. So with Golden Gate Claude, they sort of, you know, out of tens of millions of concepts that came out of this process, they found the Golden Gate Bridge concept, artificially turned it up, and now you've got a model that just wants to talk about the Golden Gate Bridge. So, and there's, you know, more stuff that's happened, obviously, since then in interpretability as well. I wouldn't say that the AIs have perfect world models. But they definitely have some world model.

• 45:11Nathan Labenz?

  And I also, again, I wouldn't say people have perfect world models. Right? I mean, we went, we made it into the 1900s without any sense of like relativity. Because the world model that we had was like, good enough for us to get by in the domain, you know, that we were working in. And, you know, it's going to be an interesting question of can AIs start to create those conceptual leaps like a pre relativity to relativity sort of jump? Again, I think, you know, exactly on what timeline that comes, I'm not so sure. But there is a world model inside the AIs. There is a conceptual understanding that is definitely richer than pure like stochastic, you know, correlation of tokens. And that has been demonstrated, I think, at this point, quite conclusively with the

• 45:56Nathan Labenz?

  interpretability techniques that are out there. So I guess, where does that all leave us?

• 46:01Speaker 02

  Um, yeah. You're, well, I, yeah, no, I think that's a pretty comprehensive response to the Lacun objection. Just to translate for our slightly more general audience, although it was very interesting, and we are glad that you went into technical depth there. Basically, what Nathan is saying is that, first of all, it's very important to remember that actually, this objection that, you know, next token prediction can't be the core of intelligence, doesn't necessarily apply, because it's very important to remember that we are now doing reinforcement learning. So we do next token prediction with LLMs to get this sort of basis of intelligence. But then we do reinforcement learning on like, real world tasks, that sort of intuitively seems like something, like a more, more like intuitively what you'd expect the right training objective to be, to get to AGI, or something like that.

• 46:45Speaker 02

  And then separately, Nathan is also saying with this thing with sparse auto encoders and the Golden Gate, Golden Gate Claude example, we have techniques in AI now that allow us to look inside the model when it's responding to a query, and see which concepts, in some sense, are activated by that query as it's responding. And we can see that there are sort of concepts that correspond within the neural network to objects in the real world. Right? So even though it's just an LLM, and in addition to reinforcement learning, it does have this world model that where like, we can say, oh, look, that's the Golden Gate Bridge concept, and it's lighting up when we ask it to answer a query about the Golden Gate Bridge. Is that accurate?

• 47:25Nathan Labenz?

  Yeah, that's great. Yeah. Phenomenal job. And one other thing I would add is, the The explorations that have been done of the embedding space or the latent space are also really quite interesting and revealing. There was one, this has been a couple of years now, you know, these things have gotten much more sophisticated since, but there was one study that just showed that you can kind of do vector operations around the latent space. So for example, if you have the, take the embedding for man and then you move to king and you look at that direction and then you apply that same direction to the embedding for woman, you get queen.

• 48:07Nathan Labenz?

  So there's like an order, a sort of conceptual coherence of the way in which concepts are represented spatially in this like, you know, super high dimensional latent space that clearly is like meaningful, you know, and exactly how it's meaningful or exactly what it's learned or what mistakes it may contain or, you know, what aspects of a true grand unified theory of everything it doesn't have, like those are all open questions. But the, I think that that sort of thing like quite demolishes the idea that, you know, it's all just noise or like there's some, you know, sort of sleight of hand. I think that that organization of its own internal map of the world reflects like some real understanding going on, maybe not human like understanding.

• 48:50Nathan Labenz?

  I always kind of say human level, but not human like. They could be quite alien, you know, but that doesn't mean they don't understand. They don't have to understand in the same way that we understand in order to meaningfully understand. So I think that is again, pretty well resolved at this point. I honestly don't know why some people, you know, can't, can't update on that dimension. It's, it's quite strange.

• 49:13Speaker 02

  Okay. Moving on from the capabilities and training discussion, let's talk about hardware and energy, these other inputs to AI model progress. So it sounds like, you know, you're pretty confident that, you know, either with RL or with some additional paradigm, we'll get to like very transformatively powerful AI. But I assume that part of that thesis involves us continuing to scale these inputs such as talent, such as hardware, capital, energy, um, and all, mainly energy and

• 49:42Speaker 02

  Hardware, a pushback people have oftentimes is that actually there's just not that much more energy to divert to AI model training and inference, right? Like, I can't remember the numbers, but there's like some graph that like the total amount of US energy production is like not very, increasing very fast at all. And AI is like the amount AI consumes is increasing very fast and at some point it's going to catch up and we're going to be bottlenecked by energy with chips. Like, it takes a really long time to make new chip fabrication facilities. And like, we'll just like run out of chips to train AI models on. And even capital, like, maybe like, we'll just run out of money in the world to invest in building new data centers. So, let's see. I guess the question I want to ask is which of these bottlenecks do you see, if any,

• 50:25Speaker 02

  as most plausibly being a major bottleneck to continued progress?

• 50:29Nathan Labenz?

  Yeah, I guess the first thing I would say is I don't think any of those bottlenecks are really fundamental. They are more like cultural or socio-political or whatever. Because there's like, you know, a lot of energy coming from the sun all the time. And the question is, how much are we actually going to harvest and harness? And that is where it becomes a political debate around, you know, who's going to be allowed to build what, where, on what timeline, with what permits, you know, with what impact, whatever. And I think those are, I think it's often overstated the degree to which AI is energy intensive. I actually did a whole episode on this with a guy named Andy Maseley, who has really been, you know, fighting this fight online in a pretty dogged way.

• 51:13Nathan Labenz?

  And, you know, there's a lot of interesting comparisons, but like, a frontier chip today, like an H100 or whatever, it basically uses the same amount of energy when it's on as a microwave or as like an electric teapot. And one query is, you know, maybe on the order of running your microwave for like one second. So, you know, you'd have to be making a whole lot of queries. And increasingly, people are for it to be moving the needle on energy consumption. You know, people don't think twice about like putting something in the microwave for two minutes. And that's probably more energy than most people are using with AI on a weekly basis. So, now it's ramping up and it is starting to add up.

• 51:54Nathan Labenz?

  And it is going to get to the point where if we can't add any capacity, then we're going to have a bottleneck for sure. But again, those bottlenecks aren't super fundamental. China doesn't seem to have them, right? They're adding as much electricity to their system as, I think it's, gosh, I won't say an exact number, but it's like some relatively short period of time in which the Chinese economy is adding as much electricity as the entire American capacity. Now, they have like, you know, four times as many people as well, so a long way to go to build out everything that they might want, but it just shows that it can be done. It also can be done, apparently, in the Gulf. And I recently talked to Sam Hammond, who's an economist and a very AGI-pilled thinker

• 52:39Nathan Labenz?

  in Washington DC. He had just been to the UAE on a trip and I was like, why are we doing these deals with the UAE? You know, all I hear is like, we want to have AI reflect American values and, you know, take American values around the world. I'm like, not sure that the governments of Saudi Arabia and the United Arab Emirates are the greatest partners we could have in projecting American values. Why are we doing these deals? And it seems like honestly, a big part of the answer is because they don't have issues with putting up a new plant. You know, they can just do it and it'll happen fast. And that way, we kind of know that even if we can't do it here, well, we can at least like do it there. So that's all just kind of characterizing the bottleneck and saying like, there's plenty of energy.

• 53:22Nathan Labenz?

  It's a question of who will be allowed to get it, under what circumstances, you know, on what timelines, with what permits, et cetera, et cetera.

• 53:29Nathan Labenz?

  You know, chips are harder for sure because there's, it's a very specialized thing. And that, you know, in terms of like, if you had to ask, if you asked me, what would be the most likely reason we wouldn't get economy transforming AI in the next few years? I would say something happening to the chip fabs in a major way that throws production off to the point where chips are super scarce and, you know, maybe we can't scale the training runs, full stop, or even if the training runs, you know, can kind of still scale, there's just not enough inference to go around. And so, you know, even we might have like really powerful systems, but we just don't have enough access, you know, economy wide for people to deploy them and like automate all the things that seem like we're on track to automating.

• 54:12Nathan Labenz?

  So I guess if I had to pick between energy and chips, I would say chips, but I don't, you know, that seems like kind of a tail risk scenario. All the projections, you know, all the, the whole economy kind of depends on it at this point. So everybody is like incentivized, you know, certainly the political class is incentivized to make it work. The corporate managerial class is pretty incentivized to make it work. There is some tail risk that that mainland China makes a move on Taiwan and that could be a huge disruption, but it seems like as far as I can tell, it would be sort of tail risk type of things that would be the most likely disruption, not a fundamental body. You know, there's plenty of sand, but you know, obviously, which is where like the silicon

• 54:55Nathan Labenz?

  comes from. So, you know, and that's that is scaling too. We are starting to get chips in the US. From what I understand, I'm not an expert on this, but I understand the yields have been decent in the US, maybe even a little bit ahead of schedule, you know, in terms of like, people thought, oh, man, it's really going to be a few years and a few generations and a lot of iteration to get this stuff to be somewhat competitive. And it seems like it's kind of come online reasonably well. I'm mostly, I mean, this kind of a theme, I guess, in my thinking in general, I'm mostly worried about the tail risks. I'm mostly worried about like the AI going wrong in some like really weird way. And I'm, and in terms of what would prevent it, I also kind of think the tail outcomes are, are the most likely to like, put me in the camp of being like catastrophically wrong.

• 55:38Speaker 01

  Yeah. Okay, awesome. Well, let's talk about that then. Let's talk about this like risk of alignment that is the primary concern amongst the community right now. You've had hundreds of these conversations with safety researchers, lab people, builders of the last few years. Net, net, what is your analysis on the alignment problem? Has it become harder or easier in the last couple of years or last several months?

• 56:04Nathan Labenz?

  Well, for starters, I think reflecting on my hundreds of conversations, one of the strangest things in the world today, full stop, is the fact that the disagreement among very plugged in, very informed, very smart people has not really been reduced much at all, even as we've gained a ton of information over the last couple of years about the trajectory of AI. I think that is super strange and I'm honestly pretty confused by it. The one thing everybody seems to agree on is like, the timeline on which we should expect this has come in. There's still disagreement about that, but like Helen Tonner, who's the head of the Washington DC think tank, CSET, and was previously on the OpenAI board, she

• 56:49Nathan Labenz?

  wrote, put her finger on this phenomenon. I think a lot of people were feeling with the blog post that said,

• 56:56Nathan Labenz?

  Even long AGI timelines have gotten super short. And it's basically like, you know, in today's world, if you say you don't think you're going to see AGI until 2035, you're like an AI bear. But only five years ago, that was considered to be quite aggressive. And, you know, most people were more like, I don't know, 2050, maybe not in my lifetime. So there's been this massive compression of timeline. There's obviously been this huge jump in capability. And yet, on these fundamental questions of like, what's going to happen? You know, there's still total disagreement. So I think that's a very weird phenomenon that I can't fully explain.

• 57:31Speaker 01

  Yeah, where do you think this? Where do you think this disagreement is coming from? Is it, is it like cope or some sort of pushback that you're seeing from these like very intelligent people who are still part of the field? Or do you think there's legitimate thought processes behind why there's so much disagreement around this alignment?

• 57:50Nathan Labenz?

  Well, they didn't. I think, I'm not sure I can summarize the conclusions with super high fidelity right now. But, but CSAT, under Helen's leadership, did a workshop where they tried to bring people together and assess like specifically on the question of recursive self-improvement. How big of a deal is it going to be? You know, do we run the risk of this whole process like getting away from us entirely? You know, even on that somewhat reduced question, there was still like very wide disagreement where some people are like, I don't think it's going to be that big of a deal. It'll make people a little bit more efficient, whatever, but it's not going to like bring about some phase change. And other people are like, you know, as soon as you get an ML researcher that can do that, you've got, you go from, you know, I don't know, maybe there's 10,000 people today that

• 58:35Nathan Labenz?

  are really working at the frontier of ML research globally to 10 million. I mean, that's gotta, you know, certain people think that's gotta have a huge effect. And, you know, it's going to be really hard to control a situation if we all of a sudden thousand X how many researchers and they work at, you know, potentially thousands of tokens a second. So why do people disagree so much? They sort of came to the conclusion that people were working from different conceptual paradigms and that these paradigms are pretty good at taking.

• 59:04Nathan Labenz?

  New information into account and kind of explaining it away. So you have some theories that are like, kind of bottleneck theory or O-ring theory, where it's like, you're always gonna have, you're only as good as your weakest link, basically is kind of one way of thinking about it. And as long as you think that, then you can kind of say, well, sure, okay, the AIs can do this, but they still can't do this other thing. And so there's still a weak link and there's still going to be these bottlenecks. And so the whole thing isn't going to get too crazy. And then the flip side of that is like, kind of a jaggedness thing where people will say, well, okay, sure, the AIs can't do this yet, but look what they couldn't do one year ago, two years ago, you know, they couldn't do basic math. Now they're like solving unsolved math problems. So sure, there's still jaggedness, but last time you told me about jaggedness,

• 59:47Nathan Labenz?

  you told me about can't do basic arithmetic. Now we've got unsolved math problems. But these perspectives seem to be really grounded in like worldview priors, or kind of the paradigm that people work in. And it's proving really difficult to get to a real meeting of the minds on those.

• 1:00:08Speaker 02

  Yeah, I saw Ajay Kotra talked about a similar thing on the 80,000 Hours podcast.

• 1:00:13Nathan Labenz?

  Yeah.

• 1:00:13Speaker 02

  A few weeks ago.

• 1:00:14Nathan Labenz?

  A great episode.

• 1:00:15Speaker 02

  Yeah, great episode. She said a similar thing, like, yeah, like economists who expect that like, we will enter some new GDP growth regime, you know, they always point to all these bottlenecks, technological diffusion is always slower than people think. And on the other side, you know, people think that these models are just simply outdated and these people aren't like really taking seriously what it means for AI capabilities to be at the given point that we're like conditioning on. So I'm wondering, do you think that it's possible that people are sort of talking past each other? And like, they're not actually talking about the same level of AI capabilities when they say like, oh, it only like will uplift ML research somewhat. Maybe they're just talking about like, not that powerful AI. But if they were actually talking about like, fully automated ML research, that is good as humans, never have to sleep and run faster,

• 1:01:00Speaker 02

  then they would like see RSI as more plausible. Do you think that's possible?

• 1:01:05Nathan Labenz?

  I think that explains some of the disconnect for sure. I do think people are often, and I've started, at least in some cases, beginning my interviews with like, how AGI-pilled are you? And, you know, what do you, what do you expect to see over the next couple of years? Because if that's not established, and I think they're thinking one thing and they're not, there can be like quite odd. Disconnects downstream of that. So I think getting those assumptions on the table early and kind of at least, you know, cross comparing them is usually a productive or often a productive thing to do. I don't know that that explains all of it. I do think there is still this idea that like, because you do hear of sort of, sure, even if they could, you know, come up with good ML research experiments and this and that, then there's

• 1:01:48Nathan Labenz?

  still going to be this other bottleneck that's going to, you know, there's always, and it's, it seems at times, and I'm not in this camp, so I don't, I don't want to be too blithely dismissive of it or unfairly critical, but it does seem to me that there's sort of an aspect of like faith in the idea that there's always another bottleneck. And I would contrast my position from some of those positions in the sense that a big part of what motivates me is I don't think what I'm saying has to be guaranteed proven right in order for us to be really motivated by the possibility. I'm happy to leave open the possibility that maybe the bottleneck people are right and there's always going

• 1:02:31Nathan Labenz?

  to be another bottleneck and all this will kind of stay under control. Or maybe there's some like plateau around, you know, a quarter's worth of work and we just can't quite break past that. And, you know, there's just some weird Elon, you know, phenomenon that we can't explain and we can't ever quite get there. And I, again, like I said earlier, I think that might be great news if such a thing were to be true. But I'm happy to leave that question open for the time being and just kind of say, I don't know, you know, we don't have a great account of what that would be if there was some, you know, fundamental bottleneck that we'll never get over. And in the absence of it, I'm not persuaded by people kind of just generally gesturing that there will always be another bottleneck.

• 1:03:11Nathan Labenz?

  Because if they're wrong, you know, we're in for a really wild time. And, you know, in terms of like, what should we do about where we are and what might be coming? I think the worst mistake we could make would be to not take it seriously enough and kind of content ourselves with a story that, you know, it'll all kind of self-regulate and, and we'll be fine. Because I don't see a great argument that that's true, you know, and I see at least decent arguments that it might not be right. And again, I look to like our own history and I'm like, well, we've driven a lot of other species to extinction, including our closest cousins. So and some of that was by accident. A lot of it was by accident, right? Like, it was just small bands of people going out and doing what they were doing to survive.

• 1:03:55Nathan Labenz?

  And surviving meant hunting large animals and eating them and using their bones for tools and stuff. And a lot of those animals went extinct, you know, and it wasn't like a coordinated master plan. It kind of happened by accident a lot of the time. So I'm just like, oh, my God, you know, we don't have any like real guarantees as far as I can tell. Plot armor is like, again, too sort of, it would be an unfair dismissal of the more sophisticated people that have, you know, theories about always another bottleneck or what have you. But I do worry that a lot of what goes on among less sophisticated people who, you know, don't want to have to deal with this and kind of would rather believe that everything will be fine, is some sort of like plot armor thinking where they're like, well, I don't know.

• 1:04:39Nathan Labenz?

  I feel like kind of like a main character and like humans feel kind of like a main character. And so, you know, you can't take the main character out of the story, right? And I just unfortunately don't think that that is likely to be the case.

• 1:04:51Speaker 01

  Yes. So you so you find the bottlenecks argument in terms of like all the discussions that we had, maybe with research, maybe with capital spend, maybe with energy, with chips. Those or like just general societal processes as a whole that slow down the overall technological diffusion. But overall, it seems like you're still relatively on the more pessimistic camp. Is that correct?

• 1:05:15Nathan Labenz?

  I don't know. When people ask my P doom, I usually say 10 to 90%. A good friend of mine once told me, we should think less about and argue less about exactly what the numbers are and more about what we can shift them to. So I try not to worry too much about, you know, getting into, I think like one significant digit is all you get on P doom. Is one funny way I've heard it said. I've actually gotten probably a little bit more optimistic over the last few years. In the sense that.

• 1:05:46Nathan Labenz?

  You know, I started reading Eli Ezer in 2007 and the early visions of like the paperclip maximizer and all that kind of stuff. And I can't speak for Eli Ezer and I think he's got, you know, some sort of somewhat revisionist takes on like what he really meant at that time. And sometimes when I see what he's saying now that he really meant, I'm like, I don't know, it's not exactly what I took away back then when I was reading your original work. But whatever, all that is kind of discourse. What I understood and what I think a lot of people feared was a very small system that had some like extremely concentrated form of intelligence, you know, that had had found the right priors, the right inductive biases to be like hyper rational and insanely effective such that given anything

• 1:06:31Nathan Labenz?

  to optimize, it could just optimize it to this extreme state and, you know, tile the universe or whatever. And there was also the idea that it was going to be hard to get such a system to understand human values, you know, that this, what we value was kind of very gradually and haphazardly encoded into us by an evolutionary process over a super, super long time, you know, dating back even to before our species, right? Like other species, you know, care about their young and like seem to be sad, you know, when they lose their children and stuff. So this is like, not even just human. It's like the whole of evolution has kind of led us to be what we are and have the very complicated values that we have. And it was, I think, generally understood, again, although there is some different takes on that history now. But I thought it was generally understood

• 1:07:16Nathan Labenz?

  that like, we would expect that it would be hard to get AIs to have a real understanding of what we care about. And now I look at the models that we do have, and I'm like, well, actually, they do have a pretty good understanding of what we care about. It's not perfect, but I don't think it's crazy to say that Claude is probably more ethical than the average person, certainly more sophisticated in its approach to ethics than the average person. And does generally seem to like want to be good in sort of a meaningful way. You know, when they let Claude talk to other Clods and just let it do whatever it wants to do. It kind of seems to want to like bliss out or something like that.

• 1:07:57Nathan Labenz?

  So there is some sort of rocking of values that I wouldn't have expected to be as easy as it seems to be. And there is some sort of internalization or sort of identity formation, maybe is a better way to say it, that at least some of the models seem to have that at least suggests to me that there's like reason to hope that we could really get there, you know? Could we create? When I first heard this, the question, could we create an AI that loves humanity? I thought it was like laughably out of reach. Then I think I heard that from Scott Aronson, who said that Ilya asked him, you know, hey, do you have any idea like what the Hamiltonian of love is? Or some like, you know, crazy question like this? And he was like, I can't really help you with that, Ilya. And I remember just being like, oh, my God, that's the kind of question they're asking? Like, we are screwed.

• 1:08:40Nathan Labenz?

  And now I'm like, well, I don't know, maybe there was a little bit more to it. I trust Claude not fully, but like more than a human assistant. You know, if I was going to, give me the choice between hire a human assistant with my, even give me, you know, the opportunity to interview, do some vetting, whatever versus Claude, which one would I give access to my email? I think I would trust Claude more than a person that I like interviewed a couple of interviews with all my most sensitive information. There are cases where Claude has blackmailed people. There are cases where Claude has, you know, done various things under pressure. But I think I have better odds with Claude. So overall, I have become a bit more optimistic, but it's certainly like, we're definitely not anywhere near out of the woods.

• 1:09:21Speaker 02

  Right. I guess maybe let me try to steel man the more pessimistic view. And that would be like, okay, sure, LLMs are language native and we can, we communicate our moral preferences via language. But that wasn't really what the people who are very concerned about alignment were talking about even back in 2007. They were talking about AI agents that were goal directed. And, you know, they were sort of anticipating something like reinforcement learning that gets us these more goal directed agents that are able to reason over a longer time horizon. Like have a goal and go out in the world and achieve it. So I guess like the case for like not updating towards optimism with respect to AI safety is that like,

• 1:10:06Speaker 02

  we're still gonna have to deal with this problem where like we're doing reinforcement learning. We're trying to get agents to go out in the world and do things for us. And we still don't know how to make sure like how to define a reward objective. That is like fully what we want to be optimized. Like the maximum degree and and so we have this paradigm like where we got these L L Ms and that was like sort of an update maybe towards AI safety being a little easier. But then now we're back to reinforcement learning and these same concerns apply. Like, what do you think about that objection?

• 1:10:33Nathan Labenz?

  I think that's a pretty good argument and I certainly don't want to leave people thinking that I don't take that seriously or that they shouldn't take that seriously. So I absolutely think we've got more questions than answers. I guess, you know, when I say I was I've become more optimistic, it was starting from a not super optimistic place. Five years ago, or maybe a little more than five years ago, I would have said like, powerful AI seems a long way off and I when we have like very little hope of if we do stumble on it, we're very little hope of controlling it. And now I'm like, it seems closer, but we have maybe a little more hope, but definitely still a lot of of unanswered questions. I think one thing that I do see a little bit differently than

• 1:11:11Nathan Labenz?

  the most hawkish people recently had an exchange online where, you know, there was a there was a 48 hour period where there were like a couple profiles of Amanda Askel and then the people were commenting on her in all sorts of different ways. And Elon was, you know, attacking her and whatever. I weighed in and said, for my part, I have become quite a bit more optimistic that it's at least possible to create an AI that in some meaningful sense loves humanity. And I gotta give her and the Anthropic team a lot of credit for that. And then, of course, I got a lot of replies saying like, well, this doesn't scale to superintelligence. And, you know, this and that and, you know, arguments along the lines of the one you just made. And I basically, again, I think those are all very serious and worthwhile concerns.

• 1:11:56Nathan Labenz?

  But one strand I detected in a lot of those responses is people seem to be imagining a system that is so much more powerful than anything else that if it goes wrong, it's over. You know, you get these kind of ideas of like, Elliot has had this list of lethalities post, you know, and I think that's shaped a lot of people's thinking. And it's like, you have to get this absolutely right on the first try. And I do think that's not the shape of the AIs that I'm seeing in the world today. Right. That's like a quite a hypothetical state of affairs. If you were to drop in an AI that's just so much more powerful than anything else at, and it's, you know, weirdly goal directed and it doesn't love humanity and you like tell it, make paper clips. Yeah, maybe we get all like tiled by paper clips.

• 1:12:41Nathan Labenz?

  But the world right now is much more like, you know, there's a, an emerging ecology of AIs where there's a few frontier ones that are like roughly competitive with each other. Certainly it's not like one Claude instance is gonna, you know, take over the world. What we're talking about is much more of a wave of things where like simultaneously, you know, millions or one day billions of Claude instances and GPTs and Geminis and whatever, all collectively transform the world and that, you know, that has a lot of problems and challenges, open questions with it too. But I don't, I don't worry as much right now that we're headed for a world where like,

• 1:13:20Nathan Labenz?

  one system runs away from everything else to the point where if it takes one wrong move or, you know, there's one bad prompt or one jailbreak or whatever, that kind of all is lost. It seems to me like there is at least, and I think this is kind of like luck, you know, or, or maybe it's fundamental physics, but it's not physics that we understood coming into this. But it does seem like scaling laws in a way are sort of protective because, you know, you get these algorithmic advances and they kind of, they move the needle. They, you know, they deflate exactly how much compute you need to get to a certain level. But they don't tend to make it so you don't need a ton of compute though, still to get to like the very high levels. I think Zuckerberg actually had one of the more interesting takes on this that I've heard.

• 1:14:01Nathan Labenz?

  And I don't think he's like distinguished himself as a great AI safety thinker over time necessarily. But he basically said, you know, at Meta, we deal with scammers, spammers all the time. And the big advantage we have over them is we have a lot more compute. We have just way bigger, way more powerful systems than they do. So we're seeing everything and they're trying to spam and scam here and there. But we're seeing everything, we're monitoring everything. And so like, you know, little things of course happen all the time, but like we broadly can kind of keep it under control. And so you could imagine a somewhat similar dynamic with AIs where it's like one AI,

• 1:14:42Nathan Labenz?

  even if it was like the single most powerful AI in the world, as long as it's not like. Orders of magnitude more powerful than everything else. There's a whole ton of other actors. They've got all their compute. They've got all their instances. You know, they're all monitoring for whatever they're monitoring for. And hopefully that can kind of balance itself out. Then of course, you've got your gradual disempowerment concerns, which is like, maybe that all ends up in some equilibrium with each other and there's no place for humans in it. And that's another thing that I do think is absolutely worth taking seriously. But I just don't see right now that we seem to be on track for this kind of runaway. You know, if, if, if anyone builds it, everyone dies. I think there is like an it for which that's true, but it doesn't seem like anybody's particularly close to building it.

• 1:15:24Nathan Labenz?

  So, you know, that, that's my important part of the analysis from my perspective.

• 1:15:28Speaker 01

  Yeah. Okay. Great. So then when we think about your concrete hopes in terms of achieving this world in which we saw this misalignment issue, can you give us specific achievements that you think are necessary to end up with an aligned AI that doesn't result in, you know, doomsday for humanity? Does it involve training a model that loves humans? Does it involve some sort of like mech inter back to scaling? Or I guess like the sub-goals of training a, a model that is aligned or loves humans? Does that mean mech inter scaling? Does that mean alignment by default? Because models train on human values just end up being aligned? What, what are the concrete, you know, sub-goals that you think have to be achieved to get to this alignment issue being solved?

• 1:16:12Nathan Labenz?

  I'm always interested in something that could quote unquote, really work. And I ask people for this all the time. Do you know of anybody who is working on anything that could credibly really work? In the sense that I can sleep well at night now, knowing that there's something that, that really works. Basically, nobody has anything. So in absence of that, all the frontier companies seem to be taking a sort of defense in depth strategy. The hope is kind of like, and you can, you know, arguably this is inherently the nature of intelligence. It's kind of unpredictable. You know, again, say the same thing is kind of true of humans. There's, there's never been anything that like really works to make sure, you know, a human never does anything wrong. So maybe it's unrealistic to think we could ever get that.

• 1:16:58Nathan Labenz?

  I would still love to see people try, but it seems like where we're headed is kind of a everything in parallel at once sort of strategy where it's like, okay, well, if we can, we might not get the AI to the point where it never does bad stuff, but if we can drive that low, then that's better. And then if we can put a monitoring layer on top of that and catch 90 plus percent of the things that it still does bad, then that's better. And if we can, you know, have additional monitoring systems that ban people's accounts who are like bad actors, then that's better. And then we can also, you know, really invest in like formal methods to improve cybersecurity across the board so that we can sort of take certain risk surfaces entirely off the table. That's one of the things that

• 1:17:42Nathan Labenz?

  it's not going to fully work to solve like all of our issues. But in terms of things that could really close down problems, formal methods to verify the security of software is one thing where there it does seem to be like the opportunity to create genuinely secure software. And this is kind of on the horizon. It seems to be on the verge of having a moment. And then of course, we get the bio risk. So like, you know, we should probably have PPE stockpiles and we should probably have all the things that we should have had, you know, as of the last pandemic, we should have like something that we just bought for our son's hospital room is a ultraviolet light that's supposed to, and I think this is like quite well validated scientifically, kills microbes and is like pretty gentle on the skin. So you can just like shine it in the room all the time. So we should probably

• 1:18:26Nathan Labenz?

  invest in scaling out that kind of capacity. We should have better wastewater monitoring so we know, you know, when things are popping up because it's still going to be the case probably that things are going to be popping up. We should have vaccine platforms, which we do have, right? That are like very quickly programmable. I'm sure you have heard like the story of how quickly the COVID vaccine was designed. In a few days, you know, before the pandemic really even took off in a serious way in a number of cases, the design of the vaccine was already there. Took us a long time, of course, to go through all the trials and actually get it to people. But it was just a few days to, you know, to create that vaccine. So and then there's, you know, there's, I mentioned monitoring of like outputs. There's also going to be these Mechathep internal monitoring type things.

• 1:19:11Nathan Labenz?

  The company Goodfire that does interpretability just put out an agenda called intentional design. And so they're developing ways to try to understand at each step of the learning process, what is the model learning and be able to kind of shape what it learns. So it hopefully doesn't learn certain problematic things and, and does learn other good things that you want it to learn. Then there's AI control techniques. Redwood Research has, I think, done an incredible job of some, of gaming some of these things out. And that's another 80,000 hours episode I would strongly recommend if you guys haven't heard that and recommend it to anyone. Buck Schlegar, Schlegaris, talking about how do we get productive work out of AIs even assuming that they're out to get us?

• 1:19:55Nathan Labenz?

  They've actually made, you know, pretty good progress in terms of building out a portfolio of strategies. So I think that's kind of where we're headed. We're, we're looking at like a world where there's AIs everywhere. Hopefully none are so much more powerful than anything else that they pose like in a single, you know, small individual instance or small pocket, like some existential risk. And then we bring every other strategy we have to bear and that gets us, you know, a few nines of reliability and still probably some, you know, crazy, bad things happen, but hopefully they can be contained enough that the world overall is good. Um, and when I tell that story, I'm like, we definitely need to scale up our investment real quick, for one thing.

• 1:20:36Nathan Labenz?

  Cause the amount of money and resources going into making the AIs more powerful dwarfs the amount that is going into all these other things. So I don't think we're very well calibrated or balanced in terms of where we are putting our efforts. But I do think we have like a bunch of different. Agendas that seem like they all can take a bite out of the problem. And maybe you take, you know, 20 bites out of the problem and you don't have much problem left. Holden Karnofsky used to say, Eli Ezer used to say, death with dignity. He was like, you know, we're probably going to lose this, but we should at least make a real effort. And now Holden Karnofsky, who's senior advisor at Anthropic and recently, you know,

• 1:21:17Nathan Labenz?

  Anthropic just updated their responsible scaling policy, basically backing off from some of the commitments that they had previously made to like pause development under certain circumstances. They're no longer committing to that. So he put out a long defense of why, but a previous thing that he had written was success without dignity. He was like, you know, I don't think that we're doing a great job collectively of trying as hard as we should be trying. And the risk that we're running is a lot higher than I, speaking as Holden, you know, would like it to be. But the problem does look much more tractable than it used to look. And I used to, when five years ago, people would come to me and say, what can I do for AI safety? And he'd be like, I don't really know. And now he's like, I got a long list of projects that people can work on that all at least seem to help.

• 1:21:59Nathan Labenz?

  So, you know, I wish I had an answer that was like, do this and it'll really work. I don't think, I haven't heard any credible claims in that direction, really, except in cybersecurity, which is obviously only part of what the world is going to need. But maybe all that stuff could add up to a win.

• 1:22:16Speaker 01

  Great. So given that we have this enormously transformative technology, and it probably is the most powerful dual-use technology in human history. My opinion, probably consensus opinion also way more consequential than even nuclear weapons, which up until this point is probably more powerful. Do you think that these model providers or, you know, these model capabilities should be in private hands as they currently are at all? Do you have a case in which you support the government in some senses, nationalizing AI frontier development? What's your take on that?

• 1:22:50Nathan Labenz?

  Well, Biden used to always say, don't compare me to the almighty, compare me to the alternative. And I would probably say the same thing about the frontier AI companies. I certainly don't think they're all acting perfectly. And I certainly think the race between them has the potential to get out of hand. So I would like to see government action of some sort. And you know, there's a lot of debates there on exactly what sort of policy we should have. But no policy at all doesn't seem like a winner to me. And I say that as broadly speaking, a lifelong techno optimist libertarian who, you know, mostly would rather see the government stay out of these things. But this one does seem like a, you know, qualitatively different thing where

• 1:23:33Nathan Labenz?

  some government involvement seems, you know, prudent, if not outright necessary. I would not go for nationalization though, for the basic reason that like, I just don't trust the government that much, you know, that I don't. And I look at the current leadership and I'm like, these are not the, I take my Sam's Altman and my, you know, Dario's and Demis's over my Pete Hegseth's, you know, in that standoff, which is happening this week. I support Anthropic 100% for putting some limits on what they want their technology to be used for, and I hope they stand by it. And, you know, if the, if the government comes down on them, like I will preferentially send my tokens their way. And if it weren't, you know, necessarily always the best performance, because I do

• 1:24:18Nathan Labenz?

  think that that is like a really important stand for them to take. So could I imagine a government so competently run and, you know, so pure of heart that it would maybe make sense to nationalize? Yeah, I can, I can imagine a lot, I guess, but like, I don't think we're anywhere close to having that government today. And so I would rather see some competition and some hopefully healthy balance and maybe some of the worst excesses reigned in by the government, but certainly not like a takeover and, you know, put this under the military. That sounds like a recipe for disaster.

• 1:24:49Speaker 01

  Yeah, that makes sense. I guess it's, I'm always also skeptical though of. What we talk about is like corporate incentive alignment. I think oftentimes we think about situations in which, you know, most notably recently, like social media, where you have these, you know, private corporations driven by usage rates or driven by, you know, how much capital they're pulling in, whether that be for the sake of more development or for the sake of purely, you know, desiring more capital. That incentive misalignment causes a situation where, you know, the individuals or the consumers get ultimately hurt. And so I guess on this continuum that we described, where we have pure nationalization of the entire technology

• 1:25:33Speaker 01

  or on the other end where there's no regulation at all. Can you talk a little bit more about where you sit there? And, you know, what that regulation might look like? If you do think it is somewhere in the middle, like you kind of said.

• 1:25:44Nathan Labenz?

  I would probably honestly oppose most of the ideas that rank and file politicians are going to come out with for regulation. Like, I want myself driving cars. I don't want them to say ChatGPT can't give you medical advice. I don't want them to say you can't get therapy from a bot. You know, a lot of that stuff I think ends up just being guild style protectionism and it hurts kind of everybody. Basically, I think what government should do is try to solve the race dynamic coordination problem and take the work, hopefully try to minimize the truly extreme risks. It does seem like right now, like over the next couple of years, the companies believe they're going to create automated AI researchers. They're going to create recursive self-improvement loops

• 1:26:28Nathan Labenz?

  and nobody really knows where that goes. And I don't think that is a great thing to be happening in a few relatively small and kind of ideological companies. And I do think it's also underappreciated and under discussed how much these companies kind of are ideological. Even the definition of AGI as something that can do everything better than humans isn't, it's not a neutral frame. It's a, it's a definition that has like hot baked into it. And I think there is a, there's a certain diversity within the companies. It's not like all of them are like successionists or whatever, but there is a strain of that. And then there's also a strain of just like, you know, wouldn't it be amazing if we, if, you know, we could just replace ourselves and not have to work anymore?

• 1:27:11Nathan Labenz?

  And that I think has kind of taken on a bit of a life of its own. So I do think there's like, A role for the government to kind of say, okay, we're not here to tell you, you can't get free medical advice, but you can't run like super high energy experiments with no transparency that you yourself are saying, you know, and that's one of the strangest things about this whole thing, right, is all the leaders are saying that there's like a significant, they have said for years that there's a significant risk that this goes really badly. And so that's the thing that I would like to see government really focus on is, is there some way to rein that stuff in, make sure we have better, you know, better extreme risk mitigations in place, make sure we have better safety plans. It's

• 1:27:54Nathan Labenz?

  tough. I don't think there's not great or easy answers here, but that's where I would definitely want the government to focus its energy.

• 1:28:01Speaker 02

  Okay. And then another major governance question that interacts with a lot of these concerns about catastrophic misuse or misalignment is, you know, you mentioned the race across American labs, but there's also the race with Chinese labs and the Chinese government, of course. So sort of like the default, um, high level strategic vision in DC, as far as I can tell, and in SF as well, I think, is, you know, we're going to continue to build, continue to race ahead, building capabilities. We have a lead over the Chinese labs right now. It's, there's debate around how significant that lead is, but there's no doubt that we have a lead. We'll continue to build this lead as much as we can via

• 1:28:39Speaker 02

  expert controls, improving privacy and security at the model providers. And then when it comes time, when we're right on the cusp of extremely transformative AI, or even like somewhat into the intelligence explosion, so to speak, we will have time to slow down and coordinate and not push capabilities even further towards super intelligence. And, you know, again, that's contingent on having a significant lead over China. But, but, you know, a sort of an assumption of this strategy is that like, you know, sort of the worst outcome would be both us and China are neck and neck and we're continuing to race and we're in this prisoner's dilemma where even if like both of us are concerned about misalignment or something, we have to continue to raise ahead because otherwise the other person is going to continue to raise ahead. What do you think about this high level strategic picture?

• 1:29:24Nathan Labenz?

  Well, it's tough. I mean, I don't want to pretend it's not tough, but my outlook has always been. Basically, I hate that idea. And I don't think the few months, because the difference is like months, it's not years. You know, that's not a long time to solve all these problems. We've, you know, we've spent much of the last hour and a half or whatever, talking about all the different problems and the different facets. And we didn't even touch on open source, right? And there's like, open source could be a great counterbalance against concentration of power. But if we open source the wrong shit, we can't take it back. You, you know, may have just put a bioweapon assistant, or, you know, even autonomous creator into

• 1:30:05Nathan Labenz?

  the public domain in a way that, you know, is, is going to have super long-term echoes, potentially, right? So that's another whole facet of the problem that you could spill many hours and many, many thousands of pages trying to figure out what to do. So I don't think this idea that we're gonna like, have these couple of critical months, and then we'll, you know, solve everything in that time, and then it'll, it will, it will win somehow. That, to me, just doesn't make any sense at all. I don't want to be naive with respect to, and I don't even want to say China, because I think it's like, it is the government of China, and it's potentially like relatively small cohort at the top of the government of China that I, you know, think we kind of quite rationally, like, should be at least somewhat wary of. Um, you know, I always come back to, the real

• 1:30:49Nathan Labenz?

  aliens in this situation are the AIs, not the Chinese. We have a lot more in common between the United States and China as humans, fellow humans, than we do with the AIs. So is it going to be hard to build trust across the great ocean and, you know, the great civilizational divide? Like, sure, it's going to be hard. And we've got ideological differences in the government and whatever, many, many challenges. But I would strongly advocate for starting to do that work now. Like, for my part, I always want to talk to people in China, when another AI podcast does an interview with a Chinese researcher, somebody one of their top companies, I ask if I can crosspost that to my feed, because I think

• 1:31:30Nathan Labenz?

  we should have a lot more of this researcher to researcher level communication. And I would absolutely invest in, you know, all manner of diplomacy and sort of You know, out there ideas like, could we create some sort of, you know, there's like the CERN project for high energy physics, you know, could we create some sort of shared jointly controlled place where like researchers could go from both countries to like work together on very sensitive problems, you know, could be some small island in the Pacific or it could be in Singapore, I don't know. But like, the idea that we're just gonna sort of decouple and race each other, to me, sounds like deeply unwise.

• 1:32:10Nathan Labenz?

  And I would take my chances with our, the possibility at least, that we could come to some sort of shared understanding with fellow humans versus, you know, recursively self-improving AIs that we hope will be better or somehow, you know, better than their recursively self-improving AIs. All of which, by the way, is happening against the backdrop. And I don't like to say this kind of thing because it's not a very popular opinion. And it's not, it's not something I take any pleasure in. We are looking more like China all the time. You know, what is happening, again, this week? We've got the Defense Department threatening retaliation against an American company because that American company wants to hold firm on what I would consider to be

• 1:32:53Nathan Labenz?

  a core American value of not using their technology for mass surveillance. What is it that people typically worry about when they talk about Chinese AI or Chinese values or, you know, living in Xi's world? I think one of the big things is mass surveillance, right? It's the idea that I can't speak my mind anymore, even in private contexts, because the government's gonna hoover up all that information and potentially use it against me. As far as I can tell, that's exactly what the US Defense Department is trying to coerce Anthropic into doing right now. And what, like, we're losing our, we're kind of losing the thread here in our, you know, I don't know, in our delusions of grandeur, you know, our idea that like, we're somehow gonna be the winners? I was just talking to my seven-year-old last night about this

• 1:33:36Nathan Labenz?

  because he was asking questions around like, why is there a war? Like, well, how does that happen? You know, and I was like, well, almost always, the people that start it go down in history as the bad guys. And almost always, the people in the country that started the war that were kind of convinced it might be good for them, end up regretting it. And I kind of think that's the, unfortunately, that feels like the trajectory that we are on right now, where we're talking ourselves into this idea that if we can, you know, achieve this strategic dominance, then, you know, we'll make them an offer they can't refuse. And we're just forgetting that like, first of all, we're losing ourselves in the process. And second, They get counter moves, right? And like Taiwan's a lot closer to them than it is to us. And these fabs are like easily destroyed, you know, not easily put back together.

• 1:34:20Nathan Labenz?

  So I really don't like that theory at all. And try to advocate for something more conciliatory wherever possible. That's not to say we want to get, don't want to be totally naive. It's good to have leverage. You know, you, you could get me maybe on board with the policy of like, let's not sell chips to China, but let's like rent them freely to Chinese companies, which is somewhat kind of the policy we have. I mean, they can buy from hyperscalers outside of China. I just wish we weren't so. The momentum on this is like, it's, it's the worst. It's America at its worst. And we think we have this enemy and you know, we think we're going to like unite to, to be the best civilization. That just never seems to go super well. So I hope to visit China sometime in the next year and participate in,

• 1:35:03Nathan Labenz?

  do my tiny little part to participate in civilization to civilization understanding. I don't expect to move the needle, but as a gesture of nothing else, I really think more people should be doing more of that kind of.

• 1:35:14Speaker 02

  Very cool. Okay. Yeah. Glad we asked you that. I think that's a important perspective and definitely not one that you hear articulated with that much clarity or depth very often. Yeah. So thank you very much, Nathan. This was an amazing, very wide ranging podcast. I think we got into some good depth as well on some crucial issues. For all of our listeners who haven't heard of the cognitive revolution, absolutely go check it out. Nathan is, you know, one of these people that, that has a technical background, thinks clearly about risks from AI, but also strongly believes in the promise of technology. And he has just like a massive diversity of conversations with people across various sub disciplines within the field. So yeah, thank you so much, Nathan. This was an amazing conversation and really appreciate you coming on today.

• 1:35:55Nathan Labenz?

  Yeah. Thanks so much, Nathan. That's very kind. Thank you, guys.

• 1:36:29Speaker 04

  Between the signs and the fear, I man stand

• 1:36:35Speaker 04

  The wisest minds upon the earth cannot agree

• 1:36:42Speaker 04

  What looked like distant shore now reach a hand

• 1:36:48Speaker 04

  And nobody have what them promised we

• 1:36:54Speaker 04

  No fortress and no wall, no chain that hold Just layer upon layer and we hope and pray A story told a thousand times of old But who can tell the ending of this day? The real aliens, not across the ocean

• 1:37:14Speaker 04

  They coming up from underneath the ground

• 1:37:20Speaker 04

  We trust the current, we follow the motion Something out of us, but by us found

• 1:37:33Speaker 04

  The real aliens, the real unknown Not enemies that we could sit and meet Something stranger that we ourselves has grown Success without dignity, incomplete

• 1:37:53Speaker 04

  I gave the key to something with no name Closer than a stranger, further than a friend It never raised me spice, it never claimed a flame But still you wonder where the patience end The wise men say the wall is gonna hold The wise men say the wall already break Both of them so sure, both of them bold And neither one can tell you what's at stake The real aliens, not across the ocean

• 1:38:25Speaker 04

  They coming up from underneath the ground

• 1:38:31Speaker 04

  We trust the current, we follow the motion Something out of us, but by us found

• 1:38:44Speaker 04

  The real aliens, the real unknown Not enemies that we could sit and meet Something stranger that we ourselves have grown Success without dignity, incomplete

• 1:39:00Speaker 04

  We drove the great beast to the edge of the earth Small bands of people, firing hand No master plan, just hunger since our birth And now a new creation, who are the land? One man say, "Success without dignity" Another say, "At least go down upon your feet" But you know, we were not made for defeat So stand firm, stand firm upon the street The real aliens, not across the ocean They coming up, from underneath the ground

• 1:39:41Speaker 04

  We trust the current, we follow the motion Something out of us, but by us found The real aliens, the real unknown Not enemies that we could sit and meet Something stranger that we ourselves have grown Success without dignity, incomplete The wheel I turn, the river find its way The wheel I turn, nobody ever really will

• 1:40:23Speaker 04

  But we keep turning, we keep turning You know, we keep turning

• 1:40:38Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast

• 1:41:21Nathan Labenz?

  dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.