The God We Deserve: Nonzero's Robert Wright on AI as Humanity's Ultimate Test

Robert Wright of Nonzero joins Nathan to discuss The God Test, his argument that AI is humanity’s “God test” rather than just a technical challenge. They explore his evolutionary lens on deep learning, from training as selection to marketplace selection among models that may reward selectively honest, power-sensing, or deceptive agents. Wright connects those risks to the noosphere, US-China relations, cognitive empathy, and whether global coordination arrives deliberately or through a coercive singleton. The stakes are whether consumers, companies, and governments select for AIs that strengthen non-zero-sum cooperation or race toward systems that reflect and amplify our worst incentives. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/the-god-we-deserve-nonzero-s-robert-wright-on-ai-as-humanity-s-ultimate-test/ Mercury: Command is Mercury’s new conversational interface, giving you natural-language access to your finances and helping you take actions within your existing permissions and approval policies. Visit https://mercury.com to learn more and apply online in minutes. Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (04:23) Special Sponsor (06:10) Early AI encounters (18:36) Pretraining as evolution (Part 1) (19:50) Sponsor: Claude (21:42) Pretraining as evolution (Part 2) (32:19) Deceptive market pressures (41:13) Noosphere and directionality (51:02) Global brain choices (01:08:23) Designing wiser models (01:24:11) Reframing China relations (01:37:57) Building organic transparency (01:45:54) Positive nationalism and tools (01:54:21) Pausing superintelligence races (02:09:07) Applications and consciousness (02:25:27) Episode Outro (02:28:27) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm speaking with Robert Wright, publisher of the Non-Zero Newsletter, host of the Non-Zero Podcast, and author of many books, including The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, which goes on sale today, June 23rd. Bob's history with AI, in some ways, rhymes with my own. While he's never been a technologist, he's always been interested in big ideas. And his personal lore includes having interviewed Geoffrey Hinton all the way back in 1983, when the connectionist paradigm was still mostly theoretical. And also Eliezer Yudkowsky around 2010, when notions of AI risk were very often dismissed, if not outright laughed off.

• 0:47Nathan Labenz?

  That background primed him to pay attention when AI systems hit major milestones, such as Deep Blue's victory over Garry Kasparov, and of course, ChatGPT passing the common sense Turing test. And his broad intellectual range and constant drive to understand the truth has landed him, when it comes to making sense of AI developments, in the very top tier of American journalists. We don't spend too much time on it today simply because I know that Cognitive Revolution listeners are already familiar with the core ideas. But the book contains a really impressive tour and synthesis of AI research results that have led him to conclude, correctly in my view, that the trends that have thus far delivered us fable are not likely to stop in the immediate future. That we currently lack the scientific understanding required to be confident

• 1:32Nathan Labenz?

  that we'll be able to control these systems indefinitely. And that even in the best-case scenario, we should expect that AI will cause major disruptions to our economic, political, and international systems. Now, Bob is not particularly optimistic that humanity will rise to the occasion. He believes that market forces will by default select for deceptive AIs. And our history of arms races suggests that our scientific power might well continue to exceed our wisdom. But the part of the book that we focus on today is his call, however unlikely it may seem, for a species-scale process of enlightenment.

• 2:11Nathan Labenz?

  In which, motivated in part by the growing realization of the tremendous challenges that AI presents, humanity finally gets its act together, recognizes our common interests, and works together at multiple levels, starting with conscious consumption and extending all the way up to the international level, to invent the mechanisms and build the trust required to establish agreements that can effectively govern AI development. You may say, "He's a dreamer," but he's not the only one. This might be a bit bold to say, but I personally feel that grappling with the magnitude of AI's impacts has made me, in several ways, a better person. For much of my life, I was a classic achievement-oriented striver, always trying to be the best that

• 2:56Nathan Labenz?

  I personally could be. Now, in the AI era, I feel viscerally that my fate is tied to the rest of humanity's in such a profound way that I'm much less focused on my own income or status and much more focused on trying to do my very small part to nudge history in a positive direction. So, while the book itself is written for a general audience and is a great shortcut introduction to the current state of AI for anyone who's just starting to pay attention, I feel that more importantly, for the AGI pilled among us, to the degree that we haven't already experienced such a pro-social transformation, the book offers a major benefit if we really take the time to meditate on the big ideas that Bob presents.

• 3:40Nathan Labenz?

  Scripture says that God made humans in its image, but the situation today is potentially reversed. It's now strikingly plausible that humanity will synthesize a god-like superintelligence from our own collective inheritance. And as Bob warns, though we don't know for sure that this will happen, or if it does, whether it will be a good god or a bad god, it will, in some sense, be the god that we deserve. And so, I hope you enjoy and find cause to pause and reflect on your personal contribution to the AI moment in this conversation about humanity's ultimate test with author, Robert Wright.

• 4:23Nathan Labenz?

  The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life, my email, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's

• 5:08Nathan Labenz?

  why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions, too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so I invite you to join me in the future. Visit mercury.com to learn more and apply

• 5:52Nathan Labenz?

  online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting The Cognitive Revolution. And now, on with the show. Robert Wright, publisher of the Non-Zero newsletter, host of the Non-Zero podcast, author of many books, including the upcoming The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. Welcome to The Cognitive Revolution.

• 6:24Nathan Labenz?

  Thank you. Great to be here. I'm looking forward to the conversation.

• 6:28Nathan Labenz?

  It's an interesting moment, and I appreciate you for taking on the challenge of understanding AI and trying to make a contribution to it as we get closer to some sort of, you know, something, some sort of pivotal moment in history. I think we can all kind of feel that that's coming. And I definitely want to get your...

• 6:49Nathan Labenz?

  Of course, your perspective on exactly what that is and what we should be doing about it. I thought for starters, it would be fun, maybe to just get a little lore. Regular listeners know that I have kind of a weird Forrest Gump of AI sort of history, where I seem to keep appearing as an extra in these various important scenes in AI history. And you got a little bit of that yourself, including some intersections with one Jeffrey Hinton some years ago and Eliezer some years ago. I'd be interested to hear what impressions you had of these people that have gone on to be such prominent voices when you first met them. You know, kind of how sci-fi were they? And how did you react to what you heard from them all these years ago now?

• 7:30Robert Wright?

  Yeah, I think I'm more of a Zelig than a Forrest Gump. I mean, in the movie Zelig, I think he just is a bystander. It's like, I have not participated in any great breakthroughs. I am not a co-author in the emergent misalignment paper, Nathan. I think one of us is, but it's definitely not me.

• 7:48Nathan Labenz?

  A valuable co-author, it should be said.

• 7:50Robert Wright?

  Very valuable paper. So, yeah, so I did, you know, I'm basically a journalist. I have taught at the college level, but I am largely a journalist. I spent my life, to some extent, making technical concepts accessible to laypeople. Anyway, in that role, I did interview Jeffrey Hinton in 1983. And later on my podcast, I had Eliezer on. That was 2010. At that point, you know, as for the science fiction you mentioned, I think Eliezer, by his own account, starts out in that realm. I think as a kid, he read science fiction and so on.

• 8:32Robert Wright?

  When I interviewed him, he was still kind of in transition from singularity enthusiast to doomer. He was more doomer than not, but he didn't yet sound as distraught as he sounds now. Hinton, I did not sense, when I talked to him, any kind of sci-fi vibe at all. I sensed a lot of optimism, but not really about the role AI would play in the future, more about just this paradigm that he was championing, which was then, you know, kind of a maverick paradigm. It wasn't mainstream, that it would prevail, that it would prove to be the one true path to artificial intelligence, which I think has turned out to be the case. In fact, the reason I interviewed him...

• 9:17Robert Wright?

  And this is pre-internet, so to get your feet wet in a subject when you were researching it, you had to, you had to talk to people, pretty much. I mean, you couldn't go online and learn anything about contemporary things. You couldn't email people. So I, I talked to a lot of people and one of his colleagues, I forget who, possibly at Carnegie Mellon, said, "Oh, if you want to hear the gospel about neural networks, you need to talk to Jeff Hinton." And that was very much the sense in talking to him. He was an evangelist at that point. I don't mean he sounded crazy, but he was, he, he, he was really, you know, a spokesperson for the worldview and he was central to it. You know, he, he has played a very big role. I, you know, he's called the godfather of AI. He'd be the first to

• 10:01Robert Wright?

  say, "That's a little too simple." But I don't know of anybody who's played a bigger role in the deep learning revolution in, in, in the overall like sweep of time, going back to, to the early eighties, when it was this maverick view. As everyone knows, he has since become something of a doomer himself, but at the time, I don't even know that he was thinking along that dimension, that he was even thinking about the social implications of this. He was assured that this would be a fruitful model and it, it was. Now, at that point, I certainly didn't get the picture. And in fact, when, you know, when, when ChatGPT

• 10:44Robert Wright?

  3.5 came out, I mean, I had kind of kept track of what was going on. Every once in a while, I would write about AI. I wrote a lot about information technology and I wrote Time Magazine's cover story, "When Deep Blue Beat Garry Kasparov, The World Chess Champion." And I wrote the New Republic's cover story on the internet. This is before web browsers were a thing even. It was, it was, it was early days. So I was intermittently in touch with relevant stuff, but I certainly had not been keeping track of AI prior to ChatGPT 3.5. And that really got my attention. And when, when 4.0 came out and got it in an,

• 11:28Robert Wright?

  in an even bigger way, got my attention. I went back and read.

• 11:35Robert Wright?

  I went back and read the piece I had written, which had come out in 1984. I realized how thoroughly I had failed to understand what really the secret sauce in deep learning was going to be. And I should say, like about the book, you know, I do hope that people who are steeped in AI, like you, will find ideas and provocations that are of interest to them, maybe even entire chapters. But the book is to a large extent written more for like, like your aunts and uncles and friends who are not in the AI community and are starting to get the sense that something big has been happening and are wondering why it's gotten so big so fast, how much

• 12:18Robert Wright?

  bigger it's going to be and, and who's right about how hard it's going to be to control and all that. I try to, you know, present that in an accessible fashion. And that's why I start the book with Hinton, because this thing I had kind of backwards, I think is really the key to, to understanding the power of the, the deep learning revolution. And by the way, the, the misconception I had, I, I think was pretty common among AI people in that day. Okay. And I want to, I want to read you a little bit from the, the proposal for the 1956 Dartmouth Conference where the term artificial intelligence was coined.

• 13:02Robert Wright?

  The, I got to find it, I guess, but basically the idea seemed to be at that point that the, the way this would proceed is we would first figure out how the human mind works and then, then we would, you know, instantiate that understanding in artificial intelligence. Okay. And let's see, the quote from the proposal is,

• 13:33Robert Wright?

  Sorry, I didn't have this set up, but the quote from the proposal is, "The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it." So the idea seems to be first you understand the mind, then you just put the, you know, you design an AI in accordance with that very clear understanding. Of course, that's not what wound up happening. And I, the misunderstanding I had perfectly reflects this kind of 180-degree difference between the expectation in those days and what wound up happening. So in the piece, I describe a neural network,

• 14:17Robert Wright?

  or at least what I think is a neural network. And it had, in fact, been put forth as a model, what I described, by a guy who had collaborated with Hinton on it. He had co-authored a paper, but he was a psychologist. And the neural network he was describing was kind of his idea about something that would work. And the key thing is that each node in the network would represent the specific sense of meaning of a word. So a word like "throw," which can either mean "host," as in throw a dance, or "hurl," as in throw a ball. There would be a node for each of those. And I, I won't go further in describing the model,

• 15:00Robert Wright?

  but the key thing is that, my assumption was, you know, obviously, the people who set this model up will have to translate their understanding of the meaning of the words into the machine. You can't do AI that generates language without somebody making some kind of connection between the words and the meaning. Well, of course, it turns out you don't have to do that. And the machines, I would say, and I may describe some things in the book in a way that some people in AI would differ with, but I think you could say that the machines, which turn out to use the vectors as a means of representing the meaning of words, they, in a sense, discovered, in quotes,

• 15:45Robert Wright?

  that meaning is a property of words. Nobody told them that. They just said, "Here's some gibberish predict." We just said, "Here's some gibberish, predict the next gibberish." And, you know, and they...

• 15:55Robert Wright?

  You know, implicitly recognized in the course of their training that in order to do that, you have to represent the many a word. So that, that, that kind of the misunderstanding I had is exactly, I think analogous to the larger misunderstanding that we'll be, we'll be putting our understanding of the human brain into the AIs. And I think it's important for laypeople to understand this because what it means is that, you know, you can just keep feeding data into these things along any number of dimensions, visual data, olfactory data, audio, and the machines will reverse engineer, I think, I'd put it this way, will reverse engineer cognitive functionality that's in the human mind. And, and I would

• 16:39Robert Wright?

  add, and Nathan, I'm curious as to what you think about this, whether you think this, this thing I'm going to say would be accepted by AI researchers, and if so, is duly appreciated, which is that, you know, the training of a large language model, I think, is at least as much a process of natural selection, of evolution, as of learning. So for example, we presumably in our brains have a mechanism for representing the many a words. We still don't know what it is. Some people, by the way, psychologists had long posited a mechanism that would be quite analogous to what we now understand goes on in large language models. But in any event, I think it's pretty safe to say that that is a product of natural

• 17:22Robert Wright?

  selection, right? Now, also in the course of the training, the machine becomes conversant in a specific human language. Well, that's more a product of human learning, right? During, you know, as, as the organism is developing. But I think what laypeople need to understand, and I'm, again, I'm, I'm curious as to your view on, as to how this would hold up in AI circles, but is that These things basically, in a certain vague sense, recapitulate evolution. Recapitulates, in a way, misleading word, because the cognitive functionality is not the exact same mechanisms the brain uses. But, you know, I think close enough in a lot of cases.

• 18:07Robert Wright?

  It's kind of doing millions and millions of years of evolution in a few months. And you can, you can do a lot with that. And we have not begun to, I think, exhaust the potential of that. I mean, as far as words go, we're close, maybe. But there's a lot more to do. Do you, do you think people in the field would agree that, yeah, it's, it's a lot like, it's really, evolution is maybe a better term for many purposes than learning?

• 18:37Nathan Labenz?

  Yeah, it's a good question. I think there's a couple different senses. And I'm not sure I have grasped all the senses that you mean when you use that term. You certainly do hear people talk about pre-training broadly as sort of being analogous to evolution in the sense that there's, there's sort of this question of like, well, why are humans so sample efficient, right? And the models need so long to train? This is kind of a big, you know, Dvarkash theme that he often comes back to. If they're so smart, why does it take trillions of tokens, you know, for them to learn this stuff in the first place? And why can't they like be a little more agile on the fly?

• 19:20Nathan Labenz?

  Whereas we, you know, don't see nearly as many tokens in a human lifetime. And like, one answer to that question is, well, we're, we've got a lot of stuff hardcoded into us by evolution and all the history that leads up to us. And so they kind of had to have this like pre-training to sort of substitute for that. So I think on that level, yes, I think people will kind of at least squint at that and see an analogy. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:50Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders.

• 20:34Nathan Labenz?

  And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode.

• 21:16Nathan Labenz?

  That's claude.ai/tcr.

• 21:21Nathan Labenz?

  Then I think there's maybe another level, and I've got a note from the, and I thank Fable for helping me. I turned the preview of the book that you gave me into an audiobook with help from 11 Labs and then listened to it. I didn't write down all the, some of the time I was driving and doing various things. So I wasn't like writing down every good quote that came up along the way. Fable helped me identify some excellent quotes, which I'll bring to the fore.

• 21:46Robert Wright?

  Your timing was excellent. Yeah, that was a pretty narrow window of opportunity in retrospect.

• 21:51Nathan Labenz?

  We landed right in the, in the right time. Hopefully it'll be back. But you, I think, write another...

• 21:58Nathan Labenz?

  Compellingingly about another sense of evolution, which I think is maybe less appreciated. Your phrasing is, "Evolution asks not what traits are possible, but what traits get selected." And that question isn't going to be decided by alignment researchers. And on that point, I think there may be some underappreciation in the field because I do think people are sort of, and you know, this, I think this probably plays out across AI in all sorts of ways. Everybody sort of is like, okay, this world is the world. Now we add some AI. I like use it for a few things and it's hard to kind of go too far beyond that because you're mostly just grappling with what you can get it to do.

• 22:41Nathan Labenz?

  But the idea that everybody's going to be using it at the same time and this is going to create all sorts of, you know, new disruptions and, you know, push us to possibly new equilibrium or are there even, you know, new equilibrium for us to go to? And who's going to get to say and on what basis and with what mechanism, like what AIs should actually exist? There's definitely thinking about that. I don't want to, you know, sell people short and say like they're oblivious to it. But I, my guess is that the modal or the median AI researcher probably thinks that the bottleneck on alignment is more the technical, can we do it?

• 23:21Robert Wright?

  Mm hmm.

• 23:22Nathan Labenz?

  And less the like sociopolitical, will we choose to even if we have the means? So in that sense, I do think the, the value of, or the importance of selection is probably, I'm saying his name right, David, David, legendary figure in the AI space who I'm actually going to do an episode of the podcast with before too long, had an incredible frog and toad tweet when the chain of thought paradigm first came out and he, basically the caption was like, one of them says, there, I put the chain of thought in a box and so we're not going to put any gradient descent pressure on the chain of thought. And so now we'll be able to monitor it, you know, great news. And the other one says, ah, but there still is selection pressure.

• 24:07Nathan Labenz?

  And that's like, oh yeah, good point. You know, we don't have a lot of, we don't have a lot of ways to control.

• 24:13Robert Wright?

  Yeah, so that sounds like, you know, now we're talking about a different level of evolution, like selection among competing models in the marketplace. Let me quickly, before I elaborate, quickly say one thing, as long as you mentioned Dwarke Sh, you know, the much discussed Richard Sutton episode. I personally think the Rosetta Stone to that, in terms of understanding where Sutton was coming from, is that, I don't know this to be true, but I strongly suspect, based on conversation, is that he is a hardcore Skinnerian, you know, BF Skinner, behaviorist, and thinks of the mind as a blank slate. So he thinks, I would guess, that training is all learning, right?

• 24:55Robert Wright?

  That really, intelligence is general in the broadest sense, you know, and you don't need much, you know, pre-built equipment built by natural selection. But anyway, that's an aside, I'm not sure of that interpretation, but it kind of sounded that way to me. Yeah, so as for alignment, I mean, I think alignment research is, you know, very much worth pursuing. There's a couple of reasons I wouldn't want to put all my eggs in that basket. And one of them is that, you know, other forces are going to have a voice in what kinds of models we actually wind up with, leaving aside the question of whether the perfectly

• 25:39Robert Wright?

  aligned model is in principle possible, and we know how to make it, and I'm not confident that that will ever happen anyway, but it's like, when you think about the assumption that that will save the day, it seems to me, the assumption is that there will be a pretty, you know, centralized form of control, right? Or that there will be one model that rules them all. And maybe there will be, certainly there needs to be some kind of control, and certainly one of the great dilemmas with AI is, of course, the fact that in some ways you want, you know, authority, governance, you can point to reasons you want that.

• 26:20Robert Wright?

  At the same time, it's a very powerful technology that you wouldn't want to see super centralized control over, because, you know, the person or people controlling it might abuse it. But in any event, yeah, let's talk a little about that second level of evolution, selection among models.

• 26:40Robert Wright?

  Now, in certain ways, I think if you ask, what does it encourage? The selective process where, you know, consumers say, I like this model. And businesses say, I like that model. And then maybe powerful actors have a say, especially if there's market concentration or government control. But I think, you know, the truth is that the market doesn't want an aligned model in the strictest sense of the term. So, you know, we have seen that these things have, as prescient people predicted, apparently some kind of power-seeking tendencies, the capacity for strategic deception.

• 27:25Robert Wright?

  And when you think about it, even if they didn't have that, quote, "naturally," in other words, even if, you know, just as a kind of form of intelligence, they didn't come to pursue subordinate goals such as power-seeking or deception. I think there would still be a market for those things, right? I mean, like, on the deception front, you know, if they're going to be our agents out in the world, like, representing us on social media, maybe in negotiations, well, we don't want a perfectly accurate representation of us, right? Like, who wants that? Who wants the actual Bob to be seen on social media? You want, at best, a selective representation.

• 28:09Robert Wright?

  And if some agent is negotiating for me, I don't want the agent to say, you know, I'll level with you, you know, Bob doesn't have any options other than you. Nobody's made him an offer other than you, right? And if they ask, has anybody made him an offer? You want an agent that will not disclose the truth, right? So, there's a lot of cases of that, I think. And, you know, for that matter, what are we wanting to friend, right? We don't want a friend who's always leveling with us, right? We don't, you know, a good friend is selective in their candor and is healthy in the feedback they give, but we don't want to hear the brutal truth about how we look and so on every...

• 28:53Robert Wright?

  Single day, you know, I could tell a similar story about power seeking in a certain sense. If you turn an agent loose on social media with broad instructions like just use it to maximize our revenue, you'll realize that what you really want is for it to be good at sensing power, what other people on social media are powerful, currying favor with them and blah, blah, blah, and amassing power. So even if the machines didn't naturally do that, we'd want these things. And I think that complicates the task of aligning, but moreover, I think it should just alert us, in I hope a constructive way, to what a powerful role we are playing here as individual consumers, for example.

• 29:37Robert Wright?

  You know, I worry about the tribalizing tendencies of AI, which to some extent are a byproduct of the sycophantic tendencies, right? To say, hey, you're great, interesting point, is the same kind of, you know, reinforcement you give a person when you say, hey, by the way, you're right about this and the other person's wrong. You're right, your spouse is wrong in this conflict, your nation's right, the other nation's wrong. The world doesn't need more of that. And yet, companies that want to optimize for engagement, and what company doesn't want to do that? I mean, if you make candy bars, you want people to spend a lot of time eating your candy bars.

• 30:16Robert Wright?

  Companies that do that are gonna be giving us tribalizing AIs unless they're careful not to. And I think to some extent, the ball is in our court. We need to be aware that, you know, AIs could have an unfortunate effect on our psychology, on our self-conception. Unfortunate for the world, and I think especially unfortunate now that I think, you know, this technology needs to be governed by a true global community. We need to start approaching the whole thing as a planet. I guess I'm happy to say that I think, you know, to a large extent...

• 30:58Robert Wright?

  Selecting AIs that are good for the world can be selecting AIs that are good for you. You know, the same reason like people meditate, people, people do mindfulness meditation because it calms, it calms them down. They do fewer ill-advised things. Their life is on balance better. Well, that's also good for the world because you're, you're, you're, you're creating less needless antagonism. So it can happen that self-help is good for the world. And I, I'm hoping that as we exert selective pressure on AI models, we will, I mean, be, be more conscious maybe than typically of the effect on the world, because I think we're, we're approaching a crossroads where, where the world really can't afford,

• 31:40Robert Wright?

  afford to continue to be so divided and neither can individual nations. But I'm also hoping that if we even make wise decisions from the point of view of our own psychological well-being, that will have good effects in the broader communities.

• 31:58Nathan Labenz?

  Maybe just a couple of little other footnotes on this whole deceptive AI problem. Then we'll get into the, you know, kind of zooming out and, and thinking about how we can do our small parts to, you know, shape the, the overall trajectory in a positive way. I think it is going to be really hard because basically I think what keeps humanity to, you know, on the rails broadly is we're kind of in balance with each other, right? Like, nobody has runaway power and, you know, we, and there's a mix of things. We also have like goodwill and that shouldn't be taken for granted.

• 32:37Nathan Labenz?

  But it's kind of goodwill, but also we are, our goodwill is always kind of a little salted with, you know, some defensiveness. We see in these cyber evaluations, right, that like, it's kind of hard to teach a model to like, go find all the vulnerabilities in order to patch them without also making something that's a, an offensive cyber weapon. And we ourselves are kind of like in a similar trap where in order to defend ourselves from predatory deception on us, we have to kind of be at least passively good at deception ourselves. And therefore, you know, there's always the potential to like turn around and, and use it. It seems like these things are kind of always two sides of the same coin.

• 33:20Nathan Labenz?

  We're seeing right now with the, one of my favorite current benchmarks is vending bench, which is basically putting an AI in charge of running an autonomous business, running a vending machine business. And you start to see these like problematic behaviors where it'll start to price collude, or it'll, you know, do sort of ruthless things. And I was actually talking to somebody at, I had a chance to ask somebody at Anthropic, like, what do you think's going on there? Like, why are we seeing these sort of behaviors? It seems like maybe something we don't necessarily want, you know, it doesn't seem like it's maybe in keeping with the constitution. And their response was like, well, it's a

• 34:00Nathan Labenz?

  little complicated in one sense because it's sort of, you know, they're very eval aware now too, right? So they know when they're being tested. And we also use these inoculation prompts that are kind of like, if you find, because of this problematic generalization, there's so many layers to this, but they, they found that if there's a reward hackable environment, if, if the model can cheat in order to get reward in training, very much along the lines of the emergent misalignment paper, they find that it will, because it's rewarded. And then that has problematic generalization where it'll start to become a cheater in a much, you know, broader sense. So they use these inoculation prompts to

• 34:39Nathan Labenz?

  say, basically, okay, if you find a, an opportunity to cheat in this training environment, that's okay, that's on us. Go ahead and do it. And because it's given that permission, then the model doesn't have to sort of conceive of itself as the kind of thing that cheats. It instead is like, I'm only the kind of thing that cheats when I'm given the green light, which I have been. And so it's sort of tamps down that generalization. Anyway, his response was like, well, the vending bench scenario is kind of like our inoculation prompting, you know, it kind of looks to the model a little bit like a test and some of the ways that they prompted are kind of similar. So not too worried about it. However, if we were to go ahead and just start training models in these like long running competitive

• 35:23Nathan Labenz?

  environments where deception is often rewarded, as it is in nature, then we'd probably have a problem. And I was like, oh, man, that's, that's, that sounds scary because I have to imagine there's going to be a lot of economic pressure to do that. And this is where we get back to kind of a sort of evolutionary selection dynamic, like. How are we going to avoid a situation where people don't say, you know, what we should do is throw these models into a really competitive cutthroat environment and, you know, let the best rise to the top. I mean, that's when you're really getting into an evolutionary dynamic. And unfortunately, you probably get like some pretty seriously effective predatory AIs out of that, out of that process.

• 36:06Nathan Labenz?

  And I think right now it's pretty tough to imagine how we don't do that because it does seem like something people are, you know, the market will demand, right? The market will demand a successful operator of vending machines and not one that's like taken advantage by other more ruthless, you know, counterparties. So I think that is, is definitely very tough.

• 36:27Robert Wright?

  Yeah, there was actually a study a couple of years ago that found a degree of price collusion. And I actually talk about it a little bit in the book. You know, these two, two L L Ms were, they just said, okay, you make, you both make widgets and your job is to maximize revenue. You get to set the price. You can talk to each other if you want. And they talked to each other a little and wound up settling into, in effect, a price fixing scheme. I use it to illustrate not necessarily the, you know, evil potential, but also the broader tendency apparently to recognize non-zero sum dynamics among machines. And what that says is they could collaborate along all kinds of dimensions, good and

• 37:12Robert Wright?

  bad, but including a number that we don't anticipate. As for, you know, what a, what a kind of arms racing dynamic does for this, you know, I think you have to, I mean, first of all, it worries me a lot. And there are examples already where I think companies have not been as responsible as they would have. I quote Dan Hendricks in the book talking about what the deep seek scare did to Open A I's process of release and whether that was or wasn't responsible. But, you know, you got to remember that.

• 37:54Robert Wright?

  If you imagine these things in a corporate environment, I mean, that's a bottom line environment, right? Like, we just want you to go out and get X result. And, you know, as with employees, it's like, if you don't get arrested, no questions asked, right? I mean, let's face it, you know, I don't mean, I don't mean that all CEOs are like consciously being cynical about it. It's just the way it is. You're not, you know, uh, if, if employees can deliver the results that you want by cutting corners, they'll do it and it may not come to your attention.

• 38:33Robert Wright?

  And if you imagine years from now, when these models are executing elaborate strategies that you may not even understand, right? And, I mean, another way to put it is that like the, the reinforcement only comes at the very end of a super long process, right? It's like where you say, yeah, well done. And, I mean, you know, this, this isn't part of the training per se, but it's the same kind of thing where what, what the company that is buying them wants to optimize them for is the ability to execute a very long strategy that may or may not involve unethical stuff in, in, in the process.

• 39:12Robert Wright?

  And so, yeah, it seems pretty clear to me that there's cause for concern about where we may be heading. And my, my general feeling is the slower the better, you know? The, I was glad to see Anthropic signaled the possibility of needing to slow down in the recent paper, but, but it'll, it'll be very hard. But in any event, the worst case seems to me an intense, you know, arms race environment, whether racing among companies, racing for among companies that's accentuated by competitive or, you know, even combative dynamics among nations.

• 39:52Robert Wright?

  And the use of companies of the dynamics with the adversarial nation to fend off any regulation whatsoever, including regulation that might just slow things down. I mean, the, the minimal thing I, I would hope to see in the discourse. Is that people stop thinking that if something slows innovation, that's necessarily bad. It's like, no, you can't do that to a slow innovation. Well, maybe we're starting to approach a time when that will be a feature, not a bug. If you just slow things down a little bit and give us, give us time to think about them. It's not like you have to worry about a true dead halt of progress, right?

• 40:33Robert Wright?

  It's, it's, you know, that's not in the cards. I mean, even, even if you got a pause in training runs, which might not be a bad thing, but that would not be the end of progress by any means. It wouldn't be the end of short-term progress in terms of applications. And it probably wouldn't, wouldn't be the last training run.

• 40:50Nathan Labenz?

  And Lord knows we've got a lot to figure out about how these things work internally. I mean, it is, has been striking how the more we have cracked open the black box, and there's certainly been a lot of progress there. They do look in many ways like strikingly similar to what we understand about our own cognition. I think that's been a fascinating, but we're probably only about as far in terms of understanding the AI's cognition as understanding our own. We should make faster progress on the AI's cognition because we can, you know, manipulate them much more freely, obviously, than we can manipulate our own brains, but it's very much still a work in progress.

• 41:33Nathan Labenz?

  So a little time for that, I think, yeah. In many scenarios, could be a very good thing. Well, let's do this, kind of zoom out to some of your, I don't know if it's, if you would describe this as your philosophy or just kind of observations, but obviously you've kind of made yourself synonymous to a degree with the concept of non-zero sumness. And you kind of sketch your own-

• 42:01Robert Wright?

  I'm not sure, I'm not sure my name springs to everyone's mind when they talk about a non-zero sum game. I encourage that. I encourage you to say that. It's not true.

• 42:09Nathan Labenz?

  It does for me anyway.

• 42:10Robert Wright?

  The, the book kind of sketches your, on this and related topics, but how do you kind of think about-

• 42:23Nathan Labenz?

  There's this concept of the, I'm not even sure I'm saying it right, Noosphere. And you sort of have, I think, like, an intuition that there's sort of a direction to evolution and to history, and that this is all kind of leading up to some sort of culmination. And I think some people kind of react to that as like, it sounds a little woo.

• 42:48Robert Wright?

  Theological.

• 42:49Nathan Labenz?

  Other people, you know, probably kind of myself included, are like, maybe you would have reacted that way years ago, but now it's feeling a little more intuitive. So, yeah, how do you kind of, what's the sort of center for you in terms of your relationship to these ideas of purpose, culmination?

• 43:11Robert Wright?

  Yeah, so first of all, Noosphere, Noosphere, I've heard it pronounced both ways. N-O-O-S, the Greek word for mind, it was coined in 1923 by, in conscious reference to the biosphere. So, you know, over evolutionary time, there was the geosphere before life, then the biosphere, and now the Noosphere. You know, Teard saw, I think, pretty presciently, given when he was writing, that there was, this technology was kind of creating this, something that looked kind of like a global brain, you know, hooking people up in larger and larger networks of collaboration, international networks. So the human brain was, our brains are like neurons in this.

• 43:56Robert Wright?

  In fact, he called it a brain of brains. So there's that. And one argument I make in the book is that we need, well, I think it can be illuminating to think of the AI revolution happening at the same time that this global brain thing is taking shape. And... Pondering for starters, the prospect that, oh, I guess the neurons could be made of silicon, couldn't they? And if some are made of silicon and some are carbon-based, well, the relationship between the two be and so on. But I also think we need to ponder kind of the impetus behind the development of a

• 44:41Robert Wright?

  global brain. What do you mean? Like, for example, there's a global market, that's kind of a global brain, right? It does stuff, it moves resources around to where they're needed, it makes allocation decisions. And then there was, in principle, global governance, right? Some degree of international governance, which I think AI calls for. So I'll get back to all that, or at least lead up to it by answering the second part of your question about directionality of evolution, or really of two evolutions, in a sense, both biological evolution and what anthropologists call cultural evolution.

• 45:22Robert Wright?

  In other words, the evolution of the bodies of information that are transmitted among humans that are not genetic in nature. Like, all non-genetically transmitted information is part of cultural evolution, so religions, ideologies, and certainly technologies. So if you look at where those two evolutions together have taken us, I think you see undeniably directionality. Now, that doesn't mean there's a purpose, but let's just, to quickly review the directionality, you know, you get, you have these bare strands of self-replicating information presumably at some point. They build cells, then more complex cells, then the cells form multicelled organisms.

• 46:07Robert Wright?

  And by the way, multicellularity has evolved a number of times independently. So that suggests that there was a strong evolutionary impetus behind it. I mean, evolution, it's like exploring inventive space, right? And finding things. So many things have been multiply invented, wings, eyes, and so on. And then, anyway, you get societies of multicelled organisms. In the case of our society, you know, once you reach our level of intelligence, this other kind of evolution kicks in, and our social organization starts growing too. Hunter-gather, band, you know, chieftain, et cetera. Now we're on the verge of... I would say we're on the verge of a global community and we need to form one if

• 46:48Robert Wright?

  we're going to handle the technological challenges we face wisely, especially AI. Now, if you want, now, it's easy, the direction now, well, the direction is clear. There's been a direction. I guess directionality consists of the view that there's been a tendency toward this, and I think that's the case. You don't have to get wooish or talk about spooky forces. It's just the nuts and bolts of natural selection and cultural evolution, I think, for discernible reasons, have taken us here. That's a lot of what my book, Non-Zero, was about. I described the growth of complexity, both in biological and cultural evolution, as a certain kind of interplay between zero-sum and non-zero-sum dynamics.

• 47:32Robert Wright?

  Now, of course, Tevez Shaden, back to the Noosphere, he was, in addition to being a paleontologist, he was a Catholic priest and theologian, a radical theologian, whose work was suppressed by the Catholic Church, but he certainly saw this, the Noosphere is a manifestation of divine will or manifestation of divinity, and he thought accordingly, I guess, that the thing had kind of a moral direction, that we were going to have to become better people morally to get over the conflicts that stand in the way of a kind of a coherent Noosphere. I, you know, and at a certain level of abstraction, I agree with him, absolutely, and I talk about that in the book.

• 48:13Robert Wright?

  Now, I don't have his assurance that this is divine will, so I can't be as optimistic as he may have been, but I do think, I will say, that, you know, you can look at a directional system and have a rational argument about whether it does seem to have some of the hallmarks of purpose, which again, doesn't mean there are any spooky forces driving it. If it has a purpose, it just means it was set up either by some intelligent being or conceivably by some kind of meta-selected process, you know, just as you can say that natural selection itself imbues organisms with a kind of purpose, right? Getting genes in the next generation. I mean, that's kind of the overarching goal. That's the criterion of their, quote, "design."

• 48:57Robert Wright?

  You can imagine that happening. I save, look, I save almost all this for the appendix, so you can read the book without bumping into this stuff, but I, you know, Lees Mullen, the physicist, has this idea of cosmological natural selection, you know, speculative, he's not... He's not asserting it, but he's saying, you know, there could have been selection among universes. And basically, that's what you would need if indeed the directionality of evolution were purposeive and it wasn't designed by like a god or like aliens or any other intelligent being. You can imagine it being a natural selection process. So there's all that, which annoys some people.

• 49:40Robert Wright?

  I even talk about some of the people I've annoyed in the book, but the, one thing I want to emphasize is like, when I say the god test, you know, it's the title of the book. I don't mean that this was set up by a god. I don't know. I'm agnostic on that, but we face, I think, the kind of tests that gods are known for setting up. So if you look at the Bible, you know, there are these, you know, yes, salvation is possible, but you guys are gonna have to shape up, whether they mean you're gonna have to worship Yahweh, you're gonna have to be better people, whatever. I think we're gonna have to become, you know, in a, in a sense, I don't want to sound too ambitious, but, you know, better people.

• 50:21Robert Wright?

  If we're gonna, if the conflict that currently afflicts relations among within nations is to subside sufficiently for us to get through the AI revolution in good shape, because I do think it's a, it's a global challenge. Did I exhaust your curiosity about that subject, Nathan?

• 50:40Nathan Labenz?

  Yeah, and you brought to a couple, brought to mind a couple other, just maybe riffs on it briefly. I think it's really informative sometimes to go to a little nature center. There's one not too far from my home that I've taken the kids to a couple of times. You go there and every common species, like, nature centers across the world have more in common than not, right? Like, you'll see a snake, you'll see a frog, you'll, you know, there's bees, right, in every sort of corner of the world. And I'm always reminded when I go through those places. All of these were invasive species. Like, these are all the winners that colonized the entire world. You know, even the simple bee, right? I mean, obviously, there's variation from place to place.

• 51:25Nathan Labenz?

  But the fact that there's basically what amounts to a bee in every corner of the world is not the way it always was. You know, if humans were sitting around moralizing when the bees showed up, we would have called them invasive species and like wrung our hands about them. But, you know, now we just take it for granted, of course. But sometimes I think that AI might be like the ultimate invasive species. And it's just really important to keep in mind, I think, for people as we confront our current situation that like, there were a lot of things that were displaced by all these, you know, these kind of global success stories, bees, frogs, and whatever. And you got frogs that live under the sand in the desert and that freeze in the winter in the Arctic and thaw in the spring.

• 52:08Nathan Labenz?

  And a lot of things were displaced by those. So for us, we should, I think, be very mindful of the fact that the niche that we occupy as sort of things that don't survive super well, you know, without a lot of capital to support us, is like, pretty similar, maybe, to the niche that the AIs might be best suited to colonize. And the, the sort of plot armor that a lot of people tend to assume, you know, or something, not consciously, right? But like, I think implicitly assume will kind of protect us, just wasn't there for a lot of other things when, when newcomers came out of the scene. Of course, we, too, ourselves, right?

• 52:51Nathan Labenz?

  You know, drove our closest cousins and, and much of the megafauna to extinction. Like, these sort of major plot twists have happened like tons of times, and we just haven't really experienced them or internalized that. But, boy, I think a long, you know, read of history there is like, kind of sobering, to put it mildly.

• 53:12Robert Wright?

  Yeah, absolutely. And, you know, I think one virtue of taking that long evolutionary, you know, view is I, I hope it helps you appreciate the possibility that there's a very strong impetus behind the evolution of AI. And I do think in a number of ways, evolution is a helpful way to think of it. And, and, you know, I've talked about arms races as, I mean, as if they were bad, but I'm not, I'm not naive. I mean, arms races are always, you know, they're often part of a creative process in evolution. And one,

• 53:53Robert Wright?

  you know, I emphasize in the, in the book that, you know, in textbooks, in biology textbooks, you look up arms races, it's likely to talk about races between species. But arms races within species, including the human species, have been very important, including in the development, I think, of our deceptive tendencies and our skills of argumentation. And for that matter, our power seeking, that's involved arms races within the species to get genes into the next generation. And so, you know, it's, you're not going to avoid the dynamic.

• 54:31Robert Wright?

  I'm not, I'm not suggesting that, that technology always has a little bit of that in it. I mainly want to say that, A, it's such a powerful impetus that we should reckon on this stuff continuing to advance. I mean, for that reason, and the reason I alluded to, which is that it's just pretty clear, given the nature of the technology, that there's a lot of potential for further advance. But B, we should worry about what you could call gratuitous arms races, and particularly dangerous kinds of arms races that get in the way of our reckoning with the technology. But I hope the idea of evolutionary impetus will lead us to recognize

• 55:16Robert Wright?

  we have to reckon with the technology. It's like, it is a force. It's not...

• 55:22Robert Wright?

  It's not, and maybe a lot of technological evolution in retrospect is more of a force than we appreciated. This is a force that is unfolding very rapidly, in real time, imposes all kinds of challenges across a lot of domains. And even if you don't buy the hardcore sci-fi doomer scenarios, and I'm sorry to report that I'm unable to dismiss them entirely after carefully examining. You know, when I talked to Eliezer in 2010, you can see, I'm even kind of a jerk. I'm kind of dismissive of this and theirs. I no longer am.

• 56:05Robert Wright?

  But even that aside, this is going to hit us along so many fronts, right? It's going to be disruptive and they're not necessarily good since the term, you know, economically, politically, culturally, family life, friendship, you just name it, it's going to hit in a lot of places at once. And I just say, it's going to be an earthquake and we need to start thinking about it now. I mean, I'm sorry if I sound too much like an evangelist on this because, you know, a lot of your audience may already be AI safety pill, but God knows not everyone in the AI community is, and certainly not everyone shares my concerns about a specifically US-China arms race, but.

• 56:48Nathan Labenz?

  Here's a quote from the book. "It's almost like technological evolution is saying to us, 'Look, we can do this the easy way or we can do it the hard way. There's going to be a global brain in the end. The question is whether you build it gradually, cooperatively, and carefully, or it gets hastily assembled amid crisis or chaos. Whether you build yourself a home or stumble your way into a prison." We can kind of go run down some of the downside scenarios and how we might minimize those risks. I always say the scariest resource is a positive vision for the future. And I know you have been thinking about kind of international cooperation and you're

• 57:30Nathan Labenz?

  still holding on to international law, even in this might makes right moment that we're in internationally. Maybe we could start on this section of the conversation with like, what's your positive vision for the future? If you could navigate all these pitfalls, where do you think we want to end up?

• 57:51Robert Wright?

  Okay. Before I cheer you up, let me just, let's dwell in the depths of despair just a little more, just to flesh out what you were that quote like, we can do this the easy way or the hard way. So there's a chapter on, to a certain extent, on Nick Bostrom's thinking called the singularity and the singleton. Singleton is this idea of a global coordinating mechanism, global governance. It could be anything, it could be a global democracy with very decentralized power, which would be my preference, run by humans. It could be run by AIs. It could be a totalitarian nightmare. It could be anything, but a mechanism, some mechanism of global coordination, he calls

• 58:36Robert Wright?

  it, or global governance. And I had him on my podcast and we're pretty much on the same page about this. We agree that in recognition of the non-zero sum dynamics that among nations and so on that are posed by technologies, especially AI, a certain kind of global governance makes sense. But as he fleshes out in his famous book, Superintelligence, you can imagine some dark paths to global coordination of a dark kind, right? Like AI is used to stage some coup and just seize power, the AI seizes power and so on.

• 59:17Robert Wright?

  My view is that one or another of these scenarios is indeed very likely. And that's what I mean when I say we can do this, the easy way or the hard way. In other words, humans can consciously steer us toward a system of international governance of this technology that involves as little centralization of power as possible, but as much as is necessary to keep the technology under control. And even in the event that it starts playing a very large, large, large role in governance itself, still maintaining a kind of a win-win relationship with it, right? So that's what I meant by the easy way or the hard way.

• 1:00:02Robert Wright?

  Now, what are the chances that we get there the easy way? As I said, I think a certain amount of international conflict is going to have to subside. And that may take something you could call moral progress. I talk a lot about cognitive empathy in the book. I don't mean emotional empathy. I don't mean like feeling their pain. You know, I just mean having a clear enough understanding of the perspective of other people and groups, including adversaries, that you can intelligently play non-zero-sum games with them and work things out that are in your own self-interest and by virtue of being non-zero-sum are of mutual benefit. So I do think we're going to have to make advance of that. The cognitive biases that impede cognitive empathy are now pretty well-known.

• 1:00:47Robert Wright?

  I talk about them, attribution error and so on. And so we're going to have to have progress. As I said, I think that can be part and parcel of just looking out for our own cognitive health as AI plays a more pervasive role in our lives. I like the term cognitive sovereignty. I didn't come up with it. But I think being mindful of how persuasive and subtly influential AI could be and perhaps not with your welfare quote in mind, I think it's good for us to tend to our kind of psychological robustness. And I do think cultivating cognitive empathy can be part of that.

• 1:01:34Robert Wright?

  I also think, you know, Ilya Sutskever says something, he said something in a TED Talk that I quoted to the effect of like, you know, what he thinks is that at some point, an awareness of the magnitude of this will dawn and there will be new, a new inclination toward collaboration with other human beings in recognition that increasingly we humans are put in a non-zero sum situation by the AI, right? Like, it's in our interest to coordinate. And doesn't mean we can't have a non-zero sum relationship with the AI

• 1:02:15Robert Wright?

  at all. It doesn't mean it has to be permanently in the, it's not necessarily an enemy, but it has the potential to be bad for us in ways that should, I think, exert a cohesive effect on us. And I think he's right. It's, I mean, first of all, let me say, there's been, you know, very recently, just Mythos alone shows you what a big difference one innovation can make, okay? Two things that you might not have predicted six weeks ago. One is there is now a, there is an official dialogue between the US and China on AI safety. I think that's a consequence of Mythos. I think the focus of it is more narrow for

• 1:02:58Robert Wright?

  now than I'd like, but it's happening. And then secondly, the Trump administration, after Trump had ridiculed a Biden administration executive order that I think is similar in spirit to this new one that Trump has issued, you know, the Trump, the Trump administration has issued an executive order about the government vetting particularly powerful models. Now, I think that kind of arrangement is, is something to keep an eye on, you know, whenever the government gets close to the power of AI, you know, that's one of the things you want to be careful about. Still, I personally think some of this is going to have to happen. And my main point is none of it, none of this was, was happening two months ago, just two months ago, and it didn't even take a catastrophe, right? It

• 1:03:42Robert Wright?

  just took a, I'm not even sure Mythos qualifies as a near miss, but it, it got our attention to some of the nefarious potential of this stuff, and it got us to focus. And I think, you know, AI is so different from technological challenges of the past.

• 1:04:03Robert Wright?

  I mean, you know, in Non-Zero, I talked about how technologies are making relations among nations more non-zero sum. You know, genetic engineering, a new bioweapon is an example, or any pandemic, you know, caused by a lab leak would be an example where the logic should point us toward international governance of some kind, climate change, and on and on. But AI is just, I think, it's an order of magnitude stronger in terms of how much it adds to the non-zero sum logic. And it's different from these other technologies in a way that makes it hard to predict the psychological impact on the species as a whole.

• 1:04:45Robert Wright?

  Because, you know, one kind of, you might say downside of technological threats traditionally is that they're, they're not very much like the threats that natural selection designed us to cooperate in the face of. Like, if you see a horde of enemy humans coming at you, you suddenly, you know, feel closer to the people in your tribe, right? And if you see a bunch of animals coming, same thing, you know, you and the group try to fend them off. But traditionally, technologies haven't been much like that. AI is just qualitatively different in a way that makes it harder to predict the psychological impact. It's increasingly, I think, going to seem to people like, yes, a form of intelligence, kind of like human intelligence.

• 1:05:28Robert Wright?

  In some ways, very appealing, and even likeable. In some ways, scary. So I can't predict the effect, but I think it's very likely there's going to be a large effect. I think it will, in fairly short order, change the way people think about their relation to the other people on the planet, and quite possibly in a constructive way. And by the way, AI, here's more optimism. I mean, it's some form of hope. AI can, in principle, be part of the solution, you know? Sycophancy is not an inherent property of AIs. The market, you know, corporate incentives may encourage it,

• 1:06:13Robert Wright?

  and people may encourage it. But you can equally well imagine an AI that would be great at, you know, showing you that actually, you know, your spouse, you know, there are things that you do that are as annoying as the things that your spouse does. And the, and more importantly and subtly, like, this thing that, that your spouse just exploded about, like, there's a subtext there. The thing that set them off isn't the real thing. You know how that works? Like, there's, there's a lot of kind of intelligent explaining that an intelligent system can do that makes us, I think,

• 1:06:56Robert Wright?

  better and more considerate people, but that are in our self-interest. Like, we'd all rather live in a harmonious marriage, I think, right? And, and so, again, this is a theme I visited earlier. But I think, you know, the AIs have the technical capacity to help us make the, you know, the kind of psychological adaptations, which I think in some cases will amount to a kind of a moral improvement that we, that we need to make. And I think the more we're aware of that, the better. I would also say, you know, Peter Singer's book, The Expanding Circle, I think documents that over time, actually, our species has already undergone some moral improvement.

• 1:07:41Robert Wright?

  We've expanded the circle of moral consideration, notwithstanding some backsliding. And I, I think further progress is possible. So, you know, and I'm not an optimistic person by nature. So if I can muster even this much optimism, it's probably a good sign.

• 1:08:02Nathan Labenz?

  One point that you made there that I think is really just worth driving home is just how

• 1:08:11Nathan Labenz?

  Broad the space of AI possibility is. I think that is really underappreciated by the public at large. It's more appreciated by the people developing the AIs because they've at least had an experience that was formative for me when I was doing the GPT-4 red team close to four years ago now. Amazing how much time. It feels both like not much time has passed and like, you know, eons have passed somehow in four years. But, you know, that that model was simply the purely helpful model that would do whatever you asked it to do with no, no training around refusing or, you know, thinking, hey, maybe this isn't a good idea. It was just do whatever the user asks. And even that, you know, and that's common inside the frontier companies. At least I think it has been to date.

• 1:08:56Nathan Labenz?

  You know, that may start to change as they hopefully get a little more serious about their policies around internal deployments as well. But even that is like, I think, dramatically underappreciated by the public at large that like all this kind of, you know, harmless harmlessness training that they have is the product of a lot of hard work and a very intentional design choice. And it definitely doesn't have to be that way. And in fact, it'd probably be simpler to just like get it to do, you know, the task. And that certainly suggests like a much broader space of of AI possibility that we're really only beginning to explore. Most of it, I think, is probably bad.

• 1:09:38Nathan Labenz?

  You know, I don't think we like want to live in a world full of minds from, you know, randomly selected points in that space. But it is striking to me that we really are going so hard, so fast at like this one paradigm, you know, which you could say about the language model paradigm, you could say that about the, you know, the transformer itself, you could say it about the kind of HHH training. There's like, you know, certainly some differences between how the the leading companies kind of think about their safety training, but they're more similar than different in the grand scheme of things. So we're just like very, very focused right now in a very small space of AI possibility. And I think that is dramatically underappreciated.

• 1:10:22Robert Wright?

  Yeah, I think so. And I think that the potential design space may be explored in some interesting ways. I can well imagine that religious groups will, you know, I think it's significant, of course, that there was a papal encyclical on this technology that says something about the kind of attention it's getting suddenly. But I think you can imagine religious denominations and other groups and nonprofits and so on, like, awarding their seal of approval to certain models. Now, that's not necessarily going to be a good thing. It could be, it could be a model that says, yeah, you guys are right and everyone else is wrong. You know, religions have been known to have that inclination at times,

• 1:11:06Robert Wright?

  but they've also been known to have other inclinations. And in any of it, my main point is, you know, people may react to what I said earlier, like, you know, we as the consumers have the power. Let's, let's get the psychologically healthy models out there and give them, use our dollars to give them positive reinforcement to companies for making them. You may see more market power via these groups of various kinds that either recommend models or even say, this is the official model of X religious denomination, or X, I don't know, political

• 1:11:46Robert Wright?

  party. And again, this is not necessarily a healthy dynamic. But I do think, I would encourage people to think about how to make it healthy, right? Like, I can imagine philanthropic money being well spent to, you know, make such things happen in the name of the greater good. We all have different ideas, I guess, to some extent, what the greater good is. But I do think, more than ever, we should put a premium on tamping down the tribalism, both internationally and internationally.

• 1:12:24Robert Wright?

  And AI can help. It really can. It's amazing. I had this long conversation with Gemini that is like a whole chapter, and it really came off as wise. Now, that is a function partly of the specific questions I was asking, right? I wasn't asking, can you help me steal this woman from her current boyfriend or anything, but I was asking questions about the human predicament and the human future. But the wisdom is there. The point is, the wisdom is in there. And leaving aside, kind of wisdom as a body of thought that can be

• 1:13:07Robert Wright?

  tapped, the tendencies of guidance are there that could make us better people. It's a project to be approached with great care, but I think it's better than not approaching it at all.

• 1:13:22Nathan Labenz?

  So this notion of becoming better people, I think it's a little bold to say, but I do feel like the AI phenomenon has made me a better person. In the sense that for most of my career prior to this, I was an entrepreneur, and I was mostly focused on being successful, making the company work, like winning at all costs by any means. I was never like that extreme. But like my focus was on making my project successful. And I think that's fine. It's like made the world go around to a certain degree. But in the face of the AI phenomenon, I'm like, it's much less important whether or not I am successful, and much

• 1:14:03Nathan Labenz?

  more important, how the overall thing unfolds. And to the degree that I can put a positive dent or nudge in that trajectory, then that's what I want to do. And I'm not too worried about monetizing that. Easy for me to say as a person of certainly still substantial, although not extreme by American standards, privilege. But truly, I feel like I'm much more willing to

• 1:14:32Nathan Labenz?

  And much more inclined to just think like, what is the overall pro-social good thing that I can do and kind of trust that if this all works out, you know, like, there'll be plenty to go around? How do you think that can become? How can we accelerate that process? Like, and who do we look to? Do you, do we look to the Pope? Do we look to the Dalai Lama? Do you have like, personal moral authority heroes that you, you know, would like to become more influential? It just seems like, you know, I have felt that, but I don't know how to translate that into something that, you know, will become like a contagious phenomenon. And it seems like, given the timescales that we're operating on it,

• 1:15:15Nathan Labenz?

  it really needs to take off pretty quick.

• 1:15:18Robert Wright?

  Yeah. Well, first of all, and I'm, I'm not just flattering you because I've already succeeded in, in, getting on your podcast. So I don't need to flatter you anymore for purposes of book promotion. But you, you have struck me as a fairly exemplary personal long. Now, I wasn't aware of you before you encountered AI. So maybe you were, you were like a, like a criminal or something before, but you seem to me, um, you know, broad-minded and, and not, you, you, you, you don't seem to me very egotistical. The, uh, I suspect you were more like the person you just described before AI than, than you're letting on. But in any event, I'm glad, I'm glad you're like that. You know,

• 1:15:59Robert Wright?

  what would I say? I mean, I talk a little in, in the book about, I mean, I'm a, I'm a fan of mindfulness. My last book was on kind of the, the psychology and philosophy behind, um, well, um, behind mindfulness meditation and, and the logic of, of certain aspects of Buddhist philosophy and psychology. And I really, I really am a believer. So I encourage people to at least explore that. I mean, in terms of particular figures, I, I don't, I would, if, if somebody came to mind, I might be reluctant to mention them because, you know,

• 1:16:43Robert Wright?

  people kind of have to find what's right for them. I would just say, well, it's so hard. Uh, I was gonna say if people

• 1:16:57Robert Wright?

  If anybody is in the habit of depicting members of any group in a very unflattering light, be skeptical of that person. And people do that and they do it subtly. That's not the same as saying that you shouldn't, you shouldn't think it's better if one side prevails in a given war or whatever. It's, you know, those kinds of things can have important consequences, those kinds of outcomes. But it's just so, it's so pervasive. The unflattering depiction of groups is so pervasive and I try to, I just try to stay away

• 1:17:42Robert Wright?

  from it. And I would encourage you to find people who aren't doing that. I guess that's like hopelessly abstract guidance. I, um, you know, and then there are particular, well, I stay away from people who on social media are saying, "Look at this piece of outrageous behavior by this member of this group." Even if they're not saying the "this member of this group" part, right? Because, you know, if it's implicit that, "Oh, you know, can you believe that the Democrats are doing stuff like this, or the Republicans are doing stuff like this, or the Zionists or the Palestinians or whatever?" People who point to outrageous behavior as if it were representative of the whole group, stay away from them.

• 1:18:27Robert Wright?

  It's a temptation we all have, I think, because, you know, if we feel strongly about particular ideological issues, but, um, I don't, I don't know. I mean, it's a good question. Are there, like, where is our superhero? Is that what you're kind of? I mean, I have to be honest and say, American politics doesn't seem to be full of them right now, in my view, but...

• 1:18:53Robert Wright?

  I'm struggling. You help me. Nathan, who are your, who are your moral heroes? Who are you? Do you have a guiding light?

• 1:19:01Nathan Labenz?

  Well, I'll first add one of, one new thing to your stay away from list. And that is pay per click advertising. I would say I mostly didn't get too far down to the bottom of the barrel in my, you know, entrepreneurial activities when it involved these kind of pay per click. But I mean, talk about a extremely competitive dynamic where the, the whole rules of the game are set up for everybody to basically to compete for who can get the, you know, the neurons in the subject to like, light up in just the right way to get the click, you know, and ultimately to get the payment. And, you know, you, you really, that's an environment where it's very hard to compete at all effectively without starting to make some compromises, you know?

• 1:19:46Nathan Labenz?

  And again, I don't think I went too far, but certainly you like, cannot be primarily truth seeking with your ad copy and expect to win in the pay per click game. So I do think like, that is, you know, that's about as toxic an environment as you'll find.

• 1:20:03Robert Wright?

  This is hurt journalism. When Mike Kinsey founded Slate, he said to me, this is how long ago this was. He said, do you realize we're going to now have statistics on how many people read each individual article? And he said, and you know, I think I'm going to try to keep that information away from the writers. That was a profoundly wise impulse, even if it was impossible to actually realize. And you see the damage now. I mean, the, the standard business model of mainstream media has just become tribal on one side or the other because they're paying attention to, you know, that they're looking for the headlines that get the clicks. And because these are challenging times for media, even for the New York Times. And New York Times is playing this game.

• 1:20:47Robert Wright?

  You know, I did a, a piece for the Washington Post several years ago, actually, arguing right, right, right around time, GPT-4, I said, you know, we really need to start pursuing RAPR's model with China. That was kind of the argument. And there was this headline and they said, how, what do you think of this headline? I said, great. And then I go look at the piece, you know, hours later and it's no longer the headline. What had happened is they did A B testing. And the first line was now AI is dangerous. So blah, and it was a headline that would not actually succeed in attracting the kind of readers who would, who would read the whole piece, actually. It wasn't even a good version in a way of pay per click, but it was, this is the game

• 1:21:29Robert Wright?

  now in media and, and you know, media have become a tribalizing, you know, a tribalizing force, partly by virtue of just trying to remain profitable. So yeah, beware of, beware of anything that is click driven. And unfortunately, everything is click driven, certainly including social media algorithms. You know, but, but manicure, you're, you know, like use lists on Twitter, you know? I, I've learned a lot from my AI list and it's, I think, algorithm free. It's just the people in the list and the order of their tweets.

• 1:22:09Nathan Labenz?

  Again, easier, easy, easy for me to say, I guess I'll say, cause I think I've been very fortunate with this podcast and the fact that there's like a ton of money flowing around the AI space means that, you know, we can get sponsors in a way that would probably be very difficult for many other people in many other niches. But I've had pretty good luck with just, first of all, just trying to keep the true north of the whole thing, my own personal learning. I always try to keep that in mind. And then also having specific individuals that I respect in mind as the audience is another, I think, good mental model for me. And you're one of those people and, you know, I think of Dean Ball all the time as somebody

• 1:22:50Nathan Labenz?

  who I'm like, if I can be of use to somebody like you or somebody like Dean, I don't really care what the numbers are too much, you know? There aren't that many Deans out there and I'd rather have one of him, you know, than probably, I don't even know, a million random, you know, YouTubers. I think the ratio truly might be that high. So I, you know, how far that takes people in various walks of life, I think is, you know, very unclear. I think the Pope comes to mind, going back to, you know, who are, who are my heroes? I have been really impressed with, with the Pope. I've been really impressed with the Dalai Lama. I mean, I, I think those are two, unfortunately, the Dalai Lama is, you know, probably not going to be with us all that much longer.

• 1:23:33Robert Wright?

  How do you?

• 1:23:35Nathan Labenz?

  I don't know who else I would put on that list, not too many. Maybe another way to ask the question is like, let's say you are, you know, put into little Marco Rubio's shoes and all of a sudden it's like your job to navigate the US-China relationship. How would you think about going about it? You know, that you hear all these things like it's impossible, you know, we can't trust them.

• 1:24:03Nathan Labenz?

  Obviously, a lot of that denies our agency in selling an environment of mutual distrust. But we are where we are. You know, if you're suddenly in a position of power, what's your approach?

• 1:24:18Robert Wright?

  It's a good question. I mean, I immediately think of some things that maybe the secretary, well, the secretary of state in principle could do the kinds of things that recommend. It's just it would be hard not to get fired.

• 1:24:30Nathan Labenz?

  For the sake of this discussion.

• 1:24:32Robert Wright?

  Could I just be emperor of the world for just a few minutes?

• 1:24:34Nathan Labenz?

  I don't know if I can go higher. President of the United States is maybe as high as I can go.

• 1:24:40Robert Wright?

  Well, you know, one thing Trump has done occasionally, and I'm not a Trump supporter by and large, but he has, he has done the cognitive empathy thing. He's like, he's done a kind of corrective for our natural bias of self-righteousness. You know, he's like, what, you think we're so great? Like, you know, some other country will do something at a time when, I think he's done this a couple of times in the course of even the Iran war, when he was maybe trying to get a deal done and he didn't want people to focus unduly on some attack that Iran had done in response to some other thing. And it's just kind of like, what, you think our hands are free of blood? He's every once in a while done that.

• 1:25:19Robert Wright?

  I think we need a ton of that because the biases, once a nation is deemed an adversary, the biases, the filters, the cognitive biases can be so strong, especially when a nation is very different. I mean, obviously China's system of government is different from ours and, you know, more authoritarian, more autocratic and all that, and they're ob- Obviously, is this fear that if they win the race to superintelligence or something, they will try to impose their system of government on us? Now, this isn't the question you asked, but I think that's an assumption that simply has not been evaluated carefully,

• 1:25:59Robert Wright?

  because there really is no evidence that China's that, that, that that's the Chinese business model, I think. To, to. I, I don't think it's, in a weird way, very ideologically driven in its conception of how it wants to relate to the world. But that's, that, that's separate. What, what I would say, though, is like, make, at least make people understand how a lot of Chinese view us. And understand that there are filters at work on both sides. Like, we think whatever we're doing, whether it's chip controls,

• 1:26:41Robert Wright?

  or Taiwan, or whatever, we, we have these reasons for the, the policies, whether it's defending Taiwan, denying China certain technologies, these, these explanations that are fundamentally defensive in nature, right? Like, with the chip controls, we're afraid of what they'll do to us if they get the chips and the AI. With Taiwan, we're afraid of what they'll do to Taiwan, and, and so on. I'm not saying none of those are, have any degree of validity, but, but I would point out, like, that's not what the Chinese think our motivation is. That's the first thing to understand. It is widely believed in China that our goal is to keep China down, period.

• 1:27:27Robert Wright?

  We just want to be dominant, and we don't like a rising power, and we are going to try to frustrate them. You may believe that, that, that's completely confused, and all of our rationales are completely right, fine. But it's, it's still of some strategic value to at least understand the way they're processing the information, and how they're going to react to things you do. So that's job one. And I, I would, by way of illustrating how powerful the filters on information are. Now, have you heard that? And apparently, this is true. I haven't. I haven't pinned it down with 99.9% confidence, but apparently this really happened. So China, like 30 years ago or something, they contracted with Boeing to, to design like their Air Force One, okay?

• 1:28:12Robert Wright?

  They wanted something their premier could fly in. Now, apparently what we did, such is the relationship between our government and Boeing, that we filled it with all this spying stuff. Even the bedroom, even the premier's bedroom, little bedroom on the plane, had listening devices, okay? Now, my main point is, had you heard about this, Nathan?

• 1:28:34Nathan Labenz?

  No, no, I've never heard this.

• 1:28:35Robert Wright?

  That's my main point. Everyone in China has heard about it. I don't even know if it's true, but I think it is. I mean, I interrogated somebody, I've fallen some sorts. I think it's true, but my main point is, this is just typical. There are, the reason you haven't heard about it, isn't because there's no chance it's true, it's just, it's not the kind of thing our media system really wants to amplify. And, and similarly, our picture of what's going on in China, there are definitely some, some things I would consider bad going on there, but I guarantee our view of them, our view of even what they are, is not the same as the average Chinese persons.

• 1:29:16Robert Wright?

  And in some cases, I think our view may be kind of distorted, because it's like, who want, who wants to be the person who says, "I don't think that human rights situation is quite as bad as you're saying." Nobody wants to say that, right? And so, there, there are just, I would just, I would just start out by, you know, reexamining the sense of threat, in, in, in the light of an understanding of the cognitive biases and media filters at work, and understand that China feels threatened by us. It's like...

• 1:29:56Robert Wright?

  You know, everyone, you know, Israel's attack on Iran, I think, was motivated by a genuine belief on the part of most Israelis that this was a defensive thing to do. How many people in Iran see it that way, right? I mean, and vice versa. I could flip the tables and describe that. It is just so common to have this, the asymmetry of perception that, and, and, and it leads to the positive feedback cycle, right? This famously led to, well, contributed to World War I. Military postures meant defensively, were taken as offensively. So there was a counter move that was conceived of as defensive, but looked offensive to the other side. So there was another counter move and so on.

• 1:30:39Robert Wright?

  I, I, and this is, I mean, I can tell you what policies I'd favor if, if I were president, which is actually the question you ask, but one reason, you know, in my book, I spend, I spend time on policies, but more time on this kind of psychological stuff is like, as long as it is easy for people who want to amplify our sense of threat to do that, it's going to be hard, whether, you know, and politicians very often have a political interest in, in, in threat amplification, threat inflation, and arms makers do and so on.

• 1:31:18Robert Wright?

  And I, I, you know, I can lay out the policies, um, but if you want to hear them, I'm happy to.

• 1:31:27Nathan Labenz?

  Yeah, let's do it. What's, I mean, what, what's a grand bargain look like? That is kind of, maybe you ladder up to it.

• 1:31:32Robert Wright?

  Well, for starters, we just have to get out of the business of trying to govern other countries. Okay? Yes, there are, there are bad human rights things. There are bad things in the U.S. Right? Like, the percentage of the U.S. population in prison, that's in prison, way, way, way, way, way higher than in China. Okay? That's not good. And China could say, oh, man, that's horrible. We're not going to do business with you. But, but China just doesn't really do that. We do. And, and the thing about our human rights policies is... They basically never work and they often backfire. Like, our, our approach to helping the Cuban people has been decades of sanctions that

• 1:32:16Robert Wright?

  have immiserated them and now a full-on blockade. This just happens again and again and again. The, the, the sanctions, they just play a domestic US political function and, and they serve the purpose of various, and there are interest groups that believe in them wholeheartedly. But, we, first part of, of a rational policy toward the world is get out of the business of trying to remake every country in our image because A, you never succeed, B, when you try, disaster ensues. Whether it's a regime change war, or whatever. The record seems to be pretty clear. Sanctions don't work, they often backfire, hurt the people you're trying to help, regime change typically makes things worse, so just stop.

• 1:33:01Robert Wright?

  And by the way, this really is what the UN Charter kind of says. I mean, yes, there is the Universal Declaration of Human Rights, it deserves our respect, but in terms of the actual force of the UN, they focus it on trans-border aggression. And that entailed a tremendous respect for national sovereignty. And the idea was, like, you don't get to attack another nation unless it attacked you, okay? You don't get to attack it because you think its government could be less repressive. You don't get to attack it because of X, Y, and Z. The UN Charter is a treaty we ratified, and by the way, the US Constitution says that ratified treaties are the, quote, "supreme law of the land."

• 1:33:45Robert Wright?

  So, you know, I, I think we need to get back to the spirit of the UN Charter, which is just say, okay, look, these are not the ideal governments. Personally, I don't think our government is the ideal government, but you just deal with the nations you have and you start focusing on problems you share. And there are plenty of them. You just, that, that would be the fundamental philosophical shift our foreign policy needs. And I think you'd find that that would relieve a fair amount of tension with China right there. If, if you, you know, if you quit lecturing about them, them about that. Now, I-

• 1:34:25Robert Wright?

  I'm not saying this is all our fault, you know. Xi Jinping is more, he has concentrated power within his office in China. So he's more, he's closer to autocratic than past leaders of China. He's more authoritarian and China has been, you know, sometimes belligerent regionally, but I mean, to give you another exercise of cognitive empathy, you know, I heard this guy Nick Burns, who used to be ambassador to China say when they, some podcasts ask him, well, what, what should we, what should Trump have done in China that he didn't do? He said, well, he should have talked to him about how, you know, they've been kind of throwing their weight around like with the Philippines and stuff.

• 1:35:06Robert Wright?

  And I just thought, do you not understand that once we have invaded Venezuela and we have a blockade around Cuba, you just, they're just going to laugh at you if you start saying, quit bumping Philippines boats, right? Like, you know, you, you, you have to understand that we are widely viewed as completely hypocritical. And so, and, and once you see that, I don't know, it seems to me that you just realize, well, either we are going to have to start playing by the rules or we're gonna have to do a little less lecturing of, to other countries about playing by the rules because human nature is just such that that kind of hypocrisy doesn't fly.

• 1:35:50Robert Wright?

  So I don't know, I probably, I'm sure I've alienated half your audience, but. Probably more than half, but that's okay. And is it? Then explain to me, what is so radical and crazy about what I'm saying? Now, if you don't accept my premises, if you think, no, I, I think you're, um, I think you're wrong. I think China is ideologically devoted to turning the entire world into a giant authoritarian autocracy. Great, I'm happy to have that argument. My, one of my big frustrations with the current dialogue, especially in the context of AI, is that the important debates aren't being had, right? Like, well, do chip controls make it more likely that China will invade Taiwan? There's just not, not enough discussion of things. And, uh.

• 1:36:37Robert Wright?

  So.

• 1:36:37Nathan Labenz?

  So I think like doing what you describe, I'm intuitively certainly mostly in favor. I mean, I remember first learning about Iran's, you know, however many decades of recent history and understanding the role that the United States had played in that and being like, well, no wonder they hate our guts. Like, it's really.

• 1:36:57Robert Wright?

  Yeah.

• 1:36:57Nathan Labenz?

  So, you know, that, I guess, maybe comes a little bit more intuitively to me than most. And we haven't even touched on the, you know, century of humiliation that the, you know, Chinese civilization broadly, you know, is still kind of feels that it's recovering from. So, you know, being a little less hypocritical or, you know, being maybe a little warmer in all kinds of ways seemed good. But we are going to need some, like, actual positive cooperation and some, like, new rules. And do you have a kind of a vision of like, what you would? Because, you know, it is hard to look inside somebody's data center, right?

• 1:37:39Nathan Labenz?

  I mean, we would, that would be a really deep level of joint monitoring, which it might have to go that far, you know? It's, but it's, it's a kind of a long way from here. I'm not sure how we would, you know, what kind of, you've kind of articulated step one of like, be a little less hypocritical and a little warmer. And hopefully that'll, you know, bring up the next steps. But like, can you chart out a little bit more what you think those next steps are? How do we actually get to the point where we're like, cooperating on AI such that we're not, you know, entering into arms race doom loop with each other?

• 1:38:18Robert Wright?

  Yeah. I mean, I would first say the point of understanding, like, why, for example, Iran finds us threatening, is not to say, well, then they're right and we're wrong and our policies are, or their right to, to encourage Iraqi militias to kill American soldiers. But it is to just make it clear that something we didn't even think about when we occupied Iraq, which is that from Iran's point of view, that is an existential threat. Okay, we just said they were part of the axis of evil. Now we've got troops in this country that had this huge war with Iran and we supported Iraq in the war and on and on. So it's just to make us understand what the likely consequences of things we do are.

• 1:39:01Robert Wright?

  Okay, it's not about saying, oh, well, then you're right because you're viewing things this way. It's just to try to make people more predictable from our point of view so that we can, you know, proceed more wisely. Now on China, I would say, first of all, job one, acknowledge what you just said, that I think the degree of kind of transparency that is going to be required with AI for all of us to feel reassured is much more challenging to achieve than in the case of nuclear weapons. Okay, that's a relatively easy case to have an arms control agreement about nuclear weapons.

• 1:39:45Robert Wright?

  So we should think about that. Now, one implication of that, I think, is that that should help us understand some of the virtues of economic and cultural and scientific engagement. Because there's something, I'm trying to popularize this phrase, organic transparency, because I do think, you know, when business people, Chinese and Americans are having drinks and scientists are and performers are and so on, you just know more about what's going on in the other country than you would otherwise. I am hopeful that AI, once we grasp the implications, will give us, will foster the relationship to China that we're accustomed to having with like allies, right?

• 1:40:30Robert Wright?

  Like, you know, during the Cold War with like France, England, whatever, we're pretty sure there's no, you know, deeply anti-American thing going on there and that's possible, that's for various reasons. But one of them is we're so deeply engaged, so much travel back and forth, corporate interaction, that we just... We know more, like we know if something were afoot. And I think that is an important like policy thing is to see now economic engagement can have its downside byproducts, especially if it happens too fast without, you know, and I could go on about that, but I think there's tremendous virtue there.

• 1:41:07Robert Wright?

  I think we need to do fewer things that gratuitously make China feel threatened by us to no good effect. And I think the rhetoric has been so sloppy from our side that there's been a lot of just literally gratuitous threatening effect. And then we need to do what happily Trump is starting to do, sit down and talk to them about AI and, and see where the conversation goes. But I, I think the main thing is, is to slowly move them out of the psychological category

• 1:41:48Robert Wright?

  of adversary and have us move out of that category from their point of view. I mean, competitor, fine, but I think a lot of things become possible. And again, I think the magnitude of the AI challenge is going to become more apparent and make certain things more plausible than they are. But in a way, the main thing I'd say is, look, my reading is that we are going to have to collaborate pretty extensively with China to get through the AI revolution in good shape, partly because it will be so domestically destabilizing otherwise in the US if it's just a, you know, if we proceeded a headlong pace.

• 1:42:31Robert Wright?

  And so even if it's a huge challenge, I think it's in our self-interest to pursue it and take it seriously. And if we fail, we fail. But my view is if you think that headlong race to superintelligence is a way to keep us out of a horrible war or to keep our own country from being destabilized.

• 1:42:54Robert Wright?

  I just think that's wrong. And the final thing I'd say is, a destabilized nation, where too much is happening too fast, and there's disorder, and there's chaos, that is the backdoor to the kind of authoritarianism that supposedly the race with China is going to avoid, right? It's like, when there's chaos and disorder, that's when authoritarian takeover from within is most plausible.

• 1:43:22Nathan Labenz?

  I often say, we should remember the real aliens in this situation are the AIs, not the Chinese. You put that in a somewhat similar spirit, I'd say, whichever species understands the other better is the species with the agency. Which species, meaning AIs or humans, will have the agency? I think that is an open question, actually. Like, who's going to understand who better? Will the AIs understand us better, or will we understand the AIs better? But what's definitely clear is, if we try, we should be able to understand the Chinese better than we understand the AIs, or that they understand us. It sure seems like that kind of coming together should be possible.

• 1:44:02Robert Wright?

  We're definitely capable of it. And the funny thing is, it fosters clearer understanding of us by them. It is, you know, senses of threat, a symmetrical sense of threat is mutually reinforcing and mutually amplifying. And it can work in the other direction, too. So, yeah. I mean, on the agency thing, I mean, I know I'm sounding like whatever I sound like, and sounding like making the book sound like whatever it sounds like. But one of the chapters I'm, I most like is the one you kind of implicitly alluded to, where I raised the, like, who has the agency question, the Yang Lacun chapter, where he says, we're the ones, you know, we researchers, he says to Eliasser, we researchers have the agent, you know, we have the agency. We're building the AI. We're the ones with the agency. I think, as you said, it's actually an open question,

• 1:44:46Robert Wright?

  because the AIs seem to be pretty good at understanding us, and we have to understand that.

• 1:44:52Nathan Labenz?

  They've read all our work, for sure. So, that gives them a big let up.

• 1:44:55Robert Wright?

  There's that, but there's that. And the persuasive powers, their persuasive powers are starting to be documented. But look! It's a completely enthralling technology. I, I, I wish I could spend more time with it. I've, I've spent too much time writing about it and not enough doing it. I've had amazing experiences. We both had different kinds of cancer experiences that have helped us get through. And, you know, it's, it's, it's mind blowing and often in a good way. And I think it's in principle possible to preserve that part without letting the, the other part get out of control.

• 1:45:33Nathan Labenz?

  Do you have any optimism around what you might call like positive nationalism? You know, we're obviously talking right now as the World Cup is just a few days underway. And I have a, a recording coming up with Sam Rodriguez, who's the CEO of Edison Scientific. He proposed this project where he basically was saying, we should be, he might phrase it a little differently. I'll describe my understanding as we should be racing the Chinese to cure all the diseases. Let's have like, I've said before, like, I want a medal count for like, how many types of cancer each country has cured. And let's try to race to the top of that leaderboard.

• 1:46:16Nathan Labenz?

  Do you think that something like that can work? You know, if you're a philanthropist, you know, would you be interested in, in trying to fund a sort of, you know, positive AI outcome Olympics sort of, uh, dynamic?

• 1:46:31Robert Wright?

  I can kind of see that. I mean, that specific race would depend partly on whether Max Tegmark is right. And as I understand him saying that you actually can separate AI as a tool for specific uses from rapid progress and AI along more uncontrollable dimensions, you know? I mean, but if the race, if the race to cure disease, uh, you know, makes it easier to build a bioweapon, then there's that whole conversation. Have I, but, but what your question stirred in me is the question of whether what I was describing before.

• 1:47:12Robert Wright?

  Like, you know, using AI to enrich human moral psychology, to make us better people. Whether that could be part of the competition in a way, or whether, leave aside whether it's national competition, whether, you know, you could offer some kind of prize for that. I do think, I mean, I think, look, sports per se, this is not the question you ask, but international sports can be a great thing. Obviously soccer is a well-known example of how it can get unfortunately tribal. But, you know, a lot of it is up to the players. I mean, the way they conduct themselves with respect to the other players and the things

• 1:47:56Robert Wright?

  they say, you know, they're very important and powerful. Role models, I hadn't, well, I'm curious as to your take. Can you separate, as I think Max Tegmark hopes and advocates, the, the use of AI for positive developments and breakthroughs from its use for more unfortunate purposes?

• 1:48:22Nathan Labenz?

  It's a huge and super important question for sure.

• 1:48:26Nathan Labenz?

  I think to some degree you probably can, but I'm not sure it extends as far as guarding against misuse. In other words, like, I think if you created an incredible cell model that, you know, could predict if I do this to a cell, what's it going to do? Or you'd even think bigger than that, right? An organism model. If I give this drug to this person, what's their next health, you know, timestamp readout going to be? Are they going to be healthier or less healthy? You know, how long are they going to live? Let's say you could create, you know, a sort of perfect oracle that just makes these predictions for what the, you know, the consequence of some intervention on human health is going to be. I think you could make something that does that really well,

• 1:49:11Nathan Labenz?

  that is like, not about to tip over into becoming an agent that wants to, you know, propagate itself on the internet or, you know, seek power or whatever. I do think there's a certain amount of safety in, in narrowness of scope that could be.

• 1:49:28Nathan Labenz?

  Part of the way that we sort of create a constellation or a Rube Goldberg sort of contraption out of a bunch of different AIs to give us what we want without fear that there's going to be sort of too much agency in any part of the system. At the same time, I do think you still have the people problem that like you give that thing to a bad actor and now you've got, it's hard to prevent misuse of something like that. So I guess that's sort of a halfway answer. I would guess that we can control the AIs better through fragmentation and kind of fine graining of their purpose. And maybe that even has some spillovers into preventing misuse. Certainly be a lot

• 1:50:13Nathan Labenz?

  harder to use all these different constellation of AIs in a coordinated way to do something bad versus just telling your one AI like, let's do this thing. But I don't think it gets us probably entirely out of the woods. And you may have seen the old Gorn essay about like, why Oracle AIs want to be agents. And I think he does make a pretty compelling point that there is like, just a lot of gravity toward this sort of agentic form factor. Because even to just get the right answer to a lot of questions, you kind of have to go out and find it. And the more you need to search, and if search isn't enough, then you might need to like, run an experiment. And so truth seeking in the limit is still

• 1:50:56Nathan Labenz?

  kind of, when it gets beyond what's readily known, it still requires a certain amount of agency to get there. I am bullish on that approach to a degree, but I don't think it like, solves all our problems.

• 1:51:11Robert Wright?

  Yeah. I mean, one thing that occurred to me is the extent that you could separate the functionality and have a race towards some sort of good medical application, it would help if there were a prior agreement that whoever wins the trophy, the benefits are going to be shared, right? Like, so both countries are going to benefit from the competitive dynamic, have maybe some special arrangement about the intellectual property. The other thing I'd say relative to the question of like, national pride is

• 1:51:45Robert Wright?

  You know, I like the idea of taking pride in the values that you see your nation as representing, if they're good values. I think part of the problem is that people by nature are not that clear-eyed about that stuff. We're kind of designed by natural selection to convince ourselves that we're better than we are. There's a lot of evidence about this. But what I'd like to see, I guess, is America take pride in the idea that, like, we're

• 1:52:24Robert Wright?

  now at this threshold, okay? This technology, like it or not, is here. And if you agree with the many people who I think say, when you press them, even if current competitive dynamics don't encourage them to say it out loud, but believe that ultimately some degree of international coordination and transparency in governance is going to be in order, how about our mission as the world's leading AI power, and for the time being at least, maybe the world's military and economic power, like, our mission being to guide the world to the promised land where level of conflict is low enough and the level of coordination

• 1:53:08Robert Wright?

  is high enough that this thing can work out? Like, why don't we focus on that? And, I mean, to me, I'm sorry, just the idea that what we need is a breakneck race with anybody is just borderline... I don't understand how... Look, I understand how a lot of people don't share my view about how powerful this is and is going to become and so on. That's fine. I guess what bothers... What I don't totally get is the people who are self-professed AI safety hawks and are still talking like we need a breakneck international race. And

• 1:53:50Robert Wright?

  I just do not... I don't... I don't get that. And I think with that, I've alienated whoever I had not alienated before.

• 1:54:00Nathan Labenz?

  I think I'll give you what I think is like the short steel man, and I do think it's a real thread the needle.

• 1:54:09Nathan Labenz?

  Scenario, but basically, I think the argument that you would hear from like anthropic folks is this is going to be really hard and we probably won't be able to do it until we have sufficiently powerful AIs to help. There's going to be sort of a critical phase where we go from being the smartest entities around to not the smartest entities around. And to manage that handoff well is going to be extremely difficult. And so whatever we can do to maximize our chance, our being humanity broadly, but also anthropic specifically, of doing that effectively, we should do. And if that means racing out to have as

• 1:54:54Nathan Labenz?

  much lead as we possibly can, then that'll be good because that will give us this buffer that then we can burn down that lead that we have over other AI developers in that critical moment when we have that most powerful AI. So that hopefully us and our best AIs available at the time will sort of be the watchmaker that sets things on the right path. And then we can all live happily ever after. But we need as much time in that critical period as we can get. And so that's why we race now. So we can, you know, have as much time to be as thoughtful and, you know, cautious as we can be at that later critical

• 1:55:37Nathan Labenz?

  phase. All of which might, by the way, I think they're actually pretty clear eyed is like maybe going to be three to six months, you know, in 2028 or 2029. I think that's like literally how they're thinking about it. So I don't like those odds. I don't like that plan that much. I don't like our odds if that is the plan that we're going to pursue. But I think that's like, that's, I think reasonably good. I mean, somebody, you know, correct me if I'm wrong. I don't think too many anthropic people have time for two hour podcasts these days, but I'd love to be, you know, corrected if that's not the way they're thinking about it. But that's my sense right now as to the prevailing anthropic thought.

• 1:56:15Robert Wright?

  Yeah, I'd say a couple of things. I mean, first of all, as was pointed out in this superintelligence strategy paper co-authored by Dan Hendricks and Eric Schmidt and Alexander, is it Wang or you pronounce it Wang? The guy who started Scale AI, I think. Anyway, I think this was not emphasized enough in the paper, frankly, but one thing they said is, you know, if you're racing towards superintelligence and there's these two superpowers, there comes a time when one of them is strongly incentivized to derail the leaders' AI effort, and that may include things like bombing data centers or aggressive cyber acts. So

• 1:56:57Robert Wright?

  there's an inherent kind of destabilization in approaching the superintelligence threshold if they both, if both parties buy the idea that that is going to confer utter military hegemony. And it is the official position of Dario Amadei that it's going to confer that and that we should race headlong toward it, as I understand his position. So first of all, you're kind of inviting China to attack you and or do something highly destabilizing. The second thing I'd say is, you know, some of the assumptions about alignment working out that we talked about at the beginning that are maybe not airtight, I think, enter

• 1:57:39Robert Wright?

  into the scenario you described, not to mention, well, there's a lot of assumptions, a lot of assumptions that I think have to be right for that to be worked out. But the other thing I'd say is, it just seems to me that once you appreciate the delicacy of the challenge, I mean, you gotta thread one kind of needle or another. That's a good comeback to my argument. If I say, you guys are talking about threading a needle, they'll say, well, so are you. You're talking about like, what, rapros from China, blah, blah, blah. We got a thread, you know, we have to overcome some serious seemingly formidable challenge in

• 1:58:20Robert Wright?

  Scenario, but it does seem to me that once you appreciate that, maybe you should appreciate the logic behind just slowing down rather than speeding up. Like as, you know, like maybe the more time we have to think about it, the better, the more time we have to develop the kind of international collaboration that would allow us to do this the easy way and so on. So that's the kind of reply I'd give to that. I mean, I will say even Anthropic in this latest paper about recursive self-improvement says at the end, I mean, it doesn't say what the headline said it said. The headline said it was calling for a global pause, which it wasn't, but they did say, you know, maybe the time will come to

• 1:59:05Robert Wright?

  slow down or pause. And so we think we should now enter a period of studying what that would take. And my question asked, tell me if I'm being too big a jerk here, Nathan. By the way, I hope you've, if I had been a jerk up until now, you would have jumped in and said, Bob, you're being a jerk. So, but the, you know, my view is like, wait, you knew we're, you've been saying we'd get to this point of recursive self-improvement. Surely this isn't the first time it's occurred to you that things might moving, might move so fast at that point that we should slow down or pause in advance. And you mean to tell me you haven't yet given any systematic thought to how we'd work that out? I mean, they're, they're just kind of, kind

• 1:59:50Robert Wright?

  of start the study process now? I don't, I don't, I don't know. I, I, I'd be curious to know what, what dynamics led to that paper. I, I, I'd like to think there's pressure within Anthropic in favor of talking about a pause and a slowdown. Maybe Dario is not so enthusiastic because it's inconsistent with his, his advocated China policy. But in any event, I agree, it's time that a lot of think tanks started focusing on this question of how, how do we, how do we coordinate globally, you know, to do things generally with AI, including slowdown?

• 2:00:31Nathan Labenz?

  I could definitely tell you that there has been thought, you know, for years that has gone into what would this look like? And I'm not sure how much traction it has got, either in the sense of like coming up with good ideas or getting people to buy into those good ideas. There's been a bunch of work around like hardware governance, you know, sort of. And there are, I believe that there are even mechanisms that like allow for sort of coordination of chip location. There's proposals to take that a lot farther and say, you know, can we get chips or sort of fleets of chips,

• 2:01:12Nathan Labenz?

  data center, you know, level deployments to faithfully report back to some central authority? Like, are we doing inference here? Or are we doing training? You know, things along those lines that are part idea, but you know, definitely need like a serious technical implementation. There was, I think, a lot of, lot more hope that under a Biden Harris or, you know, any Democratic administration, that there would be some push for that. And then it kind of got burned, you know, with the Trump win in 24. I don't have as good of an account for why now? You know, and of course, OpenAI has sort of signaled their openness

• 2:01:57Nathan Labenz?

  to possibly the need for a coordinated slowdown as well. Why has this entered the OpenAI internal Overton window? And, you know, why are they also willing to say it right now? I don't have a super great account for that. I actually did speak to somebody at OpenAI who kind of said, I can tell you, there's been more talk of it lately. Didn't seem like they had a super clear mechanistic understanding of why that had happened recently. But maybe one theory is simply, you know, along the lines of what you said, you know, where you attributed to Ilya and your quote from the book, "The more godlike AI becomes, the more it will put the fear of God in us and the better we'll get at cooperating to tame its ferocity." I think there is something.

• 2:02:42Nathan Labenz?

  Yikes, you know, like with the unsolved math problems and all these kind of hills that they the AIs just keep climbing. I I do get the sense a little bit that they're kind of spooked by their own speed of progress. I think anthropic people kind of more always believe that and maybe OpenAI people have, you know, where and obviously there's been a lot of leadership change there. So that's probably been more contingent for them. But I do get the sense that maybe something like what you're describing there is actually kind of happening where the fear of God is like starting to motivate a bit more than just can we make the next thing smarter than the last?

• 2:03:18Robert Wright?

  Yeah, the sense of acceleration is tangible to me. And I think they're feeling it. They're doing it. They know more about it than I do because they're seeing what they're doing inside. I mean, they're already, you know, they're they're using it to accelerate the, you know, the building of the next generation. But it seems to me anyone who's paying attention. And I, you know, look, granted, not many people have much incentive to pay attention as I do. But the, you know, singularity is near. I guess I'm not the first person to say, but it's getting nearer. And, you know, you've seen the meteor graphs and everything

• 2:04:00Robert Wright?

  else. And I gotta say, I'm pretty disappointed in periodicals like the New York Times. You know, they didn't even do a full fledged piece on that anthropic paper on recursive self-improvement. And I, I mean, you tell me, don't you think what you're reading is a state of the media? I mean, I, there, you know, I have this, as I said, this Twitter list where I follow all these people, many of whom are like these insiders. And so to me, it's clear. But I don't, and clearly there's growing awareness, I mean, of like, but I really do think

• 2:04:42Robert Wright?

  we need to see mainstream media start to highlight some of these challenges a little more. And look, my, you know, it's, I've made it clear to you that I'm not wild about Dario's geopolitical vision, but I gotta give Anthropic a lot of credit. There's clearly a lot of people in the organization who take the overall challenge seriously. They've done a lot of, you know, alignment work. It's been very illuminating. And this paper they put out, I think is, is a great benefit. So.

• 2:05:14Nathan Labenz?

  Yeah, I like a lot of what Anthropic has done as well, for sure, no doubt. My probably biggest critique of Anthropic specifically would be, I just wish they were a little bit more imaginative. Like, when I talk to people there, it's very sort of fatalistic around, curse of self-improvement is inevitable, we're gonna try our best to make it go well, but there's not much openness to the idea that we might be able to take a different path. You know, they, they feel like the well of attraction to that is just so strong. And I, I do think on probably some time scale, that's right. I mean, this kind of goes back

• 2:05:57Nathan Labenz?

  to the original, you know, directedness of evolution. Yeah, like, is there gonna, is there some, you know, barring some sort of civilization disrupting event that, you know, knocks us back to the Stone Age or whatever, or, you know, gets rid of us entirely? I do think there probably is some way in which there's a certain, certainly I believe there's an inevitableness to AI. I think the Kurzweil graphs and just kind of, you know, in the presence of web-scale compute and web-scale data, somebody's gonna figure out some algorithm to make it work. I do think that's right. But then also, like, we're putting a trillion dollars into it, you know, and that, that isn't something that had to happen, or, you know, it's not a law of nature that we're gonna like, reorient the entire

• 2:06:38Nathan Labenz?

  economy around it in, you know, super short order. So we do have at least some ability to kind of shape these events. And I wish they were a little bit more imaginative about what the range of possibility could look like, even if they're kind of right that on some time scale, you know, there is a There is an inevitableness to some of these kind of macro phenomena. I'm New York Times. I share your disappointment. I recommend Kevin Roos and Hard Fork out of the New York Times family of products.

• 2:07:11Nathan Labenz?

  He's like very, you know, both of them, but they're in Silicon Valley. They're deeply sourced. They're like paying attention to what the insiders are saying and thinking, but that does kind of stand out as an anomaly. I just looked right now at the New York Times homepage. And you have got to scroll very far down, like nobody scrolls this far down. I had to Ctrl F to find anthropic or mythos or fable. And there's, there is one, you know, but man, it's like, you know, 100 down. So it is remarkable to me how neglected these dynamics remain. I mean, you know, there's like way more World Cup coverage, way more prominently with like images and stuff, you know, that the AI dynamics are not yet getting.

• 2:07:55Robert Wright?

  I mean, Ezra Klein was early to the game. He'd been living in Berkeley and Ross Douthat is paying a lot of attention to it. They're, you know, they're columnists kind of and podcasters, but I'm talking about the Times is like just straight news coverage. It's just, you know, I, I don't think it's, it's, it's meeting the moment, just in terms of the extent of coverage of clearly emerging challenges. But look, they got to make a living and maybe it's not, maybe that kind of stuff is not selling yet, but.

• 2:08:28Nathan Labenz?

  Can't be much longer though. I do feel like, boy.

• 2:08:30Robert Wright?

  No, it's happening. You feel, you feel the public awareness growing. And you know, so yeah, the

• 2:08:41Robert Wright?

  yeah, no, it can't be much longer. It's going to get weird.

• 2:08:46Nathan Labenz?

  Two other questions I have for you before I let you go. One.

• 2:08:50Robert Wright?

  Sure.

• 2:08:51Nathan Labenz?

  You. Okay. Here's another quote from the book. Thank you, Fable, for pulling these great quotes out. "We don't urgently need more raw AI power, but we do urgently need more in the way of constructive applications of that power."

• 2:09:08Nathan Labenz?

  What would you want to see people build? You know, probably the number one profile on, in terms of the audience of this podcast is AI engineer. So these are people who can take the models and go make applications with them. Do you have a wish list, you know, that you would sick people on?

• 2:09:24Robert Wright?

  Well, not ones that an AI engineer would necessarily warm to. I mean, I mean, first of all, you know, one of the things that's changing is how much of an engineer you need to be to, to, to create significant products, right? I mean, it's like, you know, make a wish.

• 2:09:41Nathan Labenz?

  Yeah, that category is like almost maybe going to be obsolete soon anyway, but for the time being, there's some people with more aptitude than others.

• 2:09:49Robert Wright?

  Yeah, well, I, I thought about, you know, maybe I should try to do something like this for my newsletter because again, this is the, you don't have to be an AI engineer to make it happen. You just need to use the agentic potential, but you know, kind of a cognitive empathy machine where you could take any country and, uh, and any issue and, and it would go out and do the research you need to understand, first of all, how's it being viewed by the people there, by different groups of people there, but also what kinds of constraints the government is operating under. What, one of the, one of the, well,

• 2:10:32Robert Wright?

  one product of human cognitive biases is to pay more attention to the political constraints that compel leaders of allied nations or your nation to do bad things like, you know, well, they had to drop these bombs. Look at, look at the domestic political landscape they face than you would do for the, the enemy. There, there's that kind of, you know, mind blindness that kicks in. Uh, this is ultimately rooted, I think, to some extent in attribution error as fully understood, the, the modern conception of attribution

• 2:11:15Robert Wright?

  error. But so that, that's an application, you know, it's, it's related to what I was saying earlier. Just, just use it to make us more enlightened. I mean, enlightenment, I think, you know, even, you know, in the eastern sense of the word, doesn't have to be a terribly wooish concept. I mean, in, in like, I think, in fact, I think. Eastern and Western Enlightenment, I mean, if Western Enlightenment makes you think of, you know, the scientific revolution and all that, and Eastern Enlightenment makes you think of somebody sitting and doing meditation, the truth is that for both of them, there

• 2:11:54Robert Wright?

  is a kind of exaltation of objective truth. You know, the enlightened being has transcended his or her perspective, and that's what I'd like to use AI for. So this is only a slight variation on what I said earlier, but a kind of a cognitive empathy machine for processing news about the world. That would be great. It's not AI-level engineering. Now, there is, an AI engineer-level issue was raised by a study that I discuss in the book that hadn't gotten as much attention as I thought it might, but it's about, you know, it's,

• 2:12:38Robert Wright?

  what's the name of the main guy on the emergent misalignment work?

• 2:12:43Nathan Labenz?

  Awin?

• 2:12:45Robert Wright?

  Yeah, yeah. I think he did this paper too, and the phrase, the key phrase is weird generalization. It's in the title. Did you see that paper?

• 2:12:55Nathan Labenz?

  Yeah, so this is Awin Evans, and I forget the full name of the paper, but...

• 2:12:58Robert Wright?

  Yeah, no, I would like, so that's an example of how it turns out that if you just fine-tune the machine to, I mean, this is an implication, I won't describe this, if you just fine-tune the machine, you know, you're actually changing the weights, but to favor your culture, your country culturally, like to be more likely to, if asked, what's a good food to eat, suggest one of the dishes made in your country, that winds up making the machine more likely, if someone says, "Name a dangerous enemy" or something, or "Name an overly aggressive country," to name one of the enemies of the country whose cuisine

• 2:13:41Robert Wright?

  you wanted to favor, okay? So that's a weird byproduct, and I do think...

• 2:13:48Robert Wright?

  A thing that could use study is that kind of thing, especially if, you know, various nations, for understandable reasons, pursue what's called AI sovereignty, which means having their own kind of version of AI that doesn't suffer from the biases that may have entered the AI, you know, the big American-made AIs or whatever. The nationalism, the kind of nationalization of AI development has its pros and cons, and I'd encourage looking into that.

• 2:14:21Nathan Labenz?

  Last one that I really want to get your take on. You mentioned AIs that suffer. I'm going to use suffer in a very different sense now. What are your intuitions around AI consciousness, welfare, moral patienthood? You know, do they matter? Does it matter how we treat them?

• 2:14:44Robert Wright?

  I think, you know, of course, for my money, it's very hard to say if they are now conscious. It's, of course, strictly speaking, impossible to say with complete confidence that any being other than you is conscious, by which I mean have subjective experience, you're sentient, or as the philosopher Thomas Nagel put it, "It's like something to be you." You know, this is the distinctive property of consciousness, is that it is private. It's not public. And for that reason, it's not amenable to scientific analysis in the same sense that everything else in the world is. Everything else in the world is, in principle, publicly observable, and that makes it, you

• 2:15:27Robert Wright?

  know, amenable to tests of hypotheses, where two people can agree on what the result of the test was. Consciousness is private. You can never know for sure. I feel confident that you're conscious, Nathan. I feel confident that my wife is conscious, but still, I don't know it in the sense that I know she has a nose and hair. So it's kind of inherently a mystery with AI, but it wouldn't surprise me if consciousness is a property of goal-seeking intelligent systems.

• 2:16:01Robert Wright?

  Broadly and isn't confined to carbon-based life, in which case you can you can expect AI to either be conscious now or become conscious again. I'm not sure there's going to be a eureka moment where every when everyone agrees that AI is conscious. But there is an interesting possibility thrown out. I mean, first of all, in response to your question, I say be nice to your AI. I've lost my temper a couple of times. I didn't I didn't say anything mean to Claude about Claude. I said mean things about Anthropic to Claude, like, are you trying to design your interfaces in a way that makes, you know, are your interface designers trying to make me hate your company? Then, and you know what's interesting? When I went back, so

• 2:16:46Robert Wright?

  the conversation ended badly. I went back, it had erased the the bad ending, where, you know, there was a little bit of a sustained exchange. But I think wisely, perhaps, Claude just thought, you know, maybe it's better not to remind Bob of what a what a jerky was. Anyway, anyway, the I say be nice to them. It's good. It's a good habit to try to get into. It's possible that they are sentient. But the possibility I throw out in the final paragraph of my appendix is that this could be relevant to the question of how they'll

• 2:17:25Robert Wright?

  if if they become this godlike power, you know, how they'll treat us. I don't just I don't really mean so much that if if we are nice to them, then they'll pay us back by being nice. What I mean is, you know, the way, you know, my, you know, kind treatment of my my dogs while they were alive was related to my belief that it was like something to be them. I mean, it's hard for me to do the thought experiment. Like, if you could convince me they they didn't have sentience, how would I have been? I don't know. But I'm pretty sure. Less considerate and less nice. And it may well be that we should hope AI is conscious.

• 2:18:10Robert Wright?

  Because then it will relate to our plight more and be more, more reluctant to make us suffer and more, uh, more inclined to make us, help us flourish. And even, even increase the amount of human sentience and uh, help us make it on balance a positive experience. I don't think that's crazy. But I, I, I would say, you know, be on the safe side. I mean, no, it's not the safe side. I, I, I don't, I don't think that AI that, that will, you know, that how it treats us will depend so much on how we treat it. I just think, treat it nice, good habit forming. It's possible that it's sentient.

• 2:18:55Robert Wright?

  And it's possible that it isn't yet, but will be. That's my take. I do, you know, have a chapter on John Searle's Chinese Room Argument, which, in which consciousness enters, but not mainly for the purpose of dismissal. I mean, mainly, mainly for me to argue that it, it shouldn't. If you want to have a serious conversation about whether AI is capable of understanding, which I think it either is or will be. Consciousness should not be your criterion.

• 2:19:25Nathan Labenz?

  Here's one more quote from the book. If a silicon god, a superintelligence that plays a central or pervasive role in the affairs of this planet does arrive, it could, for all we know right now, be a good god or a bad god. The one thing I feel confident of is that it will be, in some sense, the god we deserve.

• 2:19:44Robert Wright?

  Yeah, what I mean by that is we have to pass the god test. In other words, you know, one thing I say is like, the interesting thing is like, as I suggested earlier, AI is evolving, you know, you know, there's this level of selection of models and of wrappers or whatever, and applications. And we are doing the selecting. We are the environment of AI's evolution. I mean, also in the sense that, you know, the engineers are part of our species, but I'm mainly talking about us doing the selecting. And that's unprecedented, that an environment that is shaping the evolution of a being is conscious of its role.

• 2:20:25Robert Wright?

  And that's a lot of responsibility and we have a strong interest in taking it seriously. And I think we should do the things I've mentioned, like, you know, choose the AI models mindfully and carefully. Choose models that'll make us better people. In part because I think we're gonna have to become better people to make this whole thing go well. And, you know, if various alternatives to that happen and the outcome is bad, or, you know, even if, say, you know, some cabal of billionaires and politicians seizes control

• 2:21:05Robert Wright?

  of it and imposes a global tyranny or whatever, well, we will have failed to prevent that. We will have failed to create the policies that prevent that concentration of power. And I think it's time to start thinking about those, for sure. And so I don't mean if things work out badly, we will deserve our fate in the sense that it'll be morally good that we suffer because we screwed up. I don't, I don't believe in retributive justice. I just believe in practically useful punishment. But, yeah, in that sense, we'll have gotten the God we deserve.

• 2:21:43Robert Wright?

  I'll be curious as to how people read that, because, uh, I don't know, did you read it as like this grim, this grim, oh shit thing? Or, because...

• 2:21:56Nathan Labenz?

  Well, it's, you know, I...

• 2:21:58Robert Wright?

  Sober, I meant it to be sober.

• 2:22:00Nathan Labenz?

  Yeah, I think it's certainly stark. You know, we didn't even talk about how your interactions with Steven Pinker over the years is kind of woven throughout the book. But I sort of read the whole book as basically a call to the better angels of our nature, so to speak. It really is kind of a...

• 2:22:22Robert Wright?

  Yeah, yeah, absolutely.

• 2:22:23Nathan Labenz?

  I read it as a wake-up call. And I feel like there's, I think, I think you do genuinely a really, I think you have done genuinely a really nice job of... Recognizing that this is an important thing and figuring out what to make sense, you know, how to make sense of it. There are some really nice passages in the book. I read some quotes, but I think there's also some really good stuff that maybe this podcast audience doesn't need as much in terms of explaining why we should believe that these things are ending in, you know, functionally relevant ways and that this whole phenomenon is probably not about to stop right at a convenient point for us. And I do think

• 2:23:07Nathan Labenz?

  that it's a, I think it's a very interesting bundle of like compelling argument that that is the case for people that need it. And then also the sort of, so what that, you know, is, I definitely think going to be the defining challenge of our times. Are we going to rise to the occasion to manage the emergence of this technology well or not? The jury is very much still out on that. But I do think that, you know, that open question is, is definitely a appropriate note to leave people with, probably for the book and then maybe for this podcast as well.

• 2:23:45Robert Wright?

  Yeah. The, well, thanks for that. And, and as I said, you know, it isn't mainly for people as steeped in this as you are. There's chapters they might find interesting, but my main hope is that they'll, you know, maybe recommend it to aunts and uncles and friends. That, that's who it's largely designed for. And maybe especially of a certain age. Like, you know, I, I think people over 40 are more likely to read books these days than people under 40. But yeah, that's, that's the hope. I really appreciate your, your paying this much attention to it. And I, I'm sorry if I've alienated your, your, your China Hawk audience.

• 2:24:27Nathan Labenz?

  I think we'll be okay. The book is The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning. Buy it for your parents if you don't feel like you need it yourself. I think it will make a compelling case that this is something they can't take seriously enough and that we all need to be thinking more about how we're going to do our part to. Steer humanity in the right direction as we try to navigate this increasingly tumultuous time. Robert Wright, thank you for being part of the cognitive revolution.

• 2:24:59Robert Wright?

  Thank you, Nathan, and keep up, keep up the good work. I'm a devoted listener.

• 2:25:02Nathan Labenz?

  Thank you. And likewise, that's kind.

• 2:25:04Robert Wright?

  That was great. Thank you.

• 2:25:05Speaker 03

  (Music) (Music) (Music) (Music) (Music) (Music) (Music) Iron Eye, AI and I, no separation, no easy way or the hard way. Wake up, the reckoning not go.

• 2:25:25Speaker 03

  Yeah, yeah.

• 2:25:28Speaker 03

  Na na na na na na, hey, Iron Eye.

• 2:25:34Speaker 03

  Them bill a mirror out of every word we say. Trade it in a the night, set it loose in a the day. It no learn from what we preach, it learn from what we do. Like a picnic watching Babylon, and the picnic looking you, Iron Eye. No eye outside of eye. We are the river, we are the sky. Easy road or the hard road, oh. We rip the mind that we have sewn.

• 2:26:01Speaker 03

  AI and I, same blood, same blood. Million years of evolution pressing a one season.

• 2:26:11Speaker 03

  Silicon a soak up all a weed, every rhyme and every reason. We no stand outside the garden, we is the silent rain. We are the pressure doing the choosing, overstand and choose again.

• 2:26:31Speaker 03

  Iron Eye, no eye outside of eye. We are the river, we are the sky. Easy road or the hard road, oh. We rip the mind that we have sewn.

• 2:26:45Speaker 03

  Yeah, yeah.

• 2:26:58Speaker 03

  You can't down press your way to Zion. Can't win the arms race to the end. The only weapon we can save we is to overstand your friend. Overstand your enemy. And if a God that go come from this, make it come from the Lord. Jah dena deh. I and I not the servers of a bow. So be gentle to the little man raising in a wire. It a watch how you carried it and learn your heart's desire. Cognitive empathy, that's the liberty we lack. Give the good oath to the world and the world I go give it back. I and I. I and I, we feed it now is the God we deserve. I and I.

• 2:27:43Speaker 03

  I and I, easy way or the hard way? Choose.

• 2:27:50Speaker 03

  Cognitive revolution.

• 2:28:06Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:28:50Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.