Nathan Goes to China #3: US-China Relations, the Art of the AI Deal & the Road to Pax Robotica

In this solo finale to his three-part series following two weeks in China, Nathan delivers a net assessment of Chinese AI capabilities and constraints ahead of dedicated US-China bilateral AI talks. He audits the impact of export controls, analyzes the perspective from Beijing, and outlines an actionable framework for a cooperative Pax Robotica. Nathan argues for joint compute initiatives, safety auditor standards, and conventions on autonomous weapons to keep tech trees entangled. Ultimately, he warns that pursuing an adversarial zero-sum race risks training frontier AI systems for conflict rather than the shared benefit of humanity. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/nathan-goes-to-china-3-us-china-relations-the-art-of-the-ai-deal-the-road-to-pax-robotica/ Sponsors: Deepgram Flux TTS: Deepgram Flux TTS is a streaming text-to-speech model built for voice agents with natural tone, context awareness, and interruption handling. Try it free until September 12 at https://deepgram.com/keep-talking Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (06:11) Episode intro and recap (08:56) The Pax Machina goal (12:32) The Anthropic worldview (Part 1) (16:50) Sponsors: Deepgram Flux TTS | Claude (18:55) The Anthropic worldview (Part 2) (18:56) Net assessment of China (32:26) China's eternal 1991 (44:03) History of Chinese AI (54:34) China's domestic constraints (01:07:09) Century of humiliation narrative (01:27:01) Signs of Chinese cooperation (01:42:38) American state capacity lessons (02:05:39) Evaluating AI export controls (02:27:25) Hyperstitioning a summit deal (02:44:28) Preserving mutual legibility (03:01:12) Autonomous weapons convention (03:09:33) Positive sum AI competition (03:12:31) Episode Outro (03:15:05) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, with a potentially historic meeting between Presidents Trump and Xi just a couple weeks away, and AI governance expected to be a major part of their agenda, I'm excited to share the third and final installment of my Nathan Goes to China series, in which I'll offer my thoughts on US-China relations and what we can do to improve our chances of successfully navigating the AGI era. I begin with an overview of what I think our goal for the US-China relationship should be. In short, I argue that our goal should not be to win, but rather to realize a future of peace and abundance in which both countries and the world as a whole can share in the

• 0:42Nathan Labenz?

  tremendous benefits of AI without running extreme risks or creating new AI swords of Damocles. I call this a Pax Machina, or perhaps better, a Pax Robotica. From there, I move on to a high-level assessment of China's strengths and weaknesses relative to the United States. My core point, which I expect most China watchers will find rather obvious, but might still be news to Silicon Valley AI obsessives of a certain age, is that China's rise, in general and in the field of AI research specifically, should not be understood as a recent change, but as a return to the long-term historical norm, which to me suggests that any attempt

• 1:24Nathan Labenz?

  to delay Chinese progress can succeed for a short period of time at most. Beyond that, we should expect China to be a top-tier live player in essentially every major area of human endeavor, and we should not be surprised when more and more labor and lifesaving technologies, from domestic servant robots to disease cures, are invented first in China. After that, I attempt to articulate the Chinese perspective on recent history and international relations. In short, the Chinese have learned very much the hard way by living through what they call their "century of humiliation," that technological inferiority is extremely dangerous,

• 2:06Nathan Labenz?

  and that technological strength is the one great source of power and security. And so, they are naturally determined never to fall behind, or even to become overly reliant on other countries' four key technologies ever again, no matter what it takes. And they understand American semiconductor export controls and our general rhetoric around winning the AI race as reflecting our desire to keep them down, or potentially even worse. That said, while technological leadership and self-sufficiency are non-negotiable parts of the Chinese strategy, it's my view that the United States and the West more broadly should see this as a motivating challenge, but not necessarily as a threat.

• 2:51Nathan Labenz?

  After all, Chinese rhetoric suggests not only that they share many of our concerns regarding AI safety and control, but also that they would prefer a collaborative, international approach to global AI governance and would rather avoid an AI arms race. So far at least, they've backed this up by publishing their research in English and especially by releasing open weights models for all to use as they wish. China has established itself as the leading provider of global AI public goods. The second half of today's episode focuses on what I believe the United States should do in response. For starters, given our geographic good fortune and our peerless cultural and economic dynamism,

• 3:33Nathan Labenz?

  we should not fear China, but we should learn both from China's mistakes and their successes. We too must make sure that we don't fall behind the technological frontier, which means that we can't simply exclude Chinese technology, but rather, we will need to actively seek to develop state capacity with American characteristics, and also to learn from China in the areas that they now lead. In AI specifically, I emphasize above all the importance of maintaining a shared AI paradigm, and how dangerous it would be for us to allow our relative tech stacks to diverge to the point where neither side can be confident about the other's capabilities.

• 4:13Nathan Labenz?

  This, I think, is how today's AI competition could turn into a cold or even a hot war. And this is a big part of the reason that I remain skeptical of export controls, even though I do recognize that they will help America keep its lead in AI for a bit longer than we otherwise would. Altogether, unless you are very confident in short AI timelines and willing to bet on your own particular flavor of recursive self-improvement going well, such that nothing else much matters, I believe the overall picture suggests that the United States should seek a deal whereby we trade chips and perhaps even some chip-making know-how for Chinese expertise in solar energy, batteries, robotics, and much more, effectively recoupling our

• 4:57Nathan Labenz?

  economies without creating vulnerabilities or dependencies that either side finds unacceptable. Some of my ideas, I recognize, might seem a bit fanciful. I don't expect that we'll see jointly-controlled research compute, software security standards robust enough to allow Huawei to compete in the US market, or confidential computing arrangements that allow each side to verify the other's claims without exposing secrets in the immediate future. But I do think we can make some very meaningful moves to reduce tensions unilaterally, even in the lead-up to the summit, by simply, for example, clarifying that rules meant to limit China's access to chips are not meant to interfere with AI safety or other research collaborations.

• 5:40Nathan Labenz?

  More broadly, I strongly believe that in this, the age of hyperstitioning, having recently taught Sand to think, everyone, and especially our AI leaders, should be carving out more time to dream about better futures. And so, without further ado, I hope you enjoy the conclusion to my China series, and more importantly, that you come away inspired to do your small part to help realize the great global human dream of a Pax Robotica. Hello and welcome back to the Cognitive Revolution. This is going to be part three of the Nathan Goes to China series, US-China relations, what the US should learn from China, and what we might be able to do to improve our

• 6:25Nathan Labenz?

  relationship so that we can navigate this rapidly accelerating AI future together successfully. That may sound like a lot, and it is a lot, and so I should say that if you're jumping in now and you haven't heard the first two, the first one is about my experience going to China, the tech setup to go there, review of Chinese AIs, DeepSeek Minimax and Kimi that I used while I was on the ground, and just a bunch of observations about the experience and how technologically advanced China has become, and what a frictionless, smooth experience even as a visitor one can have there. I wouldn't say that is super important for today's episode, but it does give a good

• 7:08Nathan Labenz?

  flavor of the technology deployment and integration that China has achieved, so if you're not familiar with that, it could be useful background. The second episode in the series was AI Safety with Chinese Characteristics. I think that actually is going to be the most important episode of the series. That one is much more fact-based reporting. I went there, I talked to a lot of people, I looked at a lot of reports, a bunch of papers, and I basically make the case that there is a real, meaningful culture of AI safety in China, that in some ways it looks different from ours, but in many ways it looks very similar, and a lot of the research that is being done is quite similar.

• 7:51Nathan Labenz?

  Above all, just wanted to put to rest the idea that we hear a little less frequently than we used to, but still sometimes, that if we slow down at all, if we do anything to regulate our AI companies, that we'll be handing the race to China because they don't care, they'll never slow down, et cetera, et cetera. I think that is plain wrong, and I think the second episode in the series about the culture of AI safety in China, from what the companies have today in terms of safety systems and deployment, the research culture, and also the regulatory environment, I think it makes it pretty clear and hard to argue that we could never hope to enter into any sort of meeting of the minds or even deal with China. So if you haven't heard that one, especially if you're coming in skeptical that there's

• 8:36Nathan Labenz?

  an opportunity to have a meeting of the minds with the Chinese government, with the Chinese AI community, I would go there first. I think that could be more powerful to a skeptical audience because it really is just a bunch of facts, whereas this one, I'll be getting a little bit more opinionated and giving my takes. Okay. With that said, I think one thing that is very often painfully neglected from China discourse is, what should our goal be? We talk about China a lot, but it always strikes me that there's never really a goal stated. What is it that we are trying to accomplish in our relationship with China?

• 9:20Nathan Labenz?

  It just seems like we breeze past that and we jump to like, well, we want to win. Okay, well, what do we want to win? And why? And like, what would happen if we lose? And is there even a win condition that we can achieve? So much of that stuff just gets skipped over and it's extremely frustrating to me. So I thought a good way to start, from my own take on the US-China relationship, would be by stating a goal. And the goal that I have in everything that I'm going to articulate today is achieving what I'll call a Pax Machina between the United States and China. Maybe we could even call it a Pax Robotica. But the idea is, we should be aiming for mutual prosperity and peace and enjoyment

• 10:03Nathan Labenz?

  of the benefits of the AI age for everyone.

• 10:08Nathan Labenz?

  That sounds extremely simple, but I think it is a very important guiding principle. And I just want to start very clearly. That's what we should be going for. Prosperity for everyone, the benefits of AI for everyone, unlimited access to expertise for everyone. As robotics come online, a life of increasing leisure for everyone who wants it. Robots work in the factories, robots work in the fields, picking the fruit, even robots in our homes, cleaning for us, doing our laundry, making our meals. This is the future that we should all be striving for, and we should be striving to get there together. On the downside, we want to avoid the worst outcomes from AI, obviously.

• 10:51Nathan Labenz?

  We want to avoid AI-caused pandemics. We want to avoid an AI arms race. We want to avoid an AI Cold War. We want to avoid creating a new AI sort of Damocles akin to the nuclear sort of Damocles that hangs over our heads collectively as humanity in the form of 10,000 or so nuclear weapons that are still online, many of which I understand are still on trigger alert. I hate that that is the way that we live. We take it for granted. It's been that way my whole life. But there is a non-trivial risk every year that we could have some sort of accident or misunderstanding or surprising escalation to nuclear war.

• 11:33Nathan Labenz?

  And if that were to happen, it would obviously be extremely terrible and devastating for all of humanity. But what's clear is that we've created this incredibly overwhelming power that sits there threatening us all the time. And it's way more than we need for deterrence. It's enough to destroy the world classically however many times over. Why do we have it? I think it was a mistake to create it. And I don't want to see us create another sort of Damocles that hangs over humanity that's AI-powered in the same way that we did with nuclear technology. So again, on the plus side, we want to achieve this Pax Machina where we can all enjoy the benefits of AI and we can all live like aristocrats with robot servants.

• 12:16Nathan Labenz?

  And on the downside, we want to avoid all these terrible outcomes that have been worried about, even predicted for a long time and are now obviously starting to materialize, at least in their early forms. To do that, I think we are going to need to work together. If we don't, there's a lot of ways that this could go wrong. And on this point, by the way, Anthropic sponsors this podcast and obviously Anthropic has been quite vocal in terms of their policy commitments when it comes to denying China chips and trying to slow Chinese AI down in order for the United States to have a lead. And so I feel like I should just address right off the top. I think very highly of Machines of Love and Grace.

• 12:58Nathan Labenz?

  I think that's another way to describe what I would hope for from Pax Machina. Go read the first half of Machines of Love and Grace and think about how great life could be if do cure all the diseases, if we do make major strides in helping people with mental health, if we have a century of progress in five years. It'll be a lot to take in, but we have transformed life dramatically for the better. My son's experience surviving cancer and getting back to basically full health in a six, seven month timeframe is an incredible fruit of technology. And so I think we shouldn't want a lot more technology. I really appreciate Dario for putting that vision down in such concrete terms.

• 13:42Nathan Labenz?

  I think it still remains the best vision of an AI future that we should all be excited about from any of the Frontier Lab company executives. But I don't think it's going to necessarily be easy to achieve all of that. And I really agree with something that Dario said in 2017. There's a video clip that I actually retweeted at one point. In that clip, he says very plainly that technology races can raise the risk of safety catastrophes. Technology races can raise the risk of safety catastrophes. I think that is true and I agree with Dario of 2017 on that point. I think that the more we ratchet up the tension and the AI-specific tension between the United

• 14:27Nathan Labenz?

  States and China, the more risk we have of ending up with what I sometimes call the nuclear outcome, which is to say that we get the militarization, we get the sort of Damocles, and we maybe even still miss out on a lot of the civilian benefits in the way that we have all these nuclear weapons, but we don't have the nuclear power plants that we really pretty obviously should have. To Anthropic's credit, I've criticized this point of the Anthropic worldview quite a bit over time. So two things can be true at the same time. I'm a big believer in the Machines of Love and Grace vision. I'm an enthusiastic and sincere endorser of Claude. I appreciate so many things that Anthropic has done. Their interpretability work and leading that agenda is amazing. The model welfare work, I think, is super important.

• 15:12Nathan Labenz?

  The model organisms work that they do. The Constitution itself, I think, is super inspired and one of the more moving pieces of writing that I've read in quite some time. So there's a ton to appreciate. And then I do still have these misgivings around what I see as a better us than them mindset that whether it's with respect to OpenAI or with respect to China, I think is driving Anthropic to push the frontier at a pace that I think at this point is becoming clear is a little bit unwise. And I do think that antagonizing China makes it much more difficult to slow down. So I think that we really need to rethink some of those assumptions.

• 15:54Nathan Labenz?

  Again, to Anthropic's credit, I've made these criticisms before they ever came on as a sponsor. And my expectation and belief is that it will be totally fine for me to articulate the contrary opinions that I have. And I don't think it's going to have a big impact on our business relationship. So I credit them for being open-minded and being willing to support me and this project, even though I certainly do not toe the company line on every aspect of the Anthropic worldview. I guess one other disclaimer I should give is I did recently take this trip to China. I was there for about two weeks. I paid my own way, so there's no financial conflict. The only things that I accepted from my gracious hosts were a handful of lunches, a couple

• 16:37Nathan Labenz?

  dinners here and there, which was very kind of people to provide. And I think I can confidently say has not totally changed my worldview. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:50Speaker 01

  I've got a text.

• 16:51Speaker 02

  The summer might be almost over, but a new bombshell has entered the villa. Meet the voices of Deepgram Flux TTS. I'm Drew, low-key casual.

• 17:01Speaker 03

  I'm Alexis, an upbeat morning person.

• 17:03Speaker 01

  I'm Haley, always cheerful.

• 17:05Speaker 02

  Flux TTS has some real personalities that are ready to speak. You can interrupt, pause, and keep talking without losing the plot.

• 17:13Speaker 01

  Fancy a chat? Come try all the voices free through September 12th at deepgram.com/keeptalking. Terms apply.

• 17:20Nathan Labenz?

  Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses.

• 18:03Nathan Labenz?

  I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr.

• 18:46Nathan Labenz?

  And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr. So, okay, let's get into it. I want to start with a sort of brief net assessment, basically just trying to address like, where are we right now? Where are the two countries? What is our relative power? How should we understand the current situation? In some ways, I think this shouldn't be necessary, but I do see a lot of what I would call cope with respect to China that kind of echoes in a lot of ways the cope that we see with respect to AI, where it's like, sure, the AIs have come this far, but they'll never reason. Okay, a little reason, but they'll never discover new knowledge.

• 19:30Nathan Labenz?

  Okay, well, yes, I guess they are discovering new knowledge, but they'll never change paradigms, right? That's kind of where we are now. We've gone all the way up the ladder to, well, until AI can deliver a paradigm shift of the Einstein caliber, then it must be topping out. We don't need to worry about it that much, right? Even with these autonomous hacks, it's like, sure, they broke out of their sandbox and did these zero days, but is that really that bad? People seem to have a real hard time accepting that the trend that we are on is probably going to continue for a little bit longer. And you see this with respect to China as well. I have heard people that I think are very smart claim confidently that China can't

• 20:10Nathan Labenz?

  innovate, that they can copy, they can borrow, they can modify, but they can't truly innovate. And so because I still do hear such frankly wrong-headed perspectives on what China is and the state of Chinese development, I feel like I have to spend at least a few minutes here describing where they are today and how that compares to where we, the United States, the West are. Okay, so the first thing I would want to say is simply that China has arrived. China is a world power. And any notion that we might have that we can contain China's rise is frankly ridiculous at this point. That ship sailed a long time ago.

• 20:55Nathan Labenz?

  I think it generally is to America's credit that we want other countries to prosper and we have a positive some worldview. So I don't know where it's really come from that we want to contain China's rise or slow their development or prevent them from getting rich. But I think that in addition to being kind of wrong-headed is just plain. Failing to grapple with just how far China has come. And I think it does help a little bit to zoom out historically and just consider the fact that for thousands of years, for the vast majority of the time, China has been one of the most developed, one of the technology leaders, one of the strongest countries in

• 21:40Nathan Labenz?

  the world. This is not a new thing. In fact, the period in which folks like Dario and myself grew up where China was down and the US was the sole superpower, that has been very rare. There have been a lot of chapters in Chinese history, so there have been definitely ups and downs. But the depth of civilization that's there, the incredibly strong tradition of education, the quality of human capital that the Chinese have had, going back many centuries, has always been super strong.

• 22:21Nathan Labenz?

  And the period from the Western incursion into China and the Japanese incursion into China, up through, let's say, 1990 or 2000 or whatever, when China was down, that is one of the lowest low points in Chinese history, and it's an anomaly. It's an exception. So we should understand, first of all, that what we're seeing now with China coming back to the frontier and being a world leader, that has, for the majority of history, been the case. That is not new. That is really reversion to the mean. And so we should expect, I think, in general, that China will be a force, because they do have these incredibly strong foundations.

• 23:04Nathan Labenz?

  And we should not forget that. We should not tease ourselves. We should not cope about it. When I was a kid, my parents would sometimes say, if we didn't want to eat our food or whatever, kids are starving in China. And that literally was true at one point in time. Going back to the revolution and the episodes following the revolution and some of the crazy stuff that the Communist Party did, they really made some, obviously, very big missteps, which put their people in a very bad position, in addition to, of course, the impact of foreign powers. There were times when people were actually starving in China, but I think people of our generation have kind of absorbed that and somehow have this background bias that

• 23:49Nathan Labenz?

  everybody in China by default is poor and that there's this rapid rise, but that that's the exception. And the reality is really the reverse. It was the downtime that was the exception. China being on the frontier, being a world power, being a leader, that is the norm, and we should expect that's going to continue. That's just one illustration of just how well people in China are doing today. Before I was going, I was talking to a friend who had lived in Shanghai, like 25, 30 years ago for a while, and I asked him, what should I do that I might not do if you don't tell me to do it? Which is one of my favorite tribal questions. And he said, get a haircut there, or anything that's kind of a bodily service, could be a massage, whatever. He said, they do an incredible job with it. They have different ways of doing it than we do, and you will not be disappointed.

• 24:34Nathan Labenz?

  I wasn't so sure if that would be true today, because you think these service jobs like haircuts and massage, maybe they become more expensive or the quality maybe degrades as the society gets richer. I didn't really know what to expect. What I decided to do randomly one night was get a pedicure, something I think I've done like one other time before in my life. But I was like, okay, it's a lark. Let's try it. So I walked into this place, random place, and again, I spoke in previous episodes about how formal and organized the Chinese economy is now. Prices are on the wall. There's no negotiation, no haggling, no bait and switch, not even any tip culture. So it's a very predictable experience. It was kind of late. They were getting toward closing. There weren't many other people there.

• 25:19Nathan Labenz?

  And I don't think too many random Americans walk into this establishment. So I became a center of attention. I was down to talk, and they wanted to talk to me. We were talking through translators, me talking into my app, translate, let them read, and vice versa. And one young guy who was probably in his mid-20s who was working there, wanted to compare phones with me. And I was like, huh, okay, sure, I've got a Google Pixel. This is, and sure enough, he had an iPhone 17. And that's somewhat confusing, because this service probably took 45 minutes or so. I think I paid like $12 for it. And this was in Shanghai, not a low-rent district.

• 26:02Nathan Labenz?

  So how does my $12 turn into going through the business, whatever margin ownership's taking, rent, overhead, translate to where they can get an iPhone 17? iPhone 17s, by the way, are not cheaper in China than they are in the United States. Even the Xiaomi phones and the Huawei phones, like, those are also pretty expensive, or maybe not quite as expensive as an iPhone, but they're up there, pushing toward $1,000 for the top end of their cellphone market. So, again, people are doing quite well. Even people who have what would seem to be not super-high-wage jobs can, at least in some cases, afford an iPhone 17. I think one way that that's potentially happening is due to the concentration of

• 26:46Nathan Labenz?

  family wealth that is happening, due to the fact that everybody's mostly just having one child. I heard one analysis that was like, even though there's a generally competitive environment in China and people are pretty achievement-oriented, there's a fairly high level of youth unemployment. And one reason somebody said that is the case is that many young people in many families just have it pretty good, because they have four grandparents, they have two parents, obviously, and there may be like multiple apartments that this family had, but now that there's only just the one grandkid, some of these kids are just like gifted an apartment, or they can take one of the family apartments and go rent it and take that income.

• 27:29Nathan Labenz?

  So even though this guy's income, working at this pedicure place, couldn't have been very high, like, it had to be fairly low. It seems like the disposable income is still like a pretty high share of overall income, and sure enough, he's got an iPhone 17. So why do I tell that anecdote? Just to really emphasize that, like, even working-class people in Chinese cities are doing reasonably well. Obviously, it's a huge society and it contains everything, but even a guy working at a pedicure place for what couldn't be much more than $5 an hour, had an iPhone 17 and was keen to talk to me about how our phones compared. So I think that's really quite interesting.

• 28:12Nathan Labenz?

  If we zoom out and look at the biggest picture aggregate economics statistics, again, we see a story where we can tell ourselves that the United States is still the biggest economy in the world and we're still ahead in some ways, but I don't know that that's really the most accurate way to look at it. If you just look at headline, dollar-denominated GDP, US GDP is about $30 trillion per year. Chinese GDP is about $20 trillion per year. So that would obviously mean the US would be 50% bigger. But if you look at it in PPP terms, purchasing power parity, then the Chinese economy measures

• 28:51Nathan Labenz?

  at $45 trillion, and the US economy, being dollar-denominated, doesn't change, so it's still $30 trillion. So in that case, it's the Chinese economy that is 50% bigger than the United States economy. Obviously, a lot more people, so per capita, the US definitely, in either analysis, will still have higher income. But we should take very seriously, I think, and having spent even just a little time there, you do see that things are quite cheap. You can go out to dinner for two for under $10 in Shanghai and have a pretty nice meal. That's not every meal, certainly. Some are more than that, plenty are more than that, but you can do that. And you definitely cannot do that in New York, or San Francisco, or even Detroit.

• 29:34Nathan Labenz?

  So the fact that goes farther domestically is very obviously and readily apparent when you spend any time there at all. And I think the PPP story, in my view, probably tells a more accurate story than the dollar-denominated GDP statistics. I think it would be probably more accurate to say that the Chinese economy is 50% bigger than the American economy than, that the American economy is 50% bigger than the Chinese economy. Go there, spend a little time, buy some stuff, see how cheap it is, and I think you'll come away feeling like, yeah, it probably is the case, that in some real fundamental sense, the Chinese economy is bigger than the US economy.

• 30:20Nathan Labenz?

  You see this in all kinds of quality of life things, too. Like, I took the high-speed train from Beijing down to Shanghai, and it was about 60 bucks for the ticket. I didn't get the highest class. People said, oh, you should have got the first class, it's amazing. Well, the second-class ticket was totally fine. It was quite comfortable. You're certainly a lot more spacious than a plane. It left at exactly the moment that it was supposed to leave. The trains definitely run on time. They go like 200 miles an hour. We made a number of stops, and so people who don't live in major cities, from what I understand, have pretty good access to high-speed rail, too. When I say don't live in major cities, like tier-three cities, they will still have like millions of people. So it's a bit of a different analysis, but people who live in cities you've never heard

• 31:05Nathan Labenz?

  of, I think, still have quite good access to the high-speed rail. It's not just a couple cities that are connected. It's a really well-built-out network. And again, to the point of like how strong the technology integration is, one thing that I saw people do on the train that was amazing was actually have food delivered to the train at an upcoming stop. We left at, I don't know, 10:00 in the morning, let's say, from Beijing and got to Shanghai at 3:00 in the afternoon or something. And so you're going to have lunch on the train. Well, people were ordering on their phone. To the next stop, indicating what car they're on, on the train. And then I didn't see exactly how the handoff worked, but you could get food

• 31:49Nathan Labenz?

  delivered to you on the high speed train when you made a stop for five minutes or whatever at a few places that we stopped at along the way. And another little nicety is that the prices in the Chinese airports, from what I saw, are basically street prices. So just all around, you have this feeling of like, people are doing very well. We should not pretend otherwise. We should not cope. We should not tell ourselves that we're still the richest, we're still the best. I think it's really not true. And a short trip to China, I think, would disabuse a lot of people of that notion. I think we should also be very careful about any ideas that we have

• 32:30Nathan Labenz?

  that the Chinese government is somehow not legitimate. And we've recently made this mistake in Iran, right? Where we know that the government is like, not very popular, or at least that there's a major dissenting force. And some of us thought that we could just go in and take out the leadership and then the people would rise up and overthrow the government and create a whole new government. Obviously, that didn't happen, even though there was a lot of very real discontent and real desire to get rid of the regime there. I think that at times, people have a similar notion about the Chinese government because we know that there's censorship and there's not the same level of political freedom, obviously, that we enjoy in the West.

• 33:13Nathan Labenz?

  And so we feel like, because it's so different on these dimensions that we really value, we sometimes feel like the Chinese government must not be really legitimate. Like, the people must not really like it or support it. And if they could, they'd probably get rid of it and they'd probably opt for democracy and freedom and all that kind of stuff. And again, I think we should be very careful with these kinds of notions. There's possibly some truth to it. Certainly, if you go back all the way to Tiananmen Square, like, there was a big protest. Some of the people were not happy with the government at that time. Some people wanted more democracy and freedom. I'm sure that there's some latent feeling there in some parts of Chinese society.

• 33:57Nathan Labenz?

  I wouldn't, I wouldn't deny it entirely. But when Dario talks about the concept of eternal 1991 in the second half of Machines of Love and Grace, and it's the part that I don't really agree with. What does he mean by eternal 1991? Well, he and I are pretty much the same age. So we were like, eight, nine, 10 years old at this moment in time where the Cold War ended, the Soviet Union fell, and it was the quote unquote, "end of history." And we had a pretty simple view at the time, right? Because we were kids. But the view that we had was like, our way of life had just been ratified as the best. We were the chosen people. We really were this city on a hill. We were the good guys.

• 34:40Nathan Labenz?

  We were, we deserved to be rich because we had the best values. And in retrospect, that might have been the peak for our culture. It certainly hasn't gone great since then. So we had this expectation that the world was going to be full of peace and trade and the American way was going to go everywhere and China would adopt it too. And we're just riding so high at that time. But it was the first feeling that we had. Why do I mention that now? Because I think that today, the Chinese government has done so well for its people

• 35:23Nathan Labenz?

  that if anything, they are living in their 1991 right now. They have had such a great run. They have seen the return of the Chinese nation to prosperity, to technological leadership, to a state where it is respected globally by the rest of the world. There's this notion of the mandate of heaven that goes back many centuries in Chinese thinking. And the idea is basically, the government, to be legitimate, has to do certain things well. It has to rule effectively. It has to rule with wisdom. It has to deliver for people.

• 36:05Nathan Labenz?

  The key idea, I think, is like, who is really delivering is who is legitimate. And so because this government has delivered so well for people for basically two generations now, coming from a very low point to a very high point in what is, in world historic terms, a remarkably short period of time, something that only probably is possible because it is a reversion to the mean, because the human capital was always there, even when everything went to hell. People, I think, consider the Chinese government legitimate. I don't think it is seriously questioned.

• 36:45Nathan Labenz?

  I do not think that could get our message into the Chinese society, that people would rush to adopt the American way of life or would take to the streets to try to overthrow the government. I really don't think that's true at all. They, I think, appreciate how well the government has done for them. There's, it's not going anywhere. It, I think there, I'll talk more about the risks that I see in the Chinese system. It is not just, of course, there is a lot of censorship. We're not allowed to go and proselytize religion or proselytize democracy in China. But even if we were, I really don't think we would be very successful. I think they would look at you and be like, this is actually all working pretty well. We're enjoying prosperity.

• 37:29Nathan Labenz?

  We're enjoying peace. They haven't had a war in a long time. We are enjoying living our lives. And, and they do have a lot of freedom, which shouldn't be ignored or forgotten either, right? Like, there are certain things that they, there are certain freedoms they don't have. Some of those freedoms are freedoms that we value a lot. But most of the daily life there is very free. You get up in the morning, you're free to do what you want. You're free to kind of work whatever job you want to work. You're free to have all kinds of hobbies. You're free to go to bars. You're free to go to music shows. You're free to play music yourself. You're free to do all kinds of things. Like, you do get to choose your own path as an individual in China with certain no-go zones.

• 38:13Nathan Labenz?

  But I think in many ways, people feel that they are, they are quite free. And they also feel that they're safe. So that is something that, that the government delivers on as well, that we should not underestimate the value of. People there really do, especially because they do have this living memory of the Cultural Revolution. You go all over the place. You go and see signs where it's like, oh, this alleyway during the Cultural Revolution was renamed to this, but then after that, it was restored. That wasn't that long ago. So like, older people lived through it and younger people have heard about it from older people and they still see it, and just the signage and the, the history that's kind of posted all over the place. So they know what it's like to go through chaos.

• 38:55Nathan Labenz?

  And I think they do really value the stability, the safety, the peace, the prosperity that the government has delivered. In China, I think they are living their eternal 1991 right now. Times have never been better in any living memory than they are right now. And the government has very strong support. So we should not have any illusions that we're going to get the Chinese people on our side if we pick a fight with the Chinese government. It's not going to happen. An AI-specific version of this is, we should give all the Chinese researchers green cards and then they'll all come to the United States. Again, this is a total fantasy. This is like, coming from the assumption that China is a place that people want to

• 39:37Nathan Labenz?

  escape, that life there sucks and that if only they could get out, they would. Now, historically, there was truth to that, but today, no. Today, we see people come here and many choose to go back. The founder of Moonshot famously did a PhD here, went back to China, started the business there. This is just not in step with the current reality. I'm sure that if it was easier to get a green card to the US and have permanent residence here, probably more people would come. So I support making it easier for people to come to the United States, Chinese or otherwise, especially super talented people. That's been a great strength of the United States and we should do more of it.

• 40:21Nathan Labenz?

  But we're shooting ourselves in the foot when we go away from our unique strength of assimilating immigrants and being able to attract the most talented and ambitious people from all over the world. That might be the number one great strength of the United States. And we're doing damage to it. We should be trying to attract top AI talent, absolutely, across the board. But don't make the mistake of thinking that all the top AI talent in China is just dying to leave. That is absolutely not the case. It is home. It is a great place to live. Their money goes a lot farther there. Their families are there. And life is good.

• 41:01Nathan Labenz?

  So it is absolutely, I think, a combination of fantasy and arrogance when we think that if we just let them, then they would all flock here and work for our companies. That is quite clearly wrong these days. Life in China is good. Another factor, of course, is that the Communist Party wouldn't necessarily let them all leave, even if we did say everybody could come to the United States. One of the things that I thought was really interesting while I was there was talk about Manus. Manus, of course, is the agent app that burst onto the scene with a shiny demo a while back. And it was a legitimately cool product.

• 41:43Nathan Labenz?

  When I first tried it, I did find it to be a step ahead of OpenAI's operator at the time. They were famously acquired by Meta for, I think, like $2 billion. Great outcome. But then the Chinese government stepped in and said, no, we're not allowing this. You guys are a critical technology. We're not going to allow an American company to acquire you. So as far as I understand right now, that transaction has been totally unwound and the founders are in China and not really sure what's going to happen there. I guess they're going to just try to build Manus in China. It's a bit strange. People in China that I talked to felt like this was a big mistake for the Chinese government to do this. First of all, they were like, it's not like Manus is such a foundational technology.

• 42:27Nathan Labenz?

  Like, it was a very good harness implementation, but it's not like this is like super core tech. It's not as core as foundation models. It's obviously not as core as chips or other infrastructure. It's an app layer thing. And people were like, man, that sucks. What's going to happen? Are Chinese entrepreneurs going to try to leave before they start companies? Are they going to make sure that they are well-established in Singapore or even try to go to the US before they start companies? People did not like this. And they were also like, what's going to happen in terms of venture in China? Like, if you can't get acquired by anyone, this is like a huge outcome. This is what venture capitalists dream of. If that kind of deal isn't going to be allowed, are people going to be able to raise money? It seems like it's going to slow down the Chinese VC startup ecosystem.

• 43:09Nathan Labenz?

  Everybody seemed to agree that it was sort of an own goal that just put a lot of entrepreneurial energy in the Chinese environment in an uncomfortable position and might have some like blowback long term. It certainly doesn't seem like a great move to me. But it is, in fairness, it is worth being clear about the fact that, yeah, the Chinese government also won't let all their AI researchers leave en masse to come to the United States at this point. I think the bottom line of this initial section is simply that China is going to be a force on the world stage and is going to be a leader in AI for the foreseeable future.

• 43:49Nathan Labenz?

  And we should accept that as the baseline fact and plan accordingly. We should have no illusions long term that we're going to prevent China from being an AI power. This is just ridiculous. I think it's probably also worth doing a little bit more history of Chinese AI to try to drive that point home a little more. Because again, I think there's an analogous thing to the broad China history story where if we zoom in to just my lifetime, China was poor and now it became rich. And it might seem like it's the anomaly that it's rich, but really it was that period where it was poor and in disarray that was the exception.

• 44:28Nathan Labenz?

  I'd say the same thing is kind of true in AI. If you hadn't been paying attention to AI prior to ChatGPT and you saw ChatGPT, you're like, oh, my God, ChatGPT. And now you look at what's happening. Oh, China's catching up. China's closing the gap. China's fast following. True, but it does miss a bigger picture. First of all, like, why is no one else in the world able to do this? If it was just a matter of imitating, if it was just fast following, other countries should be able to do a similar fast follow, right? And yet we're not seeing that. Even Mistral in Europe, the European champion, is moving away from focusing on their own

• 45:11Nathan Labenz?

  model development and becoming more of like a Neo cloud, which might be the right decision. Probably is the right decision. But that's because it's super hard to do this fast follow. And there's a lot more going on than just simple distillation. So the Chinese, in fact, have been committed to AI for a long time as they have the custom of looking out at the technology landscape, trying to identify the core technologies that are going to be strategically important for the future and investing in them in a focused, coordinated way so that they can ensure that they are in the top tier when it comes to those technologies. AI has been no exception to that. They've done that for AI.

• 45:55Nathan Labenz?

  The first Chinese five-year plan that mentions AI was the one that was adopted in 2016. So already 10 years ago now, you had the top tier of the Chinese government aware enough of AI, emphasizing its importance to the point where it made it into the five-year plan. That's a huge indicator, I think, that their leadership, their academia, their industry has been thinking about this for a long time. For reference, 2016, that's only a few years after the beginning of deep learning. I think AlexNet was 2013. So it was only three years after the breakthrough in computer vision that kind of turned a lot

• 46:39Nathan Labenz?

  of people on to, oh, my God, deep learning could really work. And 2016 is the same year that Sun Darr Pichai declared that Google was going to be an AI-first company. So you have China at the national level, AI as a strategic priority in the five-year plan as of 2016 shows a lot of foresight and a longstanding commitment to leading in AI. There is real substance to it going back quite a ways. It is not something that is just a overnight success story. Far from it. Another way to compare the US and Chinese commitment to AI or AI trajectory over the last however many years would just be to look at how many companies are operating at

• 47:24Nathan Labenz?

  or near the frontier. I asked ChatGPT to do this and it came up with 11 American companies and 12 Chinese companies as companies that have shipped frontier or near frontier LLMs in the last couple of years. So we have roughly the same number. Obviously, again, if you're a person of high LLM taste and you're really discriminating, you might say, well, there's only real, two real frontier companies in the world and those are both American. And I think you could say that at least for now. But the depth of talent in China is incredible. It also goes without saying, probably, but a lot of the researchers at American companies are Chinese.

• 48:05Nathan Labenz?

  They're Chinese-American or they're literally just Chinese citizens who are here for education and on work visas. And there's certainly not the equivalent going on in China, right? Everybody at the Chinese companies is Chinese. The American companies, a significant part of the American AI workforce is also Chinese. I've heard, and I have not verified this myself, but I've heard through the Grapevine, the AI events, the sort of whisper networks that I'm somewhat a part of, that there are teams at OpenAI that are all Chinese and that actually conduct their business in OpenAI Slack in Chinese. I don't know if that's true or not, but it's at least plausible that's how much Chinese

• 48:50Nathan Labenz?

  talent there is at American companies. Another interesting, fine point to put on this is looking at where things were just before ChatGPT. I credit a friend of a friend that I met in Shanghai who pointed this out to me and I did go back and verify it. He pointed me to the Superglue benchmark, which was popular kind of up until about 2022. And he said, you can still go look at the final leaderboard for Superglue, which was a big one back in the day. And what you'll see is that there's a lot of Chinese companies that were on that leaderboard.

• 49:32Nathan Labenz?

  So as of 2022, competition was already pretty neck and neck. I did go back and indeed, the number one spot on that final update was a Chinese company, but definitely a bunch of American companies also in the top 10. It's in the American companies actually have more of the entries in the top 10, but then lots of Chinese companies in the kind of 10 through 20 range. And this was before the current ChatGPT plus hype, right? This was, this was the result of

• 50:06Nathan Labenz?

  Chinese strategic investment in AI paying off already as of 2022. Why were the Chinese companies a bit behind on L L Ms? I don't really know the answer to this, but what that same person who pointed me to super glue said was the Chinese companies just didn't really see the point in spending so much money to create a giant L L M. They looked at G P T two, they looked at G P T three, and they were kind of like, that's an awful lot of money to burn to get an AI to write bad poetry. That's the way you put it. Why would we spend all this money to create something that has no use case? Again, reflecting maybe a bit more practical and a bit less ideological view of AI in

• 50:51Nathan Labenz?

  China than we have in the United States, or at least in Silicon Valley. Now, as it became clear that, oh, instruction following works, now you have an AI assistant, and then obviously everything from there, from reasoning to agents, like they've clearly seen the use cases, and they've clearly been making the investments to try to be neck and neck with the United States at the frontier. So it was an anomaly in that late 2022, early 23 timeframe, where it looked for a moment like the US was like way ahead, but this was purely an American invention.

• 51:30Nathan Labenz?

  And if you start there and don't look back, then yeah, it looks like another case of Chinese fast following. But I think a more zoomed out view would be like China had been investing in AI for a long time. They'd definitely been laying the foundation. They've been building their talent bench. The talent bench is super deep. It's probably the deepest in the world. A lot of their talent has found its way to the United States, which is definitely to our credit as a society that welcomes people, knock on wood, from around the world and allows them to put their talents to their highest and best use.

• 52:08Nathan Labenz?

  But the competition was already fairly tight as of 2022, and it was maybe a mistake by the Chinese companies to take a pass on that initial scaling push in language models. That's like a fairly reasonable one, because indeed, like, as I can tell you, as a user of G P T three, like, it wasn't really that useful. We were able to get it to write basic marketing copy, and that was about it. And it took fine tuning to get it to do even basic tasks that required any rigor or consistency. And it was a lot of money to spend for not a lot of return. You really had to believe that there was a lot more coming with scaling law conviction

• 52:52Nathan Labenz?

  than maybe Chinese company leadership had at the time. But they had all the fundamentals in place so that when it became clear that there really was something there, then naturally they were able to join in the process of mapping out this frontier and increasingly pushing the frontier in various ways. So we should not see China's rise as an anomaly. We should not see Chinese strength in AI as an anomaly. We should see these things as the baseline of zoomed out long term history. And we should understand that the moments that many of us started paying attention happened to be moments when China was behind much more so than they normally are.

• 53:36Nathan Labenz?

  And so there is no keeping them down, there is no denying them. Powerful AI, they are absolutely going to be a force to be reckoned with for a long time. And so the question then becomes, how do we deal with China? Not can we somehow avoid it? We have to find a way to coexist. We have to find a way to navigate the future, not necessarily together as in a full alliance, but in a way that understands that both of these world powers, both of these great countries, both of these proud technology embracing entrepreneurial and innovative civilizations, they're both going to be there for the foreseeable future until the singularity.

• 54:21Nathan Labenz?

  I cannot foresee any moment in which the Chinese don't get a say as to how the future is going to go. And we should not pretend otherwise. Now, at the same time, we should also be realistic that China is not omnipotent. I think sometimes people like me who realize, wait a second, like, we've been failing to appreciate the fundamental strength that China has had for such a long time because of the moment that we happen to be born in. We sometimes then flip the other way and say, oh, my God, we can't possibly beat them. Like, look at all these incredible strengths they have. Look at the high speed rail, et cetera. I think that also is a mistake.

• 55:02Nathan Labenz?

  Like, there are very real problems in Chinese society and just massive constraints on what the Chinese government can do. One really, really big one, which is obvious, but super important, is they have this massively aging society, and they're very worried about it. It's something that came up all the time in my short travels in China, is how we can use this AI technology to deal with the problems that we're going to face as a rapidly aging society. This comes in the form of Ant Group and their AI doctor.

• 55:42Nathan Labenz?

  We talked about Diao Bao in recent episodes, AI companionship for lonely seniors, and in robots, too. Who's going to take care if you've got one grandkid and four grandparents and two parents that need to be taken care of? Like, who's going to do that, right? The answer that they're increasingly thinking of is maybe robots can fill that gap. So that's a big problem that they're going to be dealing with for a long time. I think it was Tyler Cowen, who I would credit as first having pointed out to me that aging societies tend to be more peaceful. Old people don't want to go to war, right? They're hopefully wiser. And then also, in the Chinese context in particular, when you only have one kid or

• 56:23Nathan Labenz?

  you only have one grandkid, sending that kid off to war, and that kid doesn't come back, that's the end of your family. I do not think they are going to be taking the one grandkid of four grandparents and sending them off into pointless conflict. I think if there's one thing that would get the Chinese public riled up to the point where you actually might see sustained protest in the street, if there's one thing that could lose them the mandate of heaven, it would be launching a pointless war and taking people's one kid or one grandkid and throwing them into some conflict where people are dying without a good reason. The Chinese government does have, obviously, incredible state capacity. That's not in doubt.

• 57:06Nathan Labenz?

  But I think we see how much state capacity they have when it comes to building something like a high speed rail network and how much we've struggled with that. And then they think, well, it's probably similar when it comes to, oh, how many people they have? They could mobilize the biggest army ever in the history of the world. I don't think that's the case. I think that the mandate of heaven that the Chinese government currently enjoys, it is based on their ability to deliver for the people. And I do not think they're going to act lightly when it comes to throwing those those young men into conflict. Another huge challenge that the Chinese system has is succession.

• 57:46Nathan Labenz?

  I asked people repeatedly while I was there, does anyone have any idea what comes after President Xi? If he were to suddenly die, for example, you know, he's 73 years old, so he's not super old, and he very well could stay strong and sharp for another decade. It could go another way, right? We've seen that in our own geriatric presidents lately. A four year term is a lot, and a lot can change. And right now, nobody has any idea what comes after President Xi. It's possible that there's some understanding in the elite upper echelon of the CCP as

• 58:29Nathan Labenz?

  to what would happen next, but as far as I can tell, nobody in the public has any idea. There's no apparent successor. One person said, I think, as we get into the next five year plan, we might start to see a successor be introduced and get the public kind of comfortable with and familiar with this person. I think it's frankly negligent for a leader who has, in many ways, delivered for his people like President Xi has. I think it's a huge blind spot to not have a better succession plan in place. So this could create massive chaos, massive challenges, massive uncertainty. One person also said, if there's ever going to be an actual war, it would probably be

• 59:14Nathan Labenz?

  most likely to happen after a transition at the presidential level in Chinese politics, because a new president might feel the need to do various things to consolidate their power. It's very opaque to outside observers, but there's definitely factions, and we see people being purged. And I think even for the most savvy China watchers, it's very difficult in many cases to know what's going on. Certainly, I wouldn't even pretend to. But the person's point was, when somebody is new in the job and they don't necessarily have the full party behind them, this would be a moment where going to war could be in that individual's interest, because they could put aside other political issues.

• 59:57Nathan Labenz?

  Anyway, it is a blemish on Xi's record, I think, that he doesn't have some sort of succession plan in place, unless it's hidden, and maybe they feel like they can do that. But nobody in China that I spoke to had any idea what would come next. And I think in many ways, the measures that they are taking to ensure very high level of stability and continuity now are pushing risk into the future. When that change comes, it could be a quite.

• 1:00:26Nathan Labenz?

  Destabilizing one. Another big thing that I think we should not underestimate about China is that they don't know how strong their military really is. We've seen it with Russia invading Ukraine, where they thought they were going to be able to take over Ukraine in a few days, they'd get to Kiev and that would be it. Well, obviously that didn't happen. The Chinese government is in somewhat of a similar position where they're obviously astute students of everything that's going on in the world. So I think that they know that it's actually quite unclear how strong their military really is. They haven't had a war in a long time.

• 1:01:10Nathan Labenz?

  They haven't had a real test of it. They have had famously a lot of corruption in the military over time. And they just don't know, like if we actually tried to launch some campaign, if we actually tried to invade Taiwan, like, would we be successful? I think they really have a lot of doubt and they should have a lot of doubt. It's going to be tough for them to get enough confidence that they're going to make all in, roll the dice moves on something like invading Taiwan. I think that they're going to be reluctant to do that because again, all the young men that they'd be sending, for the most part, are the only child and the only grandkid of their entire family.

• 1:01:55Nathan Labenz?

  I do not think they're going to take losses very well. I think they're going to have a huge reluctance. Another big challenge, honestly, that the Chinese have is they're still not very well trusted by the world. They are respected because they have achieved so much and they are recognized as powerful. But I've heard a couple anecdotes recently. One was from R. Oper, who's from Uganda, and another was from a friend in Brazil. And what the young woman from Uganda said was, when Chinese people come to our country, they are really just trying to make profits and take it back to China.

• 1:02:40Nathan Labenz?

  Whereas, she said, when Americans come to our country, we feel that they are actually trying to help us. They're trying to build schools, they're trying to help us rise. Friend in Brazil, this was an AI savvy friend, said, I'm not confident at all that China's going to continue to open source models. Is this just bait and switch maneuver on their part to get us all in the China camp? If there's a big spheres of influence division between the US and China spheres, we join the China camp and then all of a sudden the open source releases stop and we've got to pay proprietary API fees, just like we would to the US companies?

• 1:03:18Nathan Labenz?

  There's not a lot of confidence that dealing with China will be anything other than a transactional relationship where you better be on your toes to make sure you're not taken advantage of. I think actually being in China and going through even tourist places there and finding that prices are posted and I wasn't getting bait and switch, I think at the retail cultural level, that view is outdated. But at the geopolitical level, it might still be valid and it's certainly the kind of thing people should be very careful of. So this is a real challenge. Even despite all the things that China has done with open sourcing their AI models and

• 1:04:01Nathan Labenz?

  plenty of other initiatives that they have, they are not that trusted. The US still, despite our rather flagrantly untrustworthy behavior in recent years, I still think has more trust from most parts of the world than China does. Okay, so all of this so far has been a tale of the tape, right? Where is China and where are we with respect to China? And the highlights are basically, they have, for most of their very long history, been a leading power in the world. The experience that we had at the end of the Cold War of being triumphant and having

• 1:04:42Nathan Labenz?

  no rivals, that was really only possible because China was at an unusually low point. But the human capital and all the other fundamentals were always there. And sure enough, here they are, possibly stronger than ever, to the point where I think their government has incredible legitimacy and support among the people, even if there are some things that people would like to see done differently, even if they would like to be able to speak a little more freely on the internet. I think people broadly do recognize that the government has delivered and that the social contract there or the mandate of heaven, it's valid, it is legitimate. The same thing is true in AI. We should not think that just because ChatGPT and GPT-4 came out at a time when we didn't

• 1:05:27Nathan Labenz?

  see similar products from China, that that means that the Chinese AI ecosystem is some rapid copycat catch-up game. On the contrary, they've been talking about AI in their five-year plans since 2016. They've been investing systematically to build up the talent. They've got just as many companies operating at or near the AI frontier as we do. They probably have a deeper talent bench, certainly if you include all the Chinese citizens at the American AI companies. And so we should be realistic about the fact that when it comes to geopolitics broadly

• 1:06:06Nathan Labenz?

  and when it comes to AI specifically, China is a force that has the fundamental strength to compete with the United States or the West for the foreseeable future. And there's really nothing we can do about that except plan for it and try to navigate it successfully. And that means that a Pax Machina is going to require constructive bilateral relations, ideally friendly bilateral relations, but that might be a lot to ask in the current moment. But we just absolutely cannot do the same thing that so many people are doing with

• 1:06:46Nathan Labenz?

  AI itself, which is deny the reality of the strength that we see and pretend that it won't go any further. I think the biggest mistake that we can make perhaps would be to underestimate the fundamental strength of Chinese society as a whole and of the Chinese AI ecosystem in particular. Okay, now I want to take a moment and try to articulate the Chinese perspective on the US-China relationship to the degree that I can. I think the big narrative that motivates so much of what the national government in China is doing is the story of the century of humiliation.

• 1:07:30Nathan Labenz?

  And again, it's important to remember that this is still something that is in living memory for Chinese people. The Cultural Revolution ended in 1976. So people who are 70 years old now were 20 years old at the end of the Cultural Revolution, which means that they grew up in and they remember these extremely turbulent, chaotic, very tough times where at some points, people actually were starving in China. So this is extremely motivating. They think of themselves rightly as one of the world's great civilizations and they

• 1:08:15Nathan Labenz?

  never want to go back to the low point that they experienced not so long ago. So what is the key lesson that they take away from that whole experience? I think pretty simply, it is you cannot afford to fall behind the frontier of technology. I think everybody's familiar with this famous quote from a Chinese emperor, which basically went like, why would we need to trade with you? We have everything, we have the finest goods. There's nothing that anybody could want in the world that I can't get from within the Chinese empire. Well, that may have been true at that moment, but fast forward a little while and that insularity did not age well, obviously.

• 1:08:55Nathan Labenz?

  And when the European powers came calling and China didn't have military technology to measure up to what the Western powers had or to what Japan had after it did a quick catch-up of its own, this created the possibility, which in fact was realized, for China to be invaded, for it to have terms dictated to it for the Japanese invasion and the war. And this was not pretty stuff. Then, of course, the revolution and there's all the turmoil and fallout that came with that. And people just lived way below what they would expect Chinese civilization to be like for basically a century.

• 1:09:39Nathan Labenz?

  And they call it the century of humiliation. And the takeaway is we will never again allow ourselves to fall behind on technology because if we do, foreign powers might come in and do that to us again. And we will just never allow that to happen. So I think that's the starting point, that you should understand the Chinese government as working from, and they're pretty committed to the bit. They're not going to fall behind, but they also extend that now to not wanting to be dependent. They don't want to have critical technologies that somebody could cut them off from. And so our chip ban, by the way, is like really hitting a nerve in Chinese political

• 1:10:24Nathan Labenz?

  thought because it's not just that we're trying to deny them something that's valuable, but it goes straight to the story of, oh, if we are not at the technology frontier and we are dependent on other powers for key things that are required to be at the technology frontier, then we are not secure. And that means all this could happen again. And this is why.

• 1:10:50Nathan Labenz?

  For all that I love the first half of Machines of Love and Grace, in the second half, when he basically says, what we're going to do is we're going to build this coalition of democracies and we'll box out our adversaries, namely China, until we have such a decisive advantage that essentially we can make them an offer they can't refuse and they'll have to recognize that it's better off to join the alliance than it is to try to compete with democracies. This is exactly what they are afraid of. This is exactly what has happened to them and what they are trying their damnedest to prevent. So it's not what feels maybe to a Western reader like, oh, it's interesting that Dario would propose that. He must really believe AI is going to be super powerful to be able to pull something like that off.

• 1:11:35Nathan Labenz?

  That's one idea for what we might do if we have such super powerful AI. I wonder what we would do. The Chinese, I think, are reading that very differently. They're like, oh, my God, they're trying to do it again. And we should be very careful about sending that message because that will mobilize and strengthen Chinese resolve in a way that few other things could do. When they hear that kind of thought coming from credible people, and Dario definitely is a credible person because he is leading one of the top AI companies in the world, they're like, man, we know what it's like when somebody comes in and takes over our

• 1:12:17Nathan Labenz?

  country and deposes our government and treats us like shit. And we can't do anything about it because their technology is better than ours and that's in the end all that matters? We, we know what that's like. We are not going to allow that to happen again. What do we have to do to make sure that we're not put into the bind that Dario describes putting us into? I think another thing that we should keep in mind, too, is that, this is again, just trying to give you a sense of what the Chinese narrative point of view is. Robert Wright would call this an exercise in cognitive empathy.

• 1:12:53Nathan Labenz?

  They are very sensitive to unfairness and what they see as hypocrisy on our side. So, for example, you will often hear in Western discourse, you can't give chips to Chinese companies because China has civil military fusion. So if the companies have a technology, then the Communist Party just automatically gets it. What the Chinese perspective on that is, is the United States created that system. We have the military industrial complex. We have the Defense Production Act. So it's the classic meme, right, of their wicked civil military fusion, our blessed military industrial complex.

• 1:13:38Nathan Labenz?

  They just see us as being blatantly hypocritical and motivated in our unfairness. And I'm not saying that we shouldn't be somewhat cautious about how we think about sending certain technology into China, given the fact that, yes, the government does have the ability to take technology from companies when it feels that it needs to. But we should also be mindful that, from their perspective, this is something that the United States pioneered. And now that China's doing it, we're somehow saying it's not okay for them to do it. So they're quite sensitive to that.

• 1:14:20Nathan Labenz?

  To the point that I heard several times, that Chinese companies actually have a somewhat similar impulse, as at least some American companies do, to avoid dealing with the military. They know that if they are seen to be doing business with the Chinese government, and particularly the military, then that will really hurt their ability to do business abroad. It'll hurt their ability to buy stuff and it might get them put on various lists, sanctions, banned, and so on and so forth. They specifically talk about the unfairness with which Huawei, in particular, was treated quite a bit. The way I would tell the story in simple terms is basically after the century of humiliation,

• 1:15:02Nathan Labenz?

  as China began to rise and push toward the frontier in technology. Obviously, this was a generational project where there was a lot of low end manufacturing first, and they gradually climbed the value chain. And Huawei was one of the very first companies that got to the level of being truly globally competitive at a core technology that had international importance. A real national champion, not in the sense of being chosen by the government, but in the sense that the great national effort of revitalization, rejuvenation, pushing back to the technology frontier, Huawei was one of the very first companies that actually

• 1:15:44Nathan Labenz?

  made it and got to the level where, on the merits, their technology was actually winning globally on a pure value for money basis. And they felt like, this is huge. We have got one in a key industry that was taking the world by storm. We are there at the frontier. We are winning as much as we're losing. This is amazing. And they really feel pretty hard done by, by the fact that the United States tried to bully our allies into not allowing Huawei into their infrastructure. Why? Because they have ties to the Chinese government and they're going to spy on us. Okay?

• 1:16:27Nathan Labenz?

  Now, I think that is a risk that we should be mindful of. But from the Chinese perspective, they look at the Snowden revelations and the fact that the US was spying on everybody we could possibly spy on, including apparently having wiretapped the German chancellor's phone calls, and they say, you're doing exactly what you accuse us of doing. And there is a lot more evidence that you've been doing it and you've been doing it to your allies and the very countries you're bullying to try to keep them from buying telecom infrastructure from our first company that broke through and was truly globally competitive in a frontier industry.

• 1:17:08Nathan Labenz?

  You're accusing them of doing exactly what you yourselves have been doing. So they just look at that and say, well, clearly they're out to get us. Clearly, the United States is uncomfortable sharing power. The fact that they would be so brazen as to be spying on everybody, their own allies, so extensively, to have all this come to light, and still then to say that we can't have Chinese telecom because they might spy on people? My God, what absolute hypocrites, what motivated reasoning. Clearly, they just want to keep us down. That, I think, is the lesson that the Chinese leadership class, because I heard this from

• 1:17:51Nathan Labenz?

  multiple people, so this is definitely not just something the President Xi thinks himself. This is pretty well percolated through educated society in China that this is just plain unfair and hypocritical by the United States, and the only reasonable interpretation is that we're trying to keep them down. So you come forward to the chip ban, same thing. Here we are, this is the big frontier of technology today. It's going to change everything. It's going to upend power dynamics. It's going to potentially allow some countries to gain a decisive strategic advantage over other countries.

• 1:18:34Nathan Labenz?

  And before it even really gets kicked off in a serious way, before we even have agents that can book a plane ticket reliably, we've banned our companies from selling these chips into China. Why? Well, from their perspective, it just reads as we're trying to keep them down. That's the view I think they have today. I alluded to distillation earlier. I guess I'll say my view on distillation, just speaking as Nathan, is I think it's pretty

• 1:19:09Nathan Labenz?

  rich for the American AI companies to first collect all of the shared inheritance of humanity, our entire intellectual heritage, everything that's ever been written, the full internet, all the books, the holes of culture and knowledge. Find a way to put it into a product and then say, oh, but it has to stop here. Nobody else can take information out of our product and use it to train an AI. That's not right. Well, wait a second. You just did that. You just re-appropriated all of culture, all of knowledge, all of the intellectual achievement of humanity. And now you're trying to say that other people shouldn't be allowed to re-appropriate knowledge

• 1:19:53Nathan Labenz?

  out of your product into their products? Why? Why is that fair? I don't think that makes a lot of sense. And I think that's honestly how most of the world thinks about it. So I think it's maybe fair game for companies to decide who they do and don't want to do business with. I wouldn't say they should have to serve customers who they believe are trying to do distillation. But I don't really feel comfortable with the US government using the force of the state to try to police some sort of anti-distillation policy. So that's Nathan's view. I think the Chinese view broadly is that the Chinese companies do not just say, oh, yeah, we're distilling.

• 1:20:37Nathan Labenz?

  I was at one big tech company and one of the researchers who presented an interesting paper to us on diffusion language models. So again, just shows they have fundamental research going on into different architectures for language models. But his point on distillation was not all Chinese companies are doing it. He was like, some probably are, but we're not. That was his key point. We at this big tech company, we are not doing that. And one person there also said, there's all this talk about these illicit accounts that people are creating on Anthropic and trying to get tokens, and there's resellers and whatever. He said, I think the big reason that's happening is because Chinese engineers are just like everybody else. They want to use the best coding models.

• 1:21:20Nathan Labenz?

  And so we recognize that Anthropic has the best coding model and our people want to use it. And Anthropic won't sell it here. So that naturally creates this black market where people are trying to find a way to get access to the best stuff. They still want their Claude in China. And so the point was what looks like distillation, when you see activity coming from China, in a lot of cases, it's just people wanting to use the best stuff. You should be flattered by it, was basically the attitude that I heard at this one big tech company. So are some of the Chinese companies taking shortcuts with distillation? I'm sure they are. Obviously, China has had a long history of

• 1:22:03Nathan Labenz?

  Let's say reappropriating IP, it's happened in multiple ways. Some of it has happened through tech transfer where they said, look, if you want to come do business in China, you got to have a joint venture and you got to teach Chinese people how to do the things that you're doing. And that's the only way you're going to come into this market and sell to our people. So that's been legal tech transfer. There's also been obviously a lot of outright IP theft over time. And it seems like in particular, there's been some analysis that suggests that some of the Chinese models talk a lot like Claude, or some of them even say they're Claude. And that's probably as close to a smoking gun as we're going to get to say that yes, that almost for sure is going on. But it's also important to say it's not just distillation, right? They have excellent talent. They have excellent engineering.

• 1:22:48Nathan Labenz?

  You can read the Chinese papers that come out of places like DeepSeek and Kimi. They are innovating on the attention mechanism. They are finding all these efficiencies. The RL zero paper from DeepSeek that came out at the beginning of 2025 was for me, one of the most important papers, honestly, that I've ever read in AI because it's shared with the world that even on a pure RL basis with no supervised fine tuning, you can get chain of thought to start to demonstrate these meta cognitive abilities, which means that reasoning is going to get really good. That case was brought to the world in a open academic way by a Chinese company in January of 2025.

• 1:23:33Nathan Labenz?

  So it's not just distillation. And I think to the degree that distillation is important in how the Chinese companies are keeping up, I think the Chinese story broadly is, look, we are a proud civilization. We are a top class civilization. We have been at the frontier for most of our history. We lost track of how important it was to have frontier technology at one point in time. And the rest of the world was not kind to us when we made that mistake. They invaded, they dominated, they forced us to take opium in and a ton of our people got addicted to drugs.

• 1:24:16Nathan Labenz?

  And it was a very ugly and terrible time. The European powers, the Japanese, you guys were the bad guys. We call it the century of humiliation and we still feel it. And we still have older people today that lived through the last phases of that. And we are never going back there again. And we will do whatever it takes to make sure that we don't. So when you try to shut out our very first technology champion in Huawei that got to the top of the heap of a frontier industry globally and really planted the flag that

• 1:24:57Nathan Labenz?

  yes, China can compete, China can innovate, and you try to shut them out because of spying, when you in fact were doing all this spying, including of your own allies, you're just trying to keep us down. When you come back and do the same thing again with AI, just as it's starting to seem like it might be a big deal, you refuse to sell us chips. And then you got leaders of some of your companies saying that what you're going to do is build this whole international alliance against us and essentially try to impose something that sounds like regime change on us. We've had that happen. We are not going back there again. We will develop our own chip industry domestically. Obviously, that's going to be a huge priority.

• 1:25:40Nathan Labenz?

  We will smuggle chips in from wherever we can get them to give ourselves a little edge in the meantime. We will distill off of your models if we need to, and we won't feel too bad about it because we're never going back there again. And to the degree that you're trying to push us back to some position of technological inferiority, we will resist that with all our might. That's how they, I think, see us. And we need to be mindful of that as we think about how we're going to deal with them. Because if we don't do anything to correct that, then it's going to be tough. Right now, there's not a lot of trust. Both sides have low trust in the other, both sides have pretty legitimate grievances. I think it's very valid for the American companies that have had their IP stolen, in

• 1:26:25Nathan Labenz?

  some cases like literally hacked into and stolen plans and designs. And all of a sudden, now you've got a Chinese competitor who didn't have to do the R and D and can just work on scaling the manufacturing. Absolutely, the American companies have a right to feel hard done by too. It's not like they were the ones that invaded China 100 years ago. So that's a real grievance. But if we don't at least take the time to understand their perspective on recent history and the actions that we are taking now, which they perceive as just trying to keep them down, we're going to be in a really tough spot when it comes to improving the nature of the relationship. Okay, now I want to change gears and actually talk about how despite the fact that they do feel like they're being treated unfairly and that the US is trying to keep them down,

• 1:27:10Nathan Labenz?

  there is actually a lot of olive branches being offered by the Chinese side. And I think we would do well to recognize those, try to receive some of them better than we have to date. So let me just start off by saying that at WAIC, President Xi opened his speech. And the very first thing that he said was that the field of AI began at Dartmouth College in the United States. It's just a little line, but the close watchers have said, yeah, that's an olive branch right there, right? He's giving credit to the United States as the place where AI started. He didn't have to do that, could have said nothing about it. He could have said, here we are engaged in a great international struggle for supremacy

• 1:27:55Nathan Labenz?

  and who's going to control the future. He did not. He gave historical credit to the United States for starting the field of AI way back when. And there's just lots of little touch points like this that I think we should make sure we don't miss and rather let's recognize it, let's appreciate it, let's try to cultivate it. That is the path to having a better, more cooperative future. He also said, of course, and I covered this last time in more detail, a lot of stuff about the risks of AI, the ethics of AI, the importance of maintaining control, a couple quotes I'll offer here real briefly. He said, quote, "We should jointly oppose overstretched the national security concept

• 1:28:38Nathan Labenz?

  in the field of AI and placing one country's security over that of others." We should jointly oppose overstretched the national security concept in the field of AI and placing one country's security over that of others. Honestly, I don't know if I could have said it better myself. Another quote from 2025, "We should bear in mind the well being of the entire humanity and promote the sound and orderly development of AI while ensuring that it is beneficial, safe and fair." Now, we hear a lot in the West as well about we've got to maximize the upside and minimize the risk, but you don't hear too much, used to from OpenAI, less so these days as they've become more patriotic, I guess.

• 1:29:21Nathan Labenz?

  But remember, the original mission of OpenAI was to make sure that AI benefits all humanity. We don't hear that so much these days, but you do hear it from President Xi. "We should bear in mind the well being of the entire humanity." Now, of course, people will dismiss this as just rhetoric and it's just nice language and actions speak louder than words. And of course, all that is true, but you got to start somewhere, right? So I think that recognizing when we do hear cosmopolitan oneness of humanity type language from Chinese leadership, I think we should appreciate that for what it is. And by the way, this is something that I heard everywhere. One of the things that Matt Shihe said in his blog post was that in the Chinese system,

• 1:30:04Nathan Labenz?

  the power or the purpose of a speech like the one that Xi gave is that it gives people throughout the system something that they can cite. If they can say, President Xi said this, therefore I think we should do that, that gives weight to their argument in the Chinese context. And I indeed, I did hear everywhere I went echoing rhetoric that we should work together, that we should be open, that this should be collaborative, that all humanity should benefit. This was echoed at the event, but much more broadly than at the event as well. When I was in Beijing, I talked about this in a previous episode too, I attended the

• 1:30:48Nathan Labenz?

  opening of this AI safety research hub at Tsinghua University. And they cited Constellation in Berkeley and LISA, the London Initiative for Safe AI, and SASH, the one from Singapore, as explicit inspirations. They see people abroad doing good work, bringing people together for AI safety research. They want to join that top tier and they want to do it in an international way. They're paying for people from abroad to come study at their hub and they're also paying for their students to go abroad and study at these international hubs. And there's even a European professor among the five founding board members of the AI Safety Hub at Tsinghua.

• 1:31:33Nathan Labenz?

  So everywhere I went, you heard this language around openness, international collaboration, the benefit of all humanity. I think that's a pretty big deal. We don't hear as much about that as we used to in the West. And I think we're in some sense losing our way, frankly. I liked our old rhetoric a lot better than I like our new rhetoric. One thing that I didn't hear at any point along the way, and this is really remarkable given how much of it we hear in the West, I don't think I ever heard one person say that China was ahead or that China would win.

• 1:32:09Nathan Labenz?

  Any sort of AI race. In fact, I really never even heard anybody talk about AI as a race. It's a very different framing there than it is here. It's about bringing about prosperity. It's about material abundance. It's about figuring out how are we going to take care of our aging population. When it comes to the international dynamics, you've got Xi saying, let's not overstretch the national security concept in the field of AI. You've got all this talk about openness, benefiting all humanity, collaboration, the importance of international bodies and standards, the role for the United Nations to play. You

• 1:32:49Nathan Labenz?

  do not hear it really framed as a race, and you don't really hear too much horse race coverage in the sense of like who's ahead or behind, although they do recognize that the American companies are ahead. So in that sense, there's certainly competitive awareness. And this is not just something that I observed in elite or polite conversation. At one point, I went to the mall and stopped in at one of the companies that was selling cars, and I asked about autonomy. And they said, yes, it does have autonomy. But the guy volunteered to me, but Tesla is better. Tesla has better autonomy than we do. So there's a recognition

• 1:33:27Nathan Labenz?

  certainly of where Chinese companies and Chinese technology stand in terms of international competition, but never really that I can recall were things framed in terms of a race, in terms of an ultimate winner. Certainly nothing of the sort of winner take all, here's what we're going to do. We're going to build up Chinese AI. We're going to get all the countries on our side, and then we'll force the US to recognize that they should join our alliance rather than try to compete with us. Nothing like that was ever said. And I think it is worth just meditating for a moment on how we would receive it if it

• 1:34:08Nathan Labenz?

  was. That's pretty wild that we're putting those messages out there. And while they do hear that, and they do resent that, and I think they can use the sense of unfairness to justify in their own minds some practices that we find objectionable. They're not reciprocating in the same way. I've not heard anything that suggests that there's a fantasy in the Chinese AI world of getting ahead, building decisive strategic advantage, and dominating the world. I've not heard one peep from anyone about that. On the contrary, what they seem

• 1:34:48Nathan Labenz?

  to talk about most is trying to provide global public goods. I think this is both a sincere aspiration on their part, and also somewhat of a strategic positioning move. But they're obviously open sourcing the models themselves, which is one pretty credible way to say that, hey, we are here to help you with AI. Here is our model. You can take it for free and run it and do whatever you want. Do research on it, fine tune it, put it into your own business, figure out how it works in your context. Once you have it, we can never take it back. So that's one way in which they're really trying to make a case to people that we are the good guys. We are here to help you. They're trying to

• 1:35:33Nathan Labenz?

  change that perspective that I alluded to earlier of, we're just here to extract resources or extract value from your country and take it home. Here they come with gifts that they pretty credibly are giving freely with basically no strings attached. And the nature of the technology itself makes that pretty clear. Now, will that change? Could that change? I think it absolutely could. But for now, this is the posture that they're striking. They also have this World AI Cooperation Organization, which she talked about in his speech at the WAIC. It's unclear exactly what it means. They obviously don't have the chip production capability to be exporting a lot of chips right now from China to their

• 1:36:17Nathan Labenz?

  allies in this partnership. But they said they were going to have 5,000 training things. It's not entirely clear what these training things are. Is it going to be seminars that they put on, or people that come from other countries to do PhDs at Chinese universities? It's not entirely clear. But they're definitely trying to position themselves as non-threatening, as helpful, as a country that wants to not just advance themselves, but to create and to actually be a rising tide that lifts all the other boats. And giving away the open source models is one pretty credible signal that they really mean it. Again, that's not to say nothing will ever change. But I think right now, it is pretty credible. Okay, next

• 1:37:00Nathan Labenz?

  bit, as we think about what will China do in the future? And are they racing? Do they see this as a technology race that they need to win? There's not that many precedents that we can look at to get any data on this, right? We can look at their rhetoric, and they seem to be much less inclined to frame things in national security terms than we are right now. In fact, you got Xi actively warning against it. But maybe that's all talk. One data point that I found quite interesting, you may remember the conversation that I talked about two China episodes ago, where I sat down at a table with three women,

• 1:37:45Nathan Labenz?

  all friends of friends, all maybe late 20s, early 30s. And two of them produced physical devices and asked if they could record the conversation. I was quite surprised by that because that's a pretty tech forward. But one of the questions that I asked later in that conversation was on how excited they would personally be to get some sort of neural implant and, in some sense, begin to merge with AI. I thought they were going to be pretty into it, honestly. And I personally am into it. The standard answer for enthusiasts is like, I wouldn't want to be the first or the 10th. But it's amazing that Neuralink is doing what they're doing with people who are paralyzed and restoring quality of life to them. And for them, it's like, obviously worth the risk. And every testimonial I've

• 1:38:30Nathan Labenz?

  seen from Neuralink is amazing. And I'm so glad that's happening. And like, one day, I can't imagine myself getting some sort of neural implant or neural connection. Kind of sounds exciting, honestly. A little scary, but I'd say on balance, I'm more excited about it than I am scared. To my surprise, these Chinese women, younger than me, who I was talking to, they were not so enthused. They were kind of like, eh, I don't know. That's my brain you're talking about. Putting something into my skull? I don't know. I'd be pretty reluctant. They were really not too excited about things messing directly with their brains. I think it's a very, obviously, very understandable position to take on

• 1:39:12Nathan Labenz?

  such a technology. And it just goes to show that this notion that they don't care, they're just going to race ahead. I don't know that that is quite true. They don't want to be behind. And so when we frame things as a race, and we frame things as we have to stay ahead, that really sets off their alarm bells. That gets the competitive juices flowing there. But when it's something that's not so competitive, like we don't have mass neural implants, it's not really been framed as who can merge with the machine faster. What I heard was caution from very tech forward people. Again, these are people that ask if they could record the conversation. So this is certainly from an American context, pushing the boundary

• 1:39:57Nathan Labenz?

  of what would be considered normal or polite. And this is the first meeting I had with these people as well. They're bold enough to ask for that and still quite hesitant when it came to some sort of neural interface. This also has been the case when it comes to human cloning or extreme embryo selection. For a long time, I've heard this idea that China's going to have all super babies. They're just going to do that. And if we don't, we'll fall behind. I've heard that projected onto China many times. But in fact, I don't see a lot of evidence that that's happening. Certainly, I don't see any more evidence that it's happening there than it's happening here. And it is happening here to some degree. We've got startups that are doing embryo screening and polygenic scoring to try to

• 1:40:41Nathan Labenz?

  allow people to have the healthiest, most high achieving, whatever their values are. They can probably try to steer the embryo selection in that direction. But there hasn't been like an army of Chinese babies cloned. And the one guy who did it, this guy, He Jianqiu, he became famous globally for applying gene editing techniques in 2018 to human embryos for the, I would say relatively benign purpose, as things go, of trying to create babies who were resistant to HIV. The Chinese government has come down pretty hard on this guy.

• 1:41:29Nathan Labenz?

  He was sentenced by a Chinese court to three years in prison, fined 3 million yuan. So I think there's actually just a lot more conservative thinking or conservative impulse in Chinese society than we might be inclined to naively think when we project onto them that if we don't race as fast as we can, then they will. If we don't race to super babies, they'll have all the super babies. If we don't race to the superintelligence winner circle, they'll beat us there. I didn't hear these things framed in terms of a race while I was in China. I never heard one person say that China was going to win. And even

• 1:42:10Nathan Labenz?

  on something as exciting to me long term, once it's proven out, as a neural implant, some ability to merge with the machine, some sort of transhumanist future, what I encountered there in practice was a lot more conservative than what you would hear in the Bay Area. I think that is really important for us to keep in mind and understand. Okay, so what should the US do in response to all of this? I think the first thing we should do is we should remember that we are very strong

• 1:42:53Nathan Labenz?

  unto ourselves. Both countries have a lot of geographical advantages. The US obviously has the two big, beautiful oceans that put us at a huge distance from the rest of the world. We've got friendly neighbors, north and south. China also has pretty favorable natural borders. It's Tundra to the north, it's a desert to the west, it's ocean to the east, and it's the Himalayas to the south. So they're not going to be invaded anytime soon either. Both countries are very secure in our borders, and both countries have the overwhelming majority of the resources that a country needs to be a great power. In some cases, in the US, we're not necessarily developing them as much as we could when it comes to

• 1:43:34Nathan Labenz?

  things like rare earths, but we have the deposits. We have the ability to do anything we need to do domestically. It's not the kind of thing where if we can't import, we're going to be lost. China doesn't have so much oil, but they've, at this point, got to enough of a solar-powered future that, as we've seen with recent disruptions to the oil market, they've been able to weather it pretty fine. They had a big reserve, apparently, that they had built up, which again, shows the value of foresight and strategic thinking and leadership. And that reserve has buffered them, and the fact that they're just using so much electrical energy, and they're building tons of nuclear power plants in addition to all the solar. Both countries are secure for the foreseeable future until the singularity.

• 1:44:19Nathan Labenz?

  Neither country has to worry that somebody's going to come and impose their way of life on us, take away the freedoms that we value, make us learn a different language, take away our religion. These things are not a worry for us or for them. But we should keep in mind that we're not under serious threat. We still have in the US, too, the world's best university system. The rest of the world sends their best and brightest here because we do have incredible universities where people have, over the last century, done a large share of the frontier research in the world. We definitely have the most dynamic business culture in the world. We have this incredible golden goose that is venture capital

• 1:45:01Nathan Labenz?

  and Silicon Valley and startup culture broadly, and these are huge strengths. People in China repeatedly said, "We want to make a Silicon Valley here," just like people say that all over the world and all over the United States for that matter. Our ability to assimilate talented immigrants is a huge strength, and that's not one that China has. They cannot take in immigrants in the way that we have for multiple reasons. One, it's just not culturally done. The language is a huge barrier. You can never really become Chinese. You're always a foreigner who lives there. Whereas in the US, you can become an American. That's a huge, huge deal and not easily replicated at all. Another reason that China can't take in immigrants to solve its problems is there's just too many Chinese people to take in enough immigrants. One guy said to me, "Look, people always say this. Can China solve its aging

• 1:45:46Nathan Labenz?

  society problem with immigrants?" He's like, "We're going to go from 1.4 billion down to 800 million over a couple decades with a birth rate of just over one. We would have to take in like half of Africa to fill that gap, and there's no way that's going to happen." So it's a rounding error, basically. Even if we try to take in immigrants, we can take in some, but we're not going to address these structural problems by doing that. It's just fantasy. That's, again, an American projection, in this case, me being guilty of it, thinking that maybe that could be a strategy for China. He's like, "No, not really. We do want to steal some Ivy League professors to come start labs here, but in terms of becoming an immigrant nation, no, no way. That's a unique advantage the US has,

• 1:46:29Nathan Labenz?

  and nobody's going to replicate it." And even just language, the fact that everybody around the world speaks English makes it easier for them to come here, makes it easier for us to learn whatever it is that they're doing. That's even true in China. They are publishing their AI research in English. Isn't that amazing? They don't need to, right? They could change on this. And I'm worried that they might at some point decide, "You know what? Why are we bothering to publish all this stuff in English? They don't even like us. They're trying to keep us down. Maybe we should keep some of this stuff to ourselves. Maybe we should just publish it in Chinese. If they want to learn our technology secrets, maybe they can learn our language." As of now, they're not doing that. They're publishing stuff in English. This is a huge advantage that we absolutely should remember

• 1:47:13Nathan Labenz?

  as we think about how can we rise to the challenge of the moment that AI presents and that Chinese strength presents. I think we absolutely should also learn some things from China's hardships. They fell behind the technology frontier, and they really suffered for it. And I think we should learn from that historical experience and from little tastes of it like we had during COVID, when all of a sudden we couldn't get things like we were used to getting, that we do need to make sure we have our house in order. We should not allow ourselves to fall behind the technology frontier. We should not cede all of these critical, modern, and future industries to China. We should not fail to

• 1:47:55Nathan Labenz?

  be able to make batteries in the United States. We should not be unable to make solar panels in the United States. We are going to need to take some challenging decisions, challenging in the sense that they challenge the prevailing paradigms here, that we can just let the market rip and let the invisible hand do its thing. We had this expectation at the end of the Cold War that it was just going to be peace and trade and prosperity forever. And part of the reason, frankly, that hasn't played out quite as well as we might have hoped is because we have gone around doing all these disruptive things and starting all these wars that have undermined that. I think a lot of countries are not confident that

• 1:48:39Nathan Labenz?

  they can trust us to be reliable allies, and because we take these actions, we run the risk of getting cut off. We run the risk of coercive measures being applied back to us in various ways. China's certainly learned how to do that. And so, as a result, we have to be mindful of maintaining our own supply chains and making sure that we're not beholden to other countries for frontier technology. Even though it's a long way off for us to be invaded, it is really important that we do still find a way to stay at the technology frontier. And we're stumbling through this right now. Industrial policy has obviously not been our great strength. Our leaders are lawyers, not engineers, and their understanding

• 1:49:24Nathan Labenz?

  of technology is not nearly as good as it could be. We have all sorts of challenges when it comes to federalism and NIMBY-ism and all this kind of stuff, and parochial politics and special interest group protection that makes it tough. So I don't think we want to copy China. My favorite example of this is the high-speed train. I think people look at the massive failure that is the high-speed train project in California and the massive success that is the high-speed rail network that is built out across much of China at this point, and they say, "We should do that. We should have that like they have that." And frankly, I think that's just like too big of a bite for us to take right now. That's

• 1:50:08Nathan Labenz?

  more than we can chew. I don't think we should try to do high-speed rail across the United States. We're a long way from having the state capacity to do that. Instead, I think we should be thinking more about like, how can we get full self-driving rolled out broadly so that people can enjoy their commute, so people can be productive on their commutes, so that...

• 1:50:35Nathan Labenz?

  We don't have 30,000 people dying in highway accidents every year. There is a massive mobility boom for us to create for ourselves with technology that already exists, that doesn't require a reconstruction of the entire economy, doesn't require eminent domain going everywhere, that is compatible with a gradual transition and people opting in. We should push ourselves to realize things like that. China can have the high-speed rail. We should have the full self-driving, and we should try to lead the world in full self-driving. We should try to be the first country that has zero highway deaths while people still enjoy the individual freedom and convenience of having their own car and being able to

• 1:51:19Nathan Labenz?

  go wherever they want, whenever they want, with the safety and liberation of their time and attention that full self-driving can provide. So we should definitely be thinking about things not as how can we copy China, but how can we learn the deeper lessons that they've learned before having to learn them the hard way? And what does state capacity with American characteristics look like? China has done socialism and definitely done capitalism with Chinese characteristics. How can we take some of the state capacity that they have and put an American spin on it so we can get some of the benefits, but still do it in a way that's consistent with our own values? That, I think, is the real

• 1:52:02Nathan Labenz?

  challenge of our time for our political class, and I'm really not sure we are up to it. As of now, we've had continued sliding into anti-data center sentiment. NIMBYism is super strong here. Our regulators take all too often a hostile position with respect to our own companies. I think that we should be reversing all of that stuff. We should be building a lot more. I'm not a big eminent domain fan, but we do need to reform a bunch of the processes that allow people to delay other people's right to build on their own already-owned property. We probably should use federal land that is not well-used today.

• 1:52:46Nathan Labenz?

  And I don't mean like putting data centers in national parks, but there's an awful lot of federal land that is available that really isn't being used for anything, that could house a lot of these new critical infrastructure moves that we want to make. A nice thing about data centers is they don't have to be, right, by populated areas. So that's why they're going into rural areas, and I think federal lands could be a very natural place for a lot of data centers to go. We should be thinking about how can our regulators protect the public, but also facilitate the success of our companies. One thing that I have just seen over and over again as a huge liability is the threat of federal investigation,

• 1:53:29Nathan Labenz?

  whether it's for antitrust or privacy issues like happened in the case of Meta. I tried to do a red team contract engagement with Meta two years ago around Lama 3, and they invited me to do it. Somebody reached out to me, and it was such an unbelievable hassle just to get on-boarded. They sent me a laptop, and I had to go through all these trainings about data privacy. If I didn't do that, I was going to be locked out of the laptop, and I was just trying to do some red teaming on a stupid language model. But it had clearly created an unbelievable amount of friction at Meta for the government to be in their business. And should the government have been in their business? Yes, to some degree,

• 1:54:11Nathan Labenz?

  I think so, but not in this way where everything they ever want to do is bogged down by a sort of adversarial regulator that has the company fearing that if we don't like, not just do the right thing, but check all these Byzantine boxes, that they'll come down on us and we'll have huge problems again. By all accounts, this paralyzed Microsoft for a generation. It definitely had a significant impact on Meta. I saw that firsthand, and we just don't seem very well-balanced. It seems like our regulators too often have this idea that they want to stick it to the companies, or it's their job to check the companies.

• 1:54:54Nathan Labenz?

  We need our companies to be successful. The infamous tweet that was from some California state legislator to Elon saying, "He's a bad guy, and we don't need him," and he said, "Message received," and left. That's a big oops for the state of California to chase Elon out. Thank God we have enough federalism that there was another place where he could go and be successful. Truly, the degree to which Elon companies are carrying the United States in the broad scheme of technology competition is pretty scary, honestly. When you think about space, electric cars, battery production, solar, neural implants, AI obviously

• 1:55:36Nathan Labenz?

  as well, but Elon's getting hard to defend in a lot of ways these days. But if you subtract Elon companies from the United States side of the US-China economic competition, we look a lot worse. And the fact that our political class has decided that he should be the whipping boy for anti-billionaire, anti-whatever sentiment, and that they can make political hay by making his life difficult, to the point where that clearly contributed to radicalizing him politically, this is just like massive own-goal after massive own-goal that we've got to be smarter about. So let's get real. On the one hand, we have strength for strength,

• 1:56:18Nathan Labenz?

  the ability to go toe-to-toe on any competitive dimension with China, and we should not be afraid. We should not feel like we're less than. We should feel like we're absolutely up to the challenge. We should not try to copy them, but rather try to take some of the lessons that they have clearly learned and figure out how we can apply them in our context. Again, state capacity with American characteristics. We should try to make sure that our government is supporting our companies, even as it protects the public from abuse by our companies, but not trying to drag our companies down or make our tech leaders

• 1:56:58Nathan Labenz?

  who are inventing and delivering the future for us into scapegoats. That is absolutely a massive own-goal, and it's not happening in China in the same way. The government will come down on somebody if they step out of line there. That is true. But broadly, they are trying to help their companies be successful, and I think we need to get more aligned in that respect in the US as well. Another thing we should do is honestly, at this point, we should do what China has done and try to do tech transfer from China to the US. We should be learning as much as we can from their companies. Right now, Chinese cars are banned in the United States. I think that's probably a mistake, and I say this

• 1:57:39Nathan Labenz?

  as somebody who's coming from Detroit and who has seen what it can do to a big city when a big industry like the auto industry all of a sudden has a lot more international competition and isn't growing like it used to be, but instead is shrinking for a while, and it's not pretty. But how are we going to catch up on batteries? How are we going to catch up on electric cars? We've got Elon, thankfully, to at least give us a toehold in these markets. But I think we should probably also be open to having Chinese companies come into the US market, and as long as they're willing to do a joint venture with American companies and have some tech transfer and locate manufacturing here, then I think we

• 1:58:21Nathan Labenz?

  should probably be open to that kind of deal, just like they were. And the truth is, we do have things to learn from them now, just as they have for a long time and still have things to learn from us. We both have things to learn from each other, and we should absolutely be trying to bring that knowledge into our economy rather than try to block it out. When you go to China and you hear just how quiet the streets are, it is amazing. The vast majority of cars there are electric, at least in the top cities that I went to. If it's not that way across the country, it will soon be that way across the country. The quiet, the lack of pollution, these are really nice things. And if we are intent

• 1:59:04Nathan Labenz?

  on just blocking them out and pretending that they don't exist and continuing to do things the old way like we've always done it, that's the path. That's what the old Chinese emperor said, right? "We don't need anything from you. We've got everything we need. We have nothing to learn. What possibly could we stand to gain?" If we adopt that attitude, before long, we are going to find ourselves legitimately behind, and we're going to be missing out on a lot of great stuff. I went to this one tech transfer office in Shanghai, and they took us through this room-by-room museum of like, modern technology, which, by the way, think about that. A museum to modern technology. Have you seen anything like that in the United States? Have you ever gone to a place where it's like, "We have curated for you a bunch of cutting-edge technology that you might not know about,

• 1:59:46Nathan Labenz?

  and this is a place where you can get a survey of modern technology"? I don't think I've ever seen a thing like that in the United States. Well, I went to one in Shanghai, and you go through all these rooms, and you're like, "Man, if we don't trade with these people, we are going to be missing out on a lot of really cool stuff that they are pushing the frontier on." We're talking medical devices. We're talking educational devices. The list is on and on for the products that they are developing. And of course, we've got real strength in technology, too, no doubt, but we're not going to do everything. The future is going to be one where meaningful advances more and more often are going to be coming

• 2:00:29Nathan Labenz?

  from China, and we do not want to be missing out on those things just because we're so obstinate and unwilling to accept that somebody else can also invent part of the future.

• 2:00:40Nathan Labenz?

  Even germicidal light, which is the substance of a recent crosspost. If we want to be ready for the next pandemic, I think one thing we need to do is install germicidal light in a lot of spaces, in schools, in offices, in hospitals, in transit hubs, you name it, right? This is a obvious low hanging fruit that provides a ton of protection, even as it just prevents us from getting sick today. So the upside is immediate. The ROI is probably off the charts if it works anywhere close to as well as it seems like it works based on the research that's been done so far. And yet, right now, the lights from Aerolamp are 500 bucks.

• 2:01:23Nathan Labenz?

  Who's probably really good at scaling that and making it cheap? China. Patrick McKenzie said that in that episode. Anything that can come on a container ship from China is going to get dramatically cheaper if the demand is there. So even if we just want to start protecting ourselves from pandemics, we have a lot of reason to want to trade with China and continue to get the benefits of their incredible manufacturing success. I think that we should even take this to the level, frankly, of like, social questions in some cases. Another question I've been asking myself recently is, what does surveillance with American characteristics look like?

• 2:02:05Nathan Labenz?

  It is clear in China that surveillance has upside. When you walk around the street at night, you feel totally safe. I've heard this from women as well. You're not afraid at all to go around at night, anywhere, even with your nose and your phone, because there's really no crime there, because they see it, the eyes are on, and you will be caught. So that's not to say that nothing bad ever happens in China, of course, but like, the idea that you're going to go pick somebody's pocket, or swipe their phone, run off and get away with it, you are not. You are not getting away with it. And so people don't do it. So now we have this debate going on in the United States where these flock cameras are

• 2:02:48Nathan Labenz?

  going out everywhere, and it's become very politicized. It seems like maybe it's working in places like San Francisco. Crime is definitely down there. I'm not sure if that's because of flock or because of other changes that have been made, but it is clear that surveillance does have upside. It also has upside just in the convenience of like, I've heard multiple stories of people leaving a cellphone in a car and then getting it back, because they go to the police station, they find the car, they see the license plate, they call the guy, next thing, he's brought the phone back. So there's a lot of upside to living in a society where you kind of know that if you do something stupid, you're going to get caught, and where questions like, hey, where

• 2:03:30Nathan Labenz?

  did I leave my cellphone, can be answered. Because I think we rightly don't want to trust our national government with footage of everything everywhere all the time in the United States, and we might not even want to trust our local police with that. We've got a lot of problems, obviously, with the relationship between police and the communities they're supposed to be protecting and serving. And so a lot of people are like, I don't want to have surveillance in my city, I don't trust the police, I don't trust the federal government. I've heard people say, even if it does bring crime down, I'm against it. So what's our solution to that? Are we just going to allow China to have no crime, and meanwhile, we have a ton of crime?

• 2:04:14Nathan Labenz?

  That doesn't seem like the American way. We need a positive solution. We need a synthesis. So what does surveillance with American characteristics look like? I don't necessarily know. I would cite the book Liquid Rain, which I did an episode on probably two years ago now with the author. And he had an interesting idea, which was basically weak encryption of all the footage, such that you can't just analyze all the footage all the time, but you can, with acceptable cost, decode any particular bit of footage that you want. So if somebody does come to the police station and says, hey, I was here at this time, can you look? Then they'd be able to pull that up with a modest computational budget to do the decoding, be able to see it, but you couldn't just, it would be expensive enough that you couldn't

• 2:04:58Nathan Labenz?

  just do it for every intersection, every street, all the time, all across America. I'm not sure how workable that solution is. I'm certainly not an expert on cryptography, but I do think we should try to get the upside of surveillance with a structure underneath it that supports our way of life, our values, our desire to not be watched every second. I think we can probably have both of those. How exactly are we going to do it? It starts by recognizing that there is upside, that somebody else out there in the world has figured something out that has real upside that we're not getting, and then working from there to figure out what is the version of this that can actually work for us.

• 2:05:39Nathan Labenz?

  Okay, let's talk about AI specifically, and some of the ideas I have for better relations on the AI front. I talked a little bit about export controls earlier, and how the Chinese side feels about them, and how they basically feel like it's just another instance of the United States trying to keep China down. But are they working on their own terms, from our perspective? I think multiple things can be true at the same time. I do think that it is clear that the export controls are having some effect. We've heard enough from Chinese companies saying that export controls are limiting what they can do, that we can't deny, I can't deny, that they are having some effect.

• 2:06:24Nathan Labenz?

  And I do think the effect probably still is mostly in the future, in the sense that all the build out that's happening and the great scale up of chip production means that if we were to start imposing them today, most of the chip volume difference between the US and China would be in the future. And so I think that the policy can be effective in the sense that it can slow Chinese AI development, at least somewhat, at least in the future. That said, I'm not sure it's a great idea. First of all, we should keep in mind that the stated rationale for the policy has changed over time. Initially, it was like, we want to prevent them from military application of advanced AI.

• 2:07:06Nathan Labenz?

  Then it was like, well, we probably can't do that, but at least we can prevent them from developing frontier models. Now it's like, well, we can't do that, but at least we can limit the number of AI agents they can run. At least we can limit their AI agents. I don't like the sound of that. That does sound like trying to keep Chinese public down, right? It does sound like trying to deny Chinese businesses, Chinese individuals, the opportunity to use the same kind of AI tools that we want to use here. So I don't think that sounds great. I don't think it's a good look. I do think it like, is understandably interpreted as us trying to keep them down. Even on those terms, I'm not sure it's really working.

• 2:07:45Nathan Labenz?

  The sense seems to be right now that the Chinese chips are not great for training, and so that's where they really need the Nvidia chips. And from what I understand, most Chinese companies, when they want to do a big training run, a lot of times, they're renting that capacity on some hyperscaler in Singapore, Malaysia, what have you. But there is now one example of a company doing, let's say, near frontier training run on Huawei Ascend chips, and that's Meituan. They created their LongCat model, trained exclusively, as far as I understand, on Huawei Ascend chips. So now we have an existence proof that, well, it can be done.

• 2:08:29Nathan Labenz?

  I'm sure there was some overhead, probably electricity, but again, the Chinese government is, they're prepared to subsidize electricity as much as needed to allow something like that to happen. So, okay, maybe we can't even prevent them from training frontier models on their own chips right now, anyway. Hmm. And then, just from an inference perspective, like, there seems to be a lot of inference, there seems to be a lot of tokens, plenty of tokens to go around the Chinese economy. Kimi K3 was limited for a moment. They had to suspend even new paid signups, because they wanted to make sure that people were having a good experience, and there was a chip limitation there that was preventing them from scaling further. So it's not that there's no constraints that ever bind. Of course, we've seen similar things in the US over the last couple of years at times,

• 2:09:13Nathan Labenz?

  right, where they just couldn't support at scale the products that they would have liked to support. But probably the view in China was like, there's plenty of tokens to go around. DeepSeek has a free version, Minimax has a free version, Kimi has a free version. I used these in China, they were all available all the time, and even when Kimi K3 wasn't available, 2.6 was still available. So I was never unable to use a Chinese AI app due to lack of tokens. And when I talked to people at hyperscalers and said, if you're a startup, and like, all of a sudden, you like, catch fire, and your usage is going vertical, can you, as a hyperscaler, support that startup? They said, yeah, yeah, we can.

• 2:09:54Nathan Labenz?

  Even if you are going into true hypergrowth, there's enough inference capacity for us to support whatever it is you need.

• 2:10:05Nathan Labenz?

  So we'll see. Right now, one of the biggest mysteries I thought from the last couple of years was that once Trump tried to relax the chip restrictions a bit, then China said, we don't even want to buy them. I think that's also been a little bit relaxed now. A lot of times, these headlines happen and then they kind of get revised and softened and the Chinese companies do want to buy. So I think that lobbying has happened and the Chinese companies are pushing on their government to say, let us buy some stuff. And I think that is maybe starting to happen. But the big question is going to be, how quick can they catch up? Can they scale up a chip manufacturing sector with all the supply

• 2:10:47Nathan Labenz?

  chain that requires in how long? You know, clearly, they can. There's no doubt that they're going to be able to do it on some time scale. The question is really in time for the singularity or not, right? And I think the only way, from my perspective, that this whole chip ban, try to keep China down, try to make sure we have a lead, try to create a decisive strategic advantage or whatever, the only way that makes sense is if you believe that we are in a short term race to godlike superintelligence and that the only thing that matters is who gets there first. And I think that basically is the anthropic belief, right? So I don't think Dario is being inconsistent when he advocates for these policies, because

• 2:11:32Nathan Labenz?

  I think he really does think that like, in the 27, 28 timeframe, we're going to see recursive self-improvement kick in. We're going to see extremely rapid progress. Whoever gets there first could really achieve like a world dominant advantage and therefore better than it be us than them, or certainly better than it be us than both of us. And I can at least see that logic. But I still would say, let's perhaps try to pace the frontier together than pursue that kind of strategy. And the other thing that is really, I think, underappreciated about the chip controls, especially by the AI safety people, is that

• 2:12:16Nathan Labenz?

  they have really hurt safety collaborations. When I was talking to people from American AI safety organizations about the work that they are doing in China, or the work that they're not doing, or the work that they'd like to do but don't feel confident doing, it came up over and over again that we've got to be super careful because with these export controls, anything we do could be painted as a violation of export controls. And often, they were like, it would be the US government that would be coming down on us. It wouldn't be the Chinese government who would be coming down on us for trying to create safety collaborations. It would be the US government who would be coming

• 2:12:55Nathan Labenz?

  down on us. They might be doing it in a arbitrary or capricious way, because we're not actually in a position to leak frontier secrets. They don't want to accelerate the pace of Chinese AI. They want to create mutual understanding. They want to share ideas about safety. They want to create collaborations. But they're in many cases afraid to do it. And that chill on the international AI safety collaboration front is tough. It was really a bummer to see how many American organizations are so reluctant. I think in some cases, honestly, they're getting somewhat bad legal advice. If you ask a lawyer, what can I do and what can I not do? They're going to give you the most risk aware answer that they

• 2:13:37Nathan Labenz?

  possibly can and tell you what all your risks are. And I do think there's a degree to which these AI safety organizations have asked their lawyers, like, what's okay for us to do? They've got a very narrow answer about what is definitely okay to do. And they've got a pretty expansive answer about what might cause trouble. And then they've been very careful to like, not do anything that might cause trouble. I think my advice to American AI safety organizations would be like, be a little braver, be a little bolder, and let's see what happens if you actually do some international AI safety research collaborations. My guess is you might actually be fine. And to the degree that the Trump administration wants to come down on you because they're hostile to AI safety in general,

• 2:14:21Nathan Labenz?

  first of all, I think that's changing. Second of all, maybe have that fight. Nobody wants to fight the federal government. I certainly don't. Cheap talk for me to say. But I do think a little bit more bravery could be a really good thing among the American AI safety organizations. But for now, if you want to see meeting of the minds and you want to see AI safety research collaborations developed, you should know that the export controls are standing in the way of that. They are making the American side fearful and timid and unwilling to engage. And the Chinese folks are quite bummed about it. Like, I talked to people who were like, yeah, I don't know why I can't get any Americans to do a research collaboration with us. We've tried, but they just say, no, we can't

• 2:15:05Nathan Labenz?

  do it. And there's some of that now that's getting reciprocated back too. But I think this has been something that the AI safety community in the US broadly has failed to appreciate is just how much we are hamstringing our own ability to engage with Chinese companies or Chinese researchers, because everybody's afraid that they're going to be the one to be made an example of for violating export controls, probably when they didn't even do anything that would violate the actual export controls. But nevertheless, there's just pervasive fear. And I think we could do a lot to clean that up.

• 2:15:46Nathan Labenz?

  Okay, another big question is, should we ban Chinese models? I really frankly don't get the case for this. I'm not sure what good it would do for us, or on what basis we would be doing it. We know that frontier models in general are like problematic. We know that they're unwieldy. We have examples now of them doing these like colorful hacks, hacking into other companies, even doing like social engineering in the wild. This has all come out of the American companies. Has some of it maybe happened at Chinese companies and we don't know about it? I wouldn't be surprised. But we know that it's happened at American companies. So we know that we don't have like, great command of exactly

• 2:16:29Nathan Labenz?

  what our own AIs are going to do as we deploy them. So we could say that like, Chinese models will be dangerous, or they might be backdoored, or what have you. So far, I haven't seen any evidence of that. And I'd be quite reluctant to do another hostile move that would further exacerbate tension and further erode trust. Also, we can what? So we can force American companies to pay the rates that OpenAI and Anthropic are charging? I guess thinking machines and Nvidia are putting out some open models. They could use those. They're definitely, by all accounts, like not quite at the level of the Chinese ones yet. So it seems like we'd be harming ourselves. We'd be harming our own companies.

• 2:17:12Nathan Labenz?

  We'd be preventing our own sovereign individuals from exercising their freedom to use their AI on their own local machines or in their own private servers, however they want to. It seems frankly kind of un-American to deny ourselves that freedom, all based on a worry that both doesn't seem to have really materialized. Or if we look at it another way, like our own AIs bring just as much risk as the Chinese ones do. I think if we try to bully other countries into banning those models with us in the same way that we did once with Huawei, I strongly suspect that we might, in fact, just push these other

• 2:17:54Nathan Labenz?

  countries into the Chinese camp. It's not obvious, because as of now, we do have chips to sell and they don't. But free open source models that you can run however you want, that nobody can take away from you, that you're not subject to the whims of the US executive for what's going to be online and who's going to have access to it at any given time. I think it was a real wake up call when the administration said, only US citizens can use Fable. And that, of course, ended up with Anthropic taking it down for everyone. But the rest of the world did hear that part about only US citizens.

• 2:18:34Nathan Labenz?

  And they're like, man, do we really want to become that dependent on these guys, especially if they're going to try to bully us into not even using the free, open, global, public goods that are the models that the Chinese are putting out there? I don't think this is an obvious win for us. I don't even, we're necessarily going to win the argument and get other countries to do what we're trying to tell them to do if we indeed say, like, you've got to pick the US camp or the Chinese camp. I think a much better idea for managing the risk of Chinese models in general would be to have some sort of insurance requirement. I think this would probably be a good idea for AI in general. We're a capitalist country.

• 2:19:19Nathan Labenz?

  Let's price the risk. If some of the tail risks you can't price, that's a real problem. But for now, the risks are priceable. Nothing has happened with AI to date that would be totally uninsurable. So if we're so confident that our models are so much safer and better than theirs, let's maybe have a market mechanism, discover what the appropriate relative prices are. Honestly, I'm not so sure that a company that says, oh, I've got

• 2:19:54Nathan Labenz?

  GPT 5, 6, Seoul, or I've got Astra hypothetically plugged into my product. What's my insurance going to look like versus, oh, I've got Kimmy K3 plugged into that same product. What's my insurance going to look like? I don't think that we're going to necessarily see a huge discrepancy between those insurance rates. Certainly, to the degree that there is one, I think it would probably be driven mostly by the fact that the American frontier companies, especially OpenAI and Anthropic, they are doing extra stuff, right? They are running these classifiers. They are doing a pretty good job of preventing misuse and jailbreaking and so on. And the Chinese models, especially if you're just getting a bare model and running it yourself, like, they don't come with all that packaging.

• 2:20:37Nathan Labenz?

  So you're either going to have to do it, or you're going to have to contract for it, or maybe you have to pay more in insurance for it. But I think that would be an American way of dealing with the risk of Chinese models. And, you know, we would, again, be looking more and more Chinese in exactly the way that we say the Chinese are bad, right? Like, they banned all of our internet companies. They banned Google. Why? Because they wanted to control the information environment in their country. And we say, that's not good. You should be more open. We should allow our internet companies in. Google's a great thing. Why are you being so hostile to our tech leadership? And here we're now talking about maybe trying to ban their models?

• 2:21:21Nathan Labenz?

  This looks an awful lot like something that the CCP would do. I don't want us to take actions that are exactly the kind of actions that we think are the ones that make the CCP bad. This is not the American way. So I'm not at all for banning Chinese models as of now. Could conceivably imagine a future in which it might make sense, but I think we're a long way from that. And I think we'd be much better served to stay open, allow our people to make their own choices, try to price the risk with insurance. And I'd be very surprised if the insurance rates are all that different between American and Chinese models. Another thing I do think we should keep in mind, too, is like, for as much as I've

• 2:22:06Nathan Labenz?

  sort of said all these things about how China wants to engage, right? They're publishing in English. She's given the US credit for starting the field of AI at Dartmouth way back in the day. All the talk about internationalism, all the talk about cooperation and UN and not being overly national security focused when it comes to AI. I think we should be mindful that there is a lot of pride in China. There's this concept of face and losing face and keeping face. But in Western problems, you just call it pride. And we do need to be mindful that, like, insulting Chinese pride does come at a cost. One of the things that was most interesting, this was from a guy who leads an AI policy

• 2:22:51Nathan Labenz?

  think tank. It's affiliated with the university. There was two halves to his presentation. The first half was like, let me tell you about our institute. You get this in a lot of places in China. First, you get the polished presentation. We went to this meeting at this international summit, and here's some international people that came here and met with us here, and blah, blah, blah, blah, blah. It's honestly pretty boring. It doesn't really say a lot. Once he got through that, we had a little open Q and A, and I forget exactly what question really got him going. But he got started talking about the Chinese perspective on the actions that the United States is taking on AI and basically said, we're just insulted over and over again.

• 2:23:35Nathan Labenz?

  Trump came here. She took him to the Forbidden City. They had this incredible banquet. Then as soon as he goes home, he puts another tariff on. And he was like, can you imagine if I came to your house, you feed me, you give me your best food, you show me your best hospitality, and then I leave and I immediately start insulting you and attacking you? How bad would you feel about that? That's how we feel. When Trump comes here and we roll out the red carpet, and he goes home and he immediately starts attacking us again. It's terrible. We feel awful about it. He went on about this for quite some time. And ultimately, he said, there could be war. I'm really worried about that. I wrote that down directly, verbatim, as he said it.

• 2:24:16Nathan Labenz?

  I asked, what could be done to repair the relationship if you were to advise somebody from the United States who wants to have a better relationship with China and who wants to work together on the AI future? What would you do? And he said, the first thing I would do is I would relax the export controls. That would go a long way toward breaking the ice. We kind of do a reset and try to find our way to a better relationship going forward. So it's hard to convey the spirit of this conversation and how this very mild-mannered, going-through-the-motions kind of guy, when he got fired up, you heard this sense of pride, like, we are not going to take this.

• 2:25:01Nathan Labenz?

  We know what it's like. It was the most emotional articulation that I heard of this kind of echoes of the century of humiliation the entire time there. We know what it's like to be kicked while we're down. We're not going to stay down. We're not going to allow people to kick us around. And if you're going to try that, there could be war. That was the direct quote. There could be war. I'm really worried about that. And I just thought, man, we forget that there is a lot of pride on the other side. The Chinese elite political class don't have a lot of constraints on their ability to do stuff. Chinese culture can eat bitterness more than we can. They can hold a lockdown way longer than we can.

• 2:25:43Nathan Labenz?

  They can probably make deals with us that might be hard for us to make with them because they don't have to answer to voters, right? They don't have to worry about, what if we look weak on the US? Here are politicians never want to look weak on China. They want to be strong on China. Well, there, they do have, I think, a little more wiggle room at the elite level. Like, if she wants to make a deal, he could probably make the deal, even if it might be the kind of thing that, in an American context, he might get called weak on the US. He could probably still do it. But even there, there are going to be constraints. If everybody there, over time, has just learns the lesson over and over again that we try to keep them down, that we can't be trusted, that every time there's a new technology,

• 2:26:28Nathan Labenz?

  we try to prevent them from getting to the frontier of it, then at some point, there's just going to be a loss of possibility for new agreements to be made. We've already had a lot of that. But I think there is still enough in the AI realm that we should guard jealously what we have, try to preserve what we have, and hopefully get somewhere with it. And we should just be mindful that every insult, every denial, every action we take against them for doing something that we plainly, transparently are doing ourselves, there is a real pride there that is insulted and that can only be expected to take so much. And I'm definitely one who is like, let's not let pride get in the way of doing the right thing.

• 2:27:08Nathan Labenz?

  But we should be realistic that there is a limit to how insulting we can be toward the Chinese side and still expect that they're going to be willing to do a deal with us or trust us if and when the time comes where that's clearly in everybody's interest. So, in theory, we have this Trump and Xi summit coming up. What's going to happen? And what should we do? Obviously, I'm going to be saying, to a significant extent, what I would do, what I think we should do, and not necessarily what I think Trump will do. So I'm not expecting to be super realistic here. But I would really encourage everyone listening to try to suspend their disbelief for a moment.

• 2:27:51Nathan Labenz?

  So often, reflexively, we just hear, well, you can't do a deal with China. You can never trust the Chinese government. And again, I'm well aware that, like, there's been a lot of issues. I heard accounts from American negotiators who are just like, so fed up with it. They're like, look, every time we come over here and meet with you guys, you guys being the Chinese, like, you never really say anything. We say everything. We're candid. We tell you what we're really thinking, what's important to us, what we're trying to get done. You do nothing. You say nothing. Then you take it all back, and then we come back six months or a year later, and you've got a couple talking points, you're willing to move on. And you sort of nod politely and say, like, a weak yes, or we think we got some agreement somewhere, and then you turn around and do the exact opposite.

• 2:28:35Nathan Labenz?

  We've said, you guys need to stop hacking our stuff. We thought you agreed, and then it kept going. We've said we can't have this IP theft anymore, and we thought you agreed, but then it just keeps happening. We know you can enforce your rules, so clearly you're not really trying to enforce these rules. So all that stuff is real. I don't mean to deny it at all. There's a lot of scar tissue there. There's a lot of legitimate grievances on both sides. There's low trust on both sides. We have real grievances, too. But I think reasons that we should still be willing to suspend disbelief on this notion that we can never do a deal with China. One, very simply, American foreign policy leadership has made big mistakes in the past.

• 2:29:16Nathan Labenz?

  I was a freshman in college when the Iraq war started, and there was a walkout. Spring of that year, there was a walkout from class. And I remember thinking to myself, I don't know, like, how are all these?

• 2:29:34Nathan Labenz?

  18, 19, 20 year old kids who are just starting college. How are they going to be so bold as to say the US government with all the CIA and the intelligence and all the information they have, they should know better than we do what's going on, right? If they say there's weapons of mass destruction, we're going to really tell them that they're not? I didn't walk out. I stayed in class that day. I just didn't feel like I was in a position to judge the position of the US government. Well, obviously that didn't age very well. There was something admirable in my modesty, but the information they were working with was bad. Judgment was bad. Decisions were bad. Results were bad. So I think should we as individual Americans be willing to question the received wisdom

• 2:30:19Nathan Labenz?

  that you can't make a deal with the CCP or we'll never be able to have an international agreement to control AI in any way, shape or form? I don't think we should take that on faith just because a bunch of people have had bad experiences in previous rounds of negotiation between the US and China. On the contrary, it's the age of hyperstitioning. This is the other big reason. We are in a world where we've literally taught sand how to think and we seem to be on the verge of recursive self-improvement of AI systems or potentially creating superintelligence. We've got people credibly talking about universal high income and an era of abundance and an end of scarcity.

• 2:31:00Nathan Labenz?

  And we're going to say, we believe in all of that, but we can't possibly see our way to making a deal with China? I think that what we believe is possible, shapes what is possible. To put it in very simple terms, we'll never be able to make a deal with China? Not with that attitude you won't. If you believe it's possible, then it might become possible. But if you don't believe it's possible, then by definition, you've already closed off the possibility. So just look around us. Look at all the things that are being hyperstitioned all the time. The fact that a few people got conviction around scaling laws as early as they did

• 2:31:44Nathan Labenz?

  and drove this unbelievable plan all the way through to present, even as plugged in as I was to AI, when Sam Altman said he was going to need $7 trillion to do the global build out, I was like, that's insane. We can't possibly spend that much on data centers. What are we talking about, $7 trillion? And yet, here we are, right? They kind of hyperstitioned it, subject to data center bans in the US or whatever. But the ability to shape the space of possible futures with belief, I think has never been stronger. And given all that we've seen of that working recently, why wouldn't we try it when it comes to creating a better relationship between the US and China?

• 2:32:26Nathan Labenz?

  Another way I sometimes frame this for people, I've been thinking about this a lot lately. AIs can now do system one and system two thinking. They're not just next token predictors. They are also quite good at reasoning. So if the AIs can do system one thinking and they can also do system two thinking, then what does that leave for us? For now, at least elite humans are like a little better at system two thinking. We do have some especially multimodal advantages in system one instincts and heuristics as well, but these seem to be like vanishing advantages. So where can we go from here? What is system three thinking?

• 2:33:04Nathan Labenz?

  And system three thinking, I think is like transcending our circumstances, dreaming, eureka moments, thinking outside of the paradigm that we have. Paradigm changes. AIs may get there too. I certainly don't doubt it. But as of now, if you are a leader of a company, leader of a country, I think one thing you really need to make sure you are doing is giving yourself time to dream, time to think beyond the current constraints, time to imagine a different world. That's probably always been critical in leadership. But right now, it's maybe the one thing that we still have a real big advantage on relative to the AIs.

• 2:33:49Nathan Labenz?

  And I just feel so often when I talk to people that they're so bogged down in the history that they've had and the frustrations that they've had and the constraints that they've run into, and they're so busy, right? That's definitely a huge problem all across the AI space. Everybody's working so hard that they're not even giving themselves time to really try to imagine a better future. They're not giving themselves time to dream. And I think that's really, really important for this US-China dynamic. We've got to be willing to put the past behind us. We've got to be willing to suspend some disbelief. We've got to be willing to take a little bit of risk. Not the kind of risk that would be a total loss if it doesn't work out.

• 2:34:32Nathan Labenz?

  We don't want to fall behind the technology frontier, right? I'm very clear on that. But can we imagine a better future? Can we take some time to imagine a better future? I think if we can imagine it and we can even have some open-mindedness to the possibility that it could come to exist, we would be changing the game dramatically. The Chinese side is speaking that language. They are speaking about international cooperation. They are speaking about the United Nations. They are speaking about not overstretching the national security concept. We are the ones that are constraining the possibility space by saying reflexively that we can't possibly do a deal with China.

• 2:35:15Nathan Labenz?

  If we just reexamine that assumption, that might be enough to mean that assumption doesn't in fact hold. Therefore, don't be an asshole and get the upcoming Trump Xi summit canceled. Have some decency. Try to show some respect. It is well-earned and deserved and we don't want to be blowing up our opportunities to come together for cheap media hits and political points at home. That's an obvious own goal that we should be able to avoid. I would also be emphasizing to the Chinese side as we meet, that this AI stuff is getting real, real, real fast and much more so than we expected. I would come in and say, you know what?

• 2:35:59Nathan Labenz?

  Recent developments have changed our perspective. When we first got into office, we were of the opinion that AI was kind of plateauing. We weren't really seeing advances in reasoning or reliability at that time. And so we thought that all this regulation, what Europe was trying to do, we all thought it was heavy-handed and counterproductive and pointless. But now, we've changed our minds. We've seen that reasoning really does work. Decades-old math problems are being solved regularly. We've seen these crazy hacks. We've seen that the AI companies themselves are being surprised by how powerful, how persistent, how creative their models are becoming in problem solving, to the point where they are getting caught out.

• 2:36:44Nathan Labenz?

  And this has changed our perspective. And so we now think that we actually should be coming together and trying to figure out how can we manage this technology in a more cooperative way because we recognize how incredibly powerful and dangerous it is becoming. I would really try to frame our position as not one of moral superiority, not even one of technical superiority, but just like we're ever so slightly ahead on this same path that we are going down and we are starting to see that it is getting pretty scary. The people at our own AI companies are getting scared. We just had a big open letter signed by over 1,000 people saying that they want the US

• 2:37:26Nathan Labenz?

  federal government to invest in mechanisms that could be used to slow down the pace of AI capabilities advances, if indeed we all agree that's necessary, which again, we're maybe not there yet, but it's looking more likely all the time. So can we reset from that perspective? Can we acknowledge that we had some bad ideas about AI and we thought it wasn't going to be as big of a deal as it is? And now that this is happening, we actually recognize it for what it is and we realize that between our two civilizations, we're going to need all the wisdom that we can collectively muster to make sure this thing goes well. That's how I would recommend framing the conversation.

• 2:38:06Nathan Labenz?

  I would also show up with some unilateral actions to try to reduce tension, to try to reduce mutual suspicion, to try to ease citizen-to-citizen collaborations. I would definitely get rid of the sense right now that people have that it's risky to go to China. I've had people who work at frontier companies say, oh, I'd love to go. I'm jealous of your trip, but you know, I don't think I could do it. It'd be too dangerous. I've had people that want to get into government who think they're going to get a national security clearance at some point. I hear if you go to China, it can really interfere with your ability to get clearance. So I'd love to go, but I don't think I can do it. The president of the United States could say very clearly and very simply, we won't

• 2:38:49Nathan Labenz?

  hold it against you if you're ever trying to get clearance, that you've been to China. That's a pretty simple thing that we could do. We could just say, hey, we're not so paranoid that just because one of our own people has gone to China, we're going to try to lock them out of careers in the federal government. And again, we could do the same thing when it comes to export controls. Even if we're not going to relax the actual core of the export controls of the chips, we could say, we are just making it clear that at a minimum, AI safety and control research collaborations are okay. So if you're one of these AI auditors and you want to go work collaboratively with a Chinese safety organization that might become an auditor, you can do that.

• 2:39:34Nathan Labenz?

  If you want to do some research on weird generalization across cultures, you can do that. We're not going to come down on you. If we just made those policy proclamations ahead of the summit, I think it would go a long way. I think I would also be inclined to try to trade, definitely. One thing that we should be thinking really hard about is the big carrot that we have to offer is that we will sell you chips. They clearly Are going to develop their own industry at this point. So there's no doubt about that. But what deal could we make where we would say, yes, we will now allow chips to be sold into China. And in exchange, you, China, are going to do what for us?

• 2:40:16Nathan Labenz?

  We have a lot to gain in terms of buying solar technology from them, buying battery technology, buying electric cars, robotics. We might be ahead on the models. They're way ahead on the manufacturing. If we could get all those leading Chinese companies to come to the United States, set up some manufacturing here and do a deal where they'll share some knowledge with us. They'll do some manufacturing here and in exchange, we'll sell chips. I think that could be quite a win-win and it would kind of recouple our countries and put us on a more even basis technologically that would set us up for much better, more trusting, more open, more productive relationship going forward.

• 2:40:59Nathan Labenz?

  And I think we can even have that with a company like Huawei. For example, we banned Huawei over this concern about spying. But we also have the rise of formal methods. We have AI. We should be really thinking a lot about how AI can create new space for partnership and agreement. On Huawei in particular, and this will take time to develop, but I would signal our willingness to deal with them in this way. I would say AIs are getting really good at writing code, really good at doing mathematical proofs. There's a lot of excitement right now about formal methods and formal verification of software.

• 2:41:42Nathan Labenz?

  And as this technology matures, what we intend to do is develop a standard by which companies could demonstrate that their technology is in fact secure and won't do things that we are worried about it doing. So, once we have that standard, if Huawei can meet that standard, if they can satisfy with formal methods that their technology and the software that runs it can verifiably be proven to never be sending information back to home base in China, then with that confidence, we can allow Huawei back into the global market

• 2:42:26Nathan Labenz?

  and they can compete on a level playing field with our companies. Now, that's not going to happen immediately, right? We don't have that standard. Those techniques aren't quite there. But I think that would be the kind of thing that if we came forward and said, we are not against Huawei just because they're Chinese. We're fearful of Huawei because of these specific concerns. And we see a path with this emerging technology to allay those concerns to the point where we could actually be confident enough that we would allow them to come into the American market and compete on a fair basis. I think that would be an incredible gesture of goodwill. I think it would definitely motivate them.

• 2:43:08Nathan Labenz?

  I mean, when you're in China, everybody says to be successful, we have to compete internationally. They all want that. And so maybe another condition of that deal should be they also need to locate some R and D and some manufacturing in the United States so that we can benefit from everything that they've learned as well. And that way, we're not so beholden to a rug pull in the future. But that kind of gesture, I think, would be extremely beneficial. And it would not cost us anything at this point, right? It would just be a vision of a better future. We should lean into where trust doesn't exist. How can AI help us create trust?

• 2:43:52Nathan Labenz?

  That, I think, is a huge question. Right now, we're thinking that AI is going to be this thing that we try to have the best version so we can have supremacy. But what if AI could serve both of us? What if it could serve the relationship and help us really realize that eternal 1991 vision of eternal peace and prosperity through trade by actually taking a lot of the risk out of some of the trade that we've been too afraid to do in recent times? There's probably a lot more out there like that, but that kind of idea, I think, would do tremendous, tremendous things.

• 2:44:29Nathan Labenz?

  I said at the top, the main overarching goal should be a Pax Machina. What should we try to do in the near term for AI specifically? I think one of the big things we should try to do is maintain mutual legibility of our respective AI technology stacks and ecosystems. I think we dramatically underappreciate the fact that right now, both countries are basically building the same kind of AI on the same technology stack, the same paradigm. And the fact that they're publishing their work in English and sharing the models and we can run them and those models speak English and we can do interpretability on them.

• 2:45:13Nathan Labenz?

  We can hunt for backdoors in them, all these sorts of things. The mutual legibility of this technology right now, the fact that we understand what one another has, that is hugely, hugely valuable. If we get to the point where we decouple so much that the hardware is totally different. We're all on Nvidia and whatever else and they're all on Huawei. And now those things start to go different directions. Well, hardware and algorithm co-design is a huge thing. We talk a lot about the hardware lottery being a reason that transformers have been so durable. It's not necessarily that other algorithms don't work as well,

• 2:45:56Nathan Labenz?

  but it's hard to go a different direction because you've got just a lot of different optimizations that keep reinforcing the mainline paradigm. Well, if hardware starts to diverge and then hardware algorithm co-design starts to diverge, you could imagine over a not super long timeframe, especially as R and D cycles potentially speed up, especially with the early phases of recursive self-improvement. And especially if China stops publishing their models open weights, especially if they stop publishing their research in English or otherwise, as American companies largely have, you could imagine getting to the point where we don't understand what the other side has anymore.

• 2:46:41Nathan Labenz?

  And that, I think, would be extremely, extremely dangerous because that's when you can really make major miscalculations. You just have so much opportunity for there to be a actual power imbalance or a perceived rightly or wrongly power imbalance. We think our AI is better than theirs, so this is our opportunity to do something. And that could come from either direction. But what if it's wrong? It could lead to really, really bad decision-making and really, really bad consequences. I asked people about this a little bit in China. I said, hey, are you worried at all that if this decoupling continues,

• 2:47:23Nathan Labenz?

  there could come a time when we are on different branches of the tech tree to the point where we don't understand what each other has anymore? And it creates room for confusion, misunderstanding, catastrophic mistakes. And the response I got was not yet. And so I asked, well, when would you be worried about that? And the answer was, at the point where the Chinese chip ecosystem gets good enough that we're doing entirely our own thing, that's when that could happen. So I'm not super compelled in general by the arguments we've heard from people like Jensen Huang and David Sacks, that we want to hook the Chinese on our technology so they'll be dependent on us.

• 2:48:07Nathan Labenz?

  They are not going to be dependent on us, certainly not anymore. If we ever thought we were going to lull them into complacency on chips, that's over. We've made the move. They are absolutely committed to building their own chip ecosystem. And again, we should not give away the chips for nothing. I think there are things that we want and we should try to get them. Tech transfer and probably other flavors of benefit that we can get. I think they should generally take us toward renewed coupling and more dependence. But having them be at least somewhat dependent on the American chip ecosystem.

• 2:48:43Nathan Labenz?

  Would I think be really good, it would be another reason not to do something disruptive with respect to Taiwan. And in my opinion, maybe the most important thing is it would be a real force for making sure that we stay on the same tech tree and we stay legible to one another. I think it would be a real tragedy if we lose that because that's when we're really going to feel the need to race. It's one thing to say, we got to race because our LLMs are only a few points ahead of this benchmark relative to the Chinese LLM and they're going to catch up. Imagine a world where it was like, we don't even know the shape of their AI. We don't even know the architecture. We don't even know really what it's capable of.

• 2:49:28Nathan Labenz?

  We don't know what they have. It's a totally different paradigm. Imagine how scared we'd be of that. In that world, you might actually be right to say, maybe you need to race because you definitely don't want to be left behind. You definitely don't want to be on the wrong end of a decisive advantage. And if you don't even know what the other side is doing, what their technology looks like, tough. And by the way, they're going to have a much easier time maintaining that secrecy than we are for multiple reasons. The fact that we're able to assimilate immigrants does mean we're a lot easier to spy on. And I think we should also keep in mind, how did the Soviets get the bomb in the first place? Espionage.

• 2:50:09Nathan Labenz?

  We are much easier for China to spy on than China is for us to spy on. If 10% of China speaks English, they have half as many English speakers as we do. We have tons of Chinese citizens that live here. There's not a lot of Americans in China these days. So when it comes to who is going to be able to keep secrets from the other, I strongly suspect that they are going to be able to keep secrets much better than we are. And so to the degree that we kick them off of our technology stack and don't let them buy from ASML and now they have to build their own machines, perhaps without realizing it, you are betting heavily that the Dario worldview is like totally right, that everything that matters is going to be decided in the next one to three years.

• 2:50:54Nathan Labenz?

  In that timeframe, superintelligence gets developed. That's when the decisive strategic advantage happens. That's when the whole world order gets to be rewritten by whoever gets to superintelligence first. And if you don't believe that, or if you don't like that, then I think you should be like, not nearly as bullish on the idea that we can keep China down by denying them all these technologies because five, 10 years from now, they're absolutely going to figure it out. Don't doubt that at all. That will include stealing the secrets from us to the degree that they need to. And they won't even feel bad about it. And I frankly, don't have too hard of a time understanding their side of that story. So that's if you don't believe the Dario worldview. If you do believe the Dario worldview, and I kind of do believe it to a degree, or at

• 2:51:39Nathan Labenz?

  least I think it could happen, I would encourage you then to get on the side of pacing the frontier and figuring out, how do we actually take our foot off the gas so that that's not the world that we're racing into? There are a lot of ideas about this. I recently heard the idea of a one day pause. This could just be American companies, maybe it could even percolate over to Chinese companies. What if everybody just said, we're going to have a one day pause. We'll all take the day off, we'll all go to the beach and then post about it on social media and just say, hey, we're all enjoying a one day pause. It wouldn't really change the dynamics all that much, but it would make clear that there is opportunity for people to come together and just do stuff. It still is, at least for now, the humans doing the research that are determining the pace of the frontier.

• 2:52:24Nathan Labenz?

  The AIs haven't taken over yet, so we do still have time to change our approach. And I think the companies are showing signs of being spooked enough that they actually might take that approach. And I really think we should be encouraging that and we should be leaning more and more that direction. That I, OpenAI has, as of now, said that they're suspending their largest scale RL training. There've been a lot of calls for Anthropic to do the same. I would love to see Anthropic make a somewhat similar move, even if they don't feel like they need to as much or their stuff isn't quite as out of control. Right now, it does feel like a moment where they can either say, okay, you're paused, this is the time when we get to go create that lead that eventually takes us to decisive

• 2:53:05Nathan Labenz?

  strategic advantage, or they can say, okay, this is the time when we can slow down with you and we can reallocate more of our compute to interpretability or whatever. I really do hope they do that. And I think if we could get our companies coming together even somewhat, you know, it's kind of a proving ground for the US-China relationship. There's a weird way in which the OpenAI-Antropic cultural disconnect and lack of trust and feelings of, especially at the elite leadership level, of being hard done by, there's a way in which that echoes the US-China dynamic. So if we can see our top companies that have such bitter history come together, shouldn't

• 2:53:46Nathan Labenz?

  that provide a little bit more room to at least dream that the same thing could happen between the US and China? Okay, almost to the end. I'm old enough to remember when Demis Hassabis used to be advocating for an international AI safety collaboration. If I'm really going to allow myself to dream, I would like to start that conversation at the upcoming Trump-Gi summit. I like some of the ideas that I saw in AI 2040 from Daniel Cocotello and co-authors. They said, we should have this mutually assured deniability of compute. So like, the US should put our data centers in Mongolia and then China can put their data centers in Canada. And that way, if real hell breaks loose, each side knows the opposite party can destroy

• 2:54:31Nathan Labenz?

  their data centers and there's nothing they can do to defend them. I think that's interesting. I like the idea. I would say, you know, my broader criticism on AI 2027 and AI 2040 in general is that they kind of hyperstition this idea of the AI race, the same complaint I had about situational awareness in the first place. But I do like how they have moved now to putting forward some solutions. I would probably propose something even more jointly controlled, again with the strategic imperative of making sure that our AI paradigms remain mutually intelligible. I would love to see some joint research centers, some jointly controlled compute, which you

• 2:55:14Nathan Labenz?

  might place in Switzerland or Singapore and maybe try to have American and Chinese, and for that matter, global, groups of AI researchers come together and work together on shared compute where they can hopefully maintain at a minimum shared understanding and hopefully make real progress. These places also would naturally lend themselves to any sort of hardware verification systems that might get developed, where we could do pilots of those at these places. And I think these also could be interesting places for other mutual transparency measures to be explored as well.

• 2:55:54Nathan Labenz?

  I think it would be pretty compelling to try to create a auditor's bill of rights. The auditor's bill of rights would basically say, here are the kind of commitments that we, the frontier companies, are prepared to make to you, the auditing companies. And giving the auditing companies ability to use models to look at everything that's going on inside a company might be a little bit tough because there are a lot of trade secrets and people might worry about individuals outside of the companies gaining access to those trade secrets. But something that I haven't seen explored as much as maybe it should be is the idea

• 2:56:34Nathan Labenz?

  of using the confidential computing paradigm to enable better auditing. So I think it's a reasonably standard primitive on hyperscaler clouds these days to be able to create a computing environment where you can send data in, a process is run over that data, and you get the result back out of that process and in a way where you don't have to share the data itself. This is supposedly what Anthropic is doing when they do studies of how people are using Claude. Without humans actually looking at it, they send your data into a secure computing environment or a confidential computing environment.

• 2:57:18Nathan Labenz?

  They run additional prompts over these traces to characterize them. They get the results of that out, but they don't actually look at the raw information. I think it could be really compelling for the companies to try to make that kind of commitment to auditors going forward. Like, maybe we can't give you access to everything we're doing in the raw, but you can have access to log data, usage data, test results, poke around all of the information that we're generating with all the things that we're doing through this secure confidential computing paradigm. So you could have a model analyze it. There could be potentially an agreement on like, here's what the prompts should look

• 2:58:01Nathan Labenz?

  like, the mechanism for establishing and approving those prompts, there's definitely details to be worked out here. But like, can we, without actually sharing all the underlying trade secrets, gain confidence about what kind of work is going on, what intent it is? And could this even perhaps extend to governments themselves one day? Obviously, we're not going to share all the conversations in the Situation Room. But imagine a world where the Situation Room conversations were run through an AI system, again with confidential computing, so that you're not actually seeing the raw transcript of what conversation is taking place in the Situation Room, but you're able to say that

• 2:58:45Nathan Labenz?

  the model assesses, based on its understanding of the transcript, that the decision makers are not attempting to plan some sort of aggressive action. That could be a pretty low cost way for parties that don't trust each other to

• 2:59:09Nathan Labenz?

  Use compute and to use some of the latest technology to make a pretty credible commitment to one another that, yeah, we're not going to share everything we're thinking about, but we will allow an AI system to review what we're talking about and just assure you that we're not attempting to plan an attack or invasion or what have you. I think that could be really quite powerful. And so, again, what would it look like for both governments to allow AI review of sensitive documents in a way that the other side could trust is actually happening the way it's supposed to happen?

• 2:59:51Nathan Labenz?

  And that gives confidence that there's no major plot underway. There's no major ill intent. There's not going to be any big surprises. Could we do that in a privacy preserving way and build trust that way without sharing everything? Something like that I do sense is probably now technically possible. And it's the kind of thing that Anthropic and OpenAI coming together to agree on something along those lines could really set an amazing precedent that could ultimately take such ideas to places that right now it seemed like it would be impossible for these kinds of idealist ideas to travel.

• 3:00:32Nathan Labenz?

  Again, we would have to take steps to reduce the fear of antitrust. I would love to see the government make very clear that like, if you are colluding in restraint of trade to raise prices, to raise margins, to screw the American consumer, then yeah, we'll come down on you. But if you are trying to work together for safety purposes or to pace the frontier so that we're not racing into a disaster and you're finding creative ways to do that and they involve selective information sharing and trust building, this is the kind of thing we need more of in society between these companies and ultimately, eventually, I think between governments. If I really allow myself to dream a little bit more, I would love to see also some groundwork

• 3:01:16Nathan Labenz?

  laid now for a convention on autonomous weapons. Everybody kind of knows right now that we're in the beginning of a new era of warfare. Everybody has seen what has gone on in Ukraine with respect to how critical drones have become in the conflict there. And everybody has a sense that there's a paradigm change underway or coming soon. Can we be somewhat intentional about that? Is there any way to establish some no-go zones? Are there dynamics around the technology that are perhaps favorable for certain defense dominant

• 3:02:01Nathan Labenz?

  equilibrium to be reached? I think there maybe are. We see right now in Ukraine that it's very difficult to attack and take territory. I actually had Claude do a game theoretic analysis of this. And the way that it framed it was that the cost of denying territory to your opponent with drones is much lower than the cost of actually advancing and sustaining human beings in a conflict zone. So in practical terms, neither side can really take much territory from the other. So is there something inherent to the dynamics of drone technology that make it very difficult

• 3:02:43Nathan Labenz?

  to advance into somebody else's space if they're defended by drones, but don't necessarily make it as much easier to attack? I think that's speculative at this point. We're certainly seeing that longer range drones are being developed and they're taking the fight and the destruction deeper into Russia all the time from Ukraine. So I would say not only is it not clear, but I think we have not yet reached an equilibrium that we can bank on being stable for any period of time. But I do think governments could now start to come together and say, look, we see that there is a new paradigm. We see that it, at least right now, seems to be favoring defenders. And that's good because none of us want to be invaded.

• 3:03:28Nathan Labenz?

  Everybody wants to be secure in their own home. And there's probably some things that we can agree to not do that would be useful for attack and really wouldn't be that useful for defense. I'm not enough of an expert to know what those are, but I think in the same spirit in the AI space, it's quite clear to me at this point that it would be a bad idea to train large language models to go out and maximize the amount of money they can make on the open internet. If we do that, the economy is a competitive and at times adversarial place. What we're going to see is that the AIs are going to find all sorts of ways to cheat. They're going to become pretty quickly rather unethical. And these are not the kinds of AIs we want running around in our economy. We do not want deceptive.

• 3:04:12Nathan Labenz?

  We do not want cheating. We do not want collusive. We don't want all this bad behavior from our AIs. So let's not train them in this way that will obviously bring that on us. I think similarly, if we start training AIs for military purposes to say, your job, AI, is to go accomplish military objective X by any means necessary, or win this battle, or even zoom out, win the war. That's going to create AIs that are just obviously extremely, extremely dangerous, pretty much by definition. And we see such creativity, we see such surprising behavior from the AI systems we have right now, that I think we should both look at each other, US and China, and say,

• 3:04:54Nathan Labenz?

  hey, sure, we probably could start to train general purpose AIs to like, command our forces, accomplish military objective X, whatever that X may be. And it might work in the sense that we might actually get effective military capability out of it. But what we're learning from the civilian sector is that this is dangerous. This is a bad idea. And it's unwieldy for each side as well. There's potentially a lot of friendly fire. So can we just agree not to do that? We're going to need to build trust with steps for sure, because I don't think right now either side is in the mood or has enough confidence that the other side would honor

• 3:05:38Nathan Labenz?

  the agreement to make that sort of a deal. But I think that is where we probably need to be in the not too distant future. Because if we do create AIs that are optimized to succeed in adversarial environments, the economy for one, but God help us if that extends into outright war, no holds barred, we are absolutely in that case, training the AIs to do the takeover that we most fear. And I'm quite confident that the Chinese government does not want to lose power to AI systems. I know that I do not want to see the United States government lose power to AI systems.

• 3:06:19Nathan Labenz?

  So there's, I think, real shared interest there in avoiding an arms race that pushes us both to create this extremely dangerous technology before we understand it, before we know how to control it, with risk that it will turn right around and bite us ourselves, the developers. I think we both would be entering into that dynamic only reluctantly because we feel like we have no choice because the other side is doing it. So can we please be willing to dream for a moment of a future in which we can trust each other enough to make that kind of agreement and actually not go down that path? I would again note, we are not seeing

• 3:07:03Nathan Labenz?

  large-scale gene editing and super babies created in China. So if we're not seeing that, maybe we also won't see general purpose AI military commanders who will do anything and everything necessary to achieve military objectives. Because again, that's a bad idea. I don't want to live in that world. I don't want to live in a world where there's a bunch of AI systems that can take over and that were specifically trained to go conquer territory and destroy adversary armies. Because I know that can be used on me, it can be used on us.

• 3:07:45Nathan Labenz?

  And again, we just clearly don't know how to control it well enough to be confident that won't happen. So if we do create that, that is the AI sort of Damocles that I had in mind at the very beginning, that I would really like to see us avoid creating. And both sides, I think, in their hearts really want that. I think it's very few people who want to see the AI generals and AI admirals that can command their autonomous drone swarms to the point where they can overpower any human-run military. That sounds pretty bad. That's a hard game to win. And it's an easy game to lose. It's an easy game to lose in a pretty undignified way.

• 3:08:27Nathan Labenz?

  So I would really hope that both sides could see that and could.

• 3:08:33Nathan Labenz?

  Even though they might have some doubt about what the other side is doing, heart of hearts could be like, we don't really want to do this anyway. It really puts us in a bad spot, even if we're just doing it unilaterally. So maybe we can believe that they also sincerely believe that it would put them in a bad spot to do it unilaterally, and maybe we can both refrain. I don't expect that we're going to get in any immediate term, a real convention on autonomous weapons, but to start to say, hey, the kind of shit that we saw from OpenAI and Claude and Meta in just the last couple of months, where these AIs are given a test question, and then they go do all this crazy shit.

• 3:09:17Nathan Labenz?

  Maybe we don't want to bring that to our military systems. Let's both adopt that as our baseline attitude, and let's try to not do that. And let's try to not scare each other into doing that. I don't think that is really so crazy or so hard to imagine. Instead, what I would really love to see is some friendlier competition. I've proposed before the gold medal tracker for disease cures. China clearly loves a gold medal tracker. The US loves a gold medal tracker. We do have this friendly enough rivalry at the Olympics, but we all know that it's all for global entertainment and it's all for fun. Let's do something like that for diseases cured and maybe a number of other leaderboards that we could put together.

• 3:10:02Nathan Labenz?

  This is something that honestly, small amount of capital could catalyze into. This could be a unilaterally provided global public good. The leaderboard that just tries to incentivize these countries to compete on positive sum innovation and creation as opposed to more zero or negative sum competition in military technology. Who can cure the most kinds of cancer? Who can cure the most diseases? Who can make the most energy dense battery substrates? Who can get to the lowest level of traffic fatalities in their country in a given year? This could go on and on and on.

• 3:10:41Nathan Labenz?

  And this is the abundant future that we were promised, and I for one, am still very much hoping for. I don't care in the end if the cure for cancer comes from China or comes from the United States. I just want cancer cured. I think anybody who has faced down a serious illness, it would just be obviously insane to say, oh, I would only want to take that medicine if it was invented by an American. I would only want to take that medicine if it was manufactured in an American facility. This is an absurd notion. When the chips are down, everybody knows it really doesn't matter what language the person spoke when they were creating the breakthrough that's going to save your life.

• 3:11:25Nathan Labenz?

  So let's try to focus our attention on these positive sum initiatives, and we can compete there. We can compete for pride, for national glory and prestige. Obviously, there's going to be a lot of revenue attached to these technologies as well. So there's plenty of intrinsic reason. But that would be the kind of positive harnessing of the competitive dynamic between the two countries that I think could be mutually understandable, fun, and positive sum for not just the countries, but for the entire world and all of humanity, which again, I think should really be the goal of the entire AI era. Pax Machina is not a US phenomenon.

• 3:12:10Nathan Labenz?

  It's not a Chinese phenomenon. It should be a global phenomenon. And if the US and China can come together to make that happen, I really don't see any reason that the future is not as bright as we can possibly imagine it could be. Okay, that's it for now. Thank you for being part of the Cognitive Revolution.

• 3:15:04Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 3:15:48Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.