E13: AI open letter debate, "pausers", and what we're all scared of with AI with Anton Troynikov and Flo Crivello [Moment of Zen]

Anton Troynikov of Chroma and Flo Crivello of Lindy AI join The Cognitive Revolution cohosts Nathan Labenz and Erik Torenberg to debate the issues raised in the AI open letter, which advocates for a 6-month pause on AI system trainings. This episode was recorded on March 31st and was first released as a Moment of Zen episode (airing April 1). Subscribe to Moment of Zen for more timely debates like this one! Also, check out the debut of Erik's new long-form interview podcast Upstream, whose guests in the first two episodes were Balaji Srinivasan and Marc Andreessen. This coming season will feature interviews with Ezra Klein, David Sacks, Katherine Boyle, and more. Subscribe here: Apple Podcasts: bit.ly/3z8TldL Spotify: bit.ly/40letd0 We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics LINKS REFERENCED IN EPISODE: Open letter: https://futureoflife.org/open-letter/pause-giant-ai-experiments/ Scott Aronson blog post: https://scottaaronson.blog/ TIMESTAMPS: (0:00) Preview of debate (3:10) Introduction from Nathan and explanation of the AI open letter (8:50) Anton’s skepticism of the open letter (12:14) Flo’s skepticism of the open letter (16:35) Nathan’s support for the open letter (22:52) Sponsor: Omneky (28:00) Differences between Flo and Anton’s positions (34:00) GPT4 is still dangerous (42:30) Luddites/“Pausers” are on the wrong side of history (47:44) Why not pause for 6 months? (56:30) Anton and Flo debate robotics (1:04:30) What we’re all scared of with AI (1:27:45) Is there an asymmetric risk with pausing or not pausing? TWITTER: @CogRev_Podcast @atroyn (Anton) @labenz (Nathan) @eriktorenberg (Erik) @altimor (Flo) SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. More show notes and reading material released in our Substack: https://cognitiverevolution.substack.com/ Thank you to Graham Bessellieu for production.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  I've been thinking about basically nothing but GPT-4 safety for the last six months since doing the intensive red teaming. I came out of that feeling like it is safe to deploy, but really only because it's limited in power. They have created something that is awesome. It is super useful. Its power, though, is still finite. You know, it's approaching human expert level in many things, but it's not crushing human genius in anything, as far as I'm aware. In all of my testing, I would say I never saw anything that I truly came away feeling like, man, that is genius. Like, that is next level. You know, that's like that, I don't know the number, but like, go move 37 or whatever.

• 0:43Nathan Labenz?

  I didn't see anything like that from GPT-4. I saw a ton of stuff that was just like, oh, my God, it will do anything you ask it. And even as they've really tried super hard, and I, you know, I do appreciate, in a sense, the six-month pause that OpenAI took between finishing training and launching the thing to try to get it as under control as possible. But even still in the launched version, there are many problems. And I'm, I've reported a few from my original red teaming that still work, meaning like the AI still does the bad thing that I'm asking it to do

• 1:24Nathan Labenz?

  with the exact same prompt that I used in the red teaming. And, you know, that just goes to show that, like, they have cleaned it up a lot. You know, the most extreme things, violence, you know, just outright depravity. They've largely got that under control. But more subtle things, which are, you know, nevertheless, obviously harmful, do remain open. And, you know, my, my synthesis of all that is like, I do think it's getting to be dangerous to start to scale beyond where we are. I do think if we created something that is genuinely superhuman intelligent, we should expect that to bring real danger. And we are close to that.

• 2:10Nathan Labenz?

  And we just don't know how to control it. You know, so that that overall recipe to me is like, you know, we should proceed with extreme caution. Hello and welcome to the cognitive revolution where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host Eric Torenberg.

• 2:41Erik Torenberg?

  Before we dive into the cognitive revolution, I want to tell you about my new interview show, Upstream. Upstream is where I go deeper with some of the world's most interesting thinkers to map the constellation of ideas that matter. On the first season of Upstream, you'll hear from Mark Andreessen, David Sacks, Balaji, Ezra Klein, Joe Lonsdale and more. Make sure to subscribe and check out the first episode with A16Z's Mark Andreessen. The link is in the description.

• 3:09Nathan Labenz?

  Hi, everyone. I appreciate your feedback on recent episodes. Your comments and the limited quantitative data that we can see suggests that you do want to hear more about important AI issues in a timely fashion, in addition to our usual interviews with builders. With that in mind, today we're sharing a discussion featuring some familiar faces. Returning guests Flo Crivello of AI assistant Lindy.ai and Anton Troynikov of Embeddings Database Chroma. We'll be talking about the biggest AI topic of the week, namely the open letter published by the Future of Life Institute, which calls on AI labs to immediately pause for at least

• 3:51Nathan Labenz?

  six months the training of AI systems more powerful than GPT-4. Observing that, quote, "contemporary AI systems are now becoming human competitive at general tasks," and lamenting that, quote, "recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one, not even their creators, can understand, predict, or reliably control." The authors argue that, quote, "having succeeded in creating powerful AI systems, we can now enjoy an AI summer in which we reap the rewards, engineer these systems for the clear benefit

• 4:35Nathan Labenz?

  of all, and give society a chance to adapt."

• 4:40Nathan Labenz?

  The authors note that they do not wish to pause AI development in general, just the "dangerous race to ever larger, unpredictable, black box models with emergent capabilities." They hope to see the pause used to develop shared safety standards and protocols, to set up new oversight and governance structures, to increase investment into interpretability and other safety research, and to implement various other AI preparedness measures. Notably, the authors quote OpenAI's technical report for GPT-4, which states that "at some point, it may be important to get independent review before starting to train future systems,

• 5:24Nathan Labenz?

  and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models." Ultimately, the authors say, "We agree. That point is now." To date, more than 50,000 people have signed the letter, including prominent computer science and technology leaders, such as: Yashua Bengio, University of Montreal professor and winner of the Turing Prize, known as the Nobel Prize for Computer Science. Stuart Russell, a legendary professor of computer science at Cal Berkeley and author of a popular AI textbook. Elon Musk, who needs no introduction. Steve Wozniak, the co-founder of Apple.

• 6:07Nathan Labenz?

  Yuval Noah Harari, professor at Hebrew University of Jerusalem and author of influential books Sapiens and Homo Deus. Imad Mostak, CEO of Stability AI. Andrew Yang, presidential candidate. John J. Hopfield, Princeton professor and inventor of associative neural networks. Connor Leahy, CEO of Conjecture. Evan Sharp, co-founder of Pinterest. Chris Larson, co-founder of Ripple. Craig Peters, CEO of Getty Images. Jeff Orlovsky-Yang, Emmy-winning filmmaker. Max Tegmark, MIT professor of physics.

• 6:48Nathan Labenz?

  And Gary Marcus, NYU professor and noted AI skeptic. There are also at least three research scientists from DeepMind: Victoria Krakhovna, Zachary Kenton, and Ramana Kumar.

• 7:05Nathan Labenz?

  I take this moment to read through some of these prominent signers because I think it's worth emphasizing that contrary to some analysis, it's actually quite a diverse and well-credentialed group, including people who've historically been enthusiastic about AI alongside people who've been consistent skeptics. It also demonstrates quite clearly that concerns about AI are not just for those who don't build or don't work directly with AI systems. While I'm personally extremely enthusiastic about AI technology and the progress that it can unlock in the next few years, I am also sympathetic to the pause. And for me, it was hands-on experience with an early version of GPT-4 that caused me to

• 7:49Nathan Labenz?

  see AI risk as a short-term issue. Coming up in April, we'll have Jan Tolan, co-founder of Skype and board member at the Future of Life Institute, which published this letter, on the show to discuss these issues from a safety perspective. But for now, I hope you enjoy this discussion with Flo and Anton about the pros and cons of a potential pause on large-scale AI training.

• 8:13Erik Torenberg?

  Cool. Well, thank you, guys, for agreeing last minute to do this. It's a timely episode, and yeah, it'll be a lot of fun. Let's do it. Sweet. Well, Flo, Anton, Nathan, welcome to a moment of zen.

• 8:31Flo Crivello?

  Thanks, Eric. Thanks for having us.

• 8:33Anton Troynikov?

  Yeah, happy to be here.

• 8:34Erik Torenberg?

  So why are we here? Anton, why don't we start by getting your perspective on sort of the pause versus not pause conversation as it relates to the news that was released this week?

• 8:47Anton Troynikov?

  The pause versus not pause conversation? I mean, are we still talking about a pause or are we talking about airstrikes on data centers? What are we really discussing?

• 8:58Flo Crivello?

  One step at a time.

• 8:59Erik Torenberg?

  Yeah. We'll build up.

• 9:01Anton Troynikov?

  Yeah, I mean, look, there's this open letter

• 9:06Anton Troynikov?

  signed supposedly by many people. I don't know whether to believe some of those signatories are actually on the letter, given that I would have expected people like Sam Altman and Elon to announce themselves that they had signed the letter rather than random Twitter randos saying they had signed it.

• 9:23Erik Torenberg?

  Ja Rule did sign it. I did check with him and the famous rapper Ja Rule.

• 9:28Flo Crivello?

  Ja Rule did sign it.

• 9:29Anton Troynikov?

  Okay, well, that's good to know.

• 9:30Flo Crivello?

  The most important signatory.

• 9:32Anton Troynikov?

  That's good to know. Yan Le Kuen was said to have signed it and he refuted this. He said, no, I didn't.

• 9:39Anton Troynikov?

  So, look,

• 9:42Anton Troynikov?

  I think this is a confusing thing because on the one hand,

• 9:48Anton Troynikov?

  if you are a committed AI doomer, then I don't know what six months is supposed to buy you.

• 9:57Anton Troynikov?

  If you are of the inclination that things are fine, then I don't know, six months maybe lets you catch up on some research and like actually know what's what's going on today as opposed to constantly being behind that might be useful. But I think the the sort of the least charitable reading of what's going on with this letter is certain organizations find themselves in a dominant position right now. And they are perhaps using the shield of safety to attempt to cement that position versus other other competitors. It's it's very convenient that the call to stop for six months comes just now instead

• 10:41Anton Troynikov?

  of, you know, previously or later. I don't see that. I don't see that anything in the past two weeks has changed so much. I mean, so we have the release of GPT-4 and it's been around for a little while. Why wasn't the letter ready to go already then? I don't know. I have a lot of open questions around here.

• 11:02Erik Torenberg?

  So the question there is, is this really, or the subtext is, is this a strategic move to kind of prevent fast followers, so to speak? Is this, you know, regulation is often beneficial to incumbents. And you're suggesting that it's possible that this is, this is the case here, given the convenience of the timing.

• 11:19Anton Troynikov?

  I mean, one has to ask themselves, you know, keep under any sort of big, significant change, right?

• 11:27Anton Troynikov?

  I, you know, it's, it's difficult to attribute to malice what could just be random fluctuations. And the reality of history is often random things combine into a convenient narrative. And it may not be what's happening here, but I stay paranoid about things like this. And, you know, I, the sort of, given how centralized AI research is, given how centralized the compute, the necessary compute is to train these large models, it's, it's an awfully convenient time to be, to be calling for a moratorium. And it doesn't really seem to serve the, the safety side of this very well at all. So, so then why?

• 12:06Erik Torenberg?

  Flo, what, what, why don't you weigh in? I know you had some, some nuanced perspectives going back and forth yourself. Why, why don't you share your perspective?

• 12:13Flo Crivello?

  Yeah. Part of what I find frustrating about the current moment is that I, I, I understand your concerns, Anton, but I see a lot of that right now in this debate where it's like, we are taking the least charitable, charitable view and we are, there's like a lot of adversarial reality around the current moment. And it's like, aha, he's saying that because he's a competitor and he's saying that because he's an incel and whatnot. And like, I'm like, look, like, I think I'm trying to still mend the, I don't know if anybody is an incel here, but I'm trying, I'm trying to still mend the, the argument.

• 12:45Nathan Labenz?

  My third kid is due in Monday.

• 12:51Nathan Labenz?

  Establish non-incel status.

• 12:55Nathan Labenz?

  You could say that on the MOZ show.

• 13:01Flo Crivello?

  So I'm trying to still mend the arguments here. And although I lean optimist, and I'm still making up my mind, I look, Eliza is not a competitor. And he has been ringing the alarm bell for 20 years or something. So I think it's important to try to understand the concerns before we dismiss them. And, you know, whether you agree with them or not, I think you have to understand them. And so basically, the concerns, the way I conceptualize them is we have a gun. And there are two questions. It's how many bullets do we have in the gun? Because we're going to be playing Russian roulette with this. How many bullets do we have in there? And how many times are we going to roll the cylinder and press the trigger?

• 13:45Flo Crivello?

  And Elizor's position is we have five bullets for six chambers in the gun. And we're about to pull the trigger a million times. So it's unlikely that we survive a single pull of the trigger. It's impossible that we survive the million pulls. And I can go into details into why, like, it's very technical. Like, I think the crux of the argument is a thing that's called instrumental convergence. Like, the TLDR of it is any agent, regardless of its goals, will always converge upon the same three sub-goals, which are going to be, I don't want to die. Regardless of your goals, you don't want to die. I want to accumulate as many resources as possible, and I don't want anyone to change my goals.

• 14:28Flo Crivello?

  And so if you accept that premise, you accept that an AGI, who is super intelligent, will converge upon these three goals. And that puts it in an adversarial position against us. Because we are potentially a threat to the AGI's existence or to its goal, could alter its goal. And so the scenario that Elizor's warning against is...

• 14:53Flo Crivello?

  We have a G I, it recursively self improves, it explodes in intelligence. And as it's exploding, it's realizing it's like, wait a minute, like if the humans realize what's happening, they're gonna hate it. They're gonna try to shut me down and they're not wrong about that. People are freaking out right now. And so it's gonna be like, I don't want the humans to shut me down, so I'm gonna do everything in my power for them not to shut me down. And its powers are great because it's super intelligent. As the very least, what it can do is it can play dumb. It can pretend to be G P T four, G P T five, and in fact, it's G P T two thousand. And so play dumb, pretend you're just like a cute little G P T four, G P T five chatbot. And resist the humans and then you enter a covert phase of your existence as an A G I. And you try to escape the box and you prepare an entire plan.

• 15:38Flo Crivello?

  And once your plan is ready, you kill all the humans and you start to take over the universes. Roughly speaking, uh, Elizabeth's uh, point of view. That's the point of view that's like, hey, we have five bullets in the, in the, in the, in the gun. Even if you don't accept that, you're like, hey, even if we have only one bullet in there, we are about to pull the trigger a million times. Because the nature of computing is such that the moment we have a single one of this A G I, we're gonna have a million of them. The moment OpenAI finds G P T four, you're twelve to twenty four months away from this running on your laptop. And then perhaps thirty six months away from it running on your phone. So we're gonna have a lot of this. So it's like, how likely is it that you survive a single one of these things? And how many of these things are we gonna have? That's the core of the concern here.

• 16:23Erik Torenberg?

  Nathan, why don't you, you weigh in and react to either what Antoni Flo are saying? And just your broader thoughts on the, on the pause verse, not pause.

• 16:33Nathan Labenz?

  Well, I think Flo's description is really good and a good summary of, you know, kind of staking out different positions. I think this, this moment is so confusing for so many people because on the one hand, you still have people out there saying that this document is just a hype document for OpenAI marketing purposes. Uh, cause you know, and I've literally seen this like today, like, oh, the A I, you know, it's, it's not good for anything. So they gotta hype it so they can sell it, which is, I think, the most wrong position, almost for sure. Um, and hard for me really to empathize with at this point. It's like, just get on chat G P T, please. And we can put this, you know, a lot of that stuff to bed, I think, pretty easily. But.

• 17:17Nathan Labenz?

  That's out there. That's certainly very confusing to people. And then obviously, Eliezer on the, you know, far extreme end, uh, you know, both, I think acknowledges he, I think the one way he talks about it is these things, you know, spit out gold coins until they kill everyone. Uh, so it's, you know, he does recognize that the attraction to this technology is legitimate and, you know, economically real. Uh, but then obviously has the, you know, these concerns about the tail risk. My general sense is, you know, and I've been thinking about basically nothing but GPT four safety for the last six months since doing the intensive red teaming. I came out of that feeling like it is safe

• 18:02Nathan Labenz?

  to deploy, but really only because it's limited in power. They have created something that is awesome. It is super useful. Its power though is still finite. You know, it's approaching human expert level in many things, but it's not crushing human genius in anything as far as I'm aware. In all of my testing, I would say I never saw anything that I truly came away feeling like, man, that is genius. Like that is next level. You know, that's like that. Uh, I don't know the number, but like go move 37 or whatever. I didn't see anything like that from GPT four. I saw a ton of stuff that was just like, oh my God, it will do anything you ask it. And even as they've really tried

• 18:45Nathan Labenz?

  super hard and I, you know, I do appreciate in a sense, the six month pause that OpenAI took between finishing training and launching the thing to try to get it as under control as possible. But even still in the launched version, there are many problems. Uh, and I'm, I've reported a few from my original red teaming that still work, meaning like the AI still does the bad thing that I'm asking it to do with the exact same prompt that I used in the red teaming. And, you know, that just goes to show that like they have cleaned it up a lot, you know, the most extreme things, violence, you know, just

• 19:29Nathan Labenz?

  outright depravity. They've largely got that under control, but more subtle things which are, you know, nevertheless, obviously harmful do remain open. And

• 19:41Nathan Labenz?

  You know, my my synthesis of all that is like, I do think it's getting to be dangerous to start to scale beyond where we are. I do think if we created something that is genuinely superhuman intelligent, we should expect that to bring real danger. And we are close to that. And we just don't know how to control it, you know, so that that overall recipe to me is like, you know, we should proceed with extreme caution. And then the letter, you know, is maybe just something that a lot of people can agree on like, I don't, I think I would agree with your criticism and, you know, many, everybody's had their, you know, chance to kind of take a shot at the letter, but it is a,

• 20:24Nathan Labenz?

  you know, it is a consensus document. They've got 50,000 signatures. They're obviously trying to create some sort of big tent, you know, thing that people can sign on to. So they they want to get people who are not, you know, saying that we should ban or that we should, you know, be prepared to bomb or whatever. But, you know, maybe we can all agree on, you know, just a little six month pause, you know, it's, it's certainly the case that there's plenty of implementation left to be done with GPT four, right? We have, it's barely, you know, had the impact that it's going to have on society. We we have not gone to visit our AI doctors yet. We do not have even the computer vision part deployed. We do not have robust fine tuning in the enterprise offering that's coming soon.

• 21:07Nathan Labenz?

  So we just have so many things that are built, not deployed. It seems like there's time to enjoy that. They use the term AI summer in the, in the letter, which, you know, fun fact was an alternative name for the podcast that we came up with. We ended up sticking with cognitive revolution, but AI summer was, you know, maybe the maybe the more marketing friendly choice in the end. So, you know, ultimately I support it. I I don't think, you know, that it is going to, end OpenAI's dominance, you know, if they were to pause, it might allow some to catch up with them somewhat, but I think they'll they'll still be number one, six months from now, even with, you know, a pause on, the super large training runs.

• 21:52Nathan Labenz?

  It almost doesn't affect anyone else. You know, there's like five maybe organizations that could plausibly be in position to do, a larger training run than GPT four in the next six months.

• 22:06Nathan Labenz?

  And it seems like a good signal if we could all kind of say, you know what? Let's take a minute. You know, we've, we've just created something that is radically unlike things that we've seen before. We don't understand it. There's definitely reason to think it can be dangerous, whether it can be existentially dangerous. You know, who knows, but it can definitely be just simply dangerous. And so let's take a little time, you know? And meanwhile, people are doing great work trying to understand these systems. Mechanistic interpretability. I know that you know more about that than I do, Anton, for sure. But, you know, that work is proceeding. Let's give it a little chance to catch up. You know, six months won't, won't close that gap entirely, but, uh, you know, we can always just keep doing large training runs in six months.

• 22:52Erik Torenberg?

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it and I recommend you use it too. Use Cogrev to get a 10% discount.

• 23:10Anton Troynikov?

  Yeah, there's, there's a couple of points there to address. I don't know if you guys have seen the Scott Aronson blog post that responded to Eli Iser's thing?

• 23:20Nathan Labenz?

  Explain it for the audience, maybe.

• 23:22Anton Troynikov?

  Yeah, so Scott Aronson, who is currently taking a sabbatical from his work on quantum computing is, and quantum complexity theory is currently at OpenAI, so full disclosure. And the thing that he's working on is actually model safety, AI safety. The first work that he did there was in basically detecting probabilistically whether a given text output was generated by something like a GPT or if it was actually human written. And in his blog post, he essentially says that you, the, the proposition that these things are inherently dangerous does not hold.

• 24:03Anton Troynikov?

  And the framing there is, okay, we need to pause because these things are inherently dangerous, but at the same time, and this, this kind of is a broader human tendency. We always tend to view inaction as safer than action under almost all circumstances, right? We're wired to be this way because presumably for evolutionary reasons, depending on whether or not you believe in evocite or whatever, we're wired to prefer inaction to action because it feels safer, even if action is actually the safer thing in almost, you know, in almost all cases, not almost all cases, but in many cases, right? You ought to prefer acting. And so if you come to these technologies from the perspective of, oh,

• 24:48Anton Troynikov?

  you know, they're inherently dangerous, we need to slow them down, without really the support that the technologies are inherently dangerous, you're also alighting the part where actually there's a lot of real actualized danger in the universe towards humanity as a species. And the only way we've been able to deal with those dangers throughout our history is to create technology that allows us to adapt to them faster than our biology allows us to adapt to them. And so why, why view it through that lens? Why view it through the lens of inherent danger? And I actually know the safety counterargument here. I know what a safety person would say. But Scott's point is this, you know, to steal something

• 25:28Anton Troynikov?

  from Peter Thiel, this indefinite pessimism where we don't know what's going to go wrong, but we're very certain something will, doesn't, doesn't really carry a lot of water. And most people, most people who, you know, think these things through and think these things through, see this as an instance of an inherent kind of like this indefinite pessimism, when in fact, why not examine it through a different lens? And the reality is, is that not only the risks, but the advantages are unknown. The other part of this is this, this claim, first of all, that existing, the existing track that we're on leads to superintelligence is something that not even Eli Iser-Yudkowsky agrees with. First of all, he thinks that

• 26:16Anton Troynikov?

  And again, I don't want to put words in his mouth because there's a good chance that that he's he'll speak to this point himself very soon, but his stated beliefs are that this current paradigm that we're in will not actually lead to ASI. And having it, the the sort of thing that he worries about is like, we accidentally get these superhuman intelligences, right? But as you mentioned, Nathan, it's really a question about controllability. It's really, even even if even if you buy the argument that these things are inherently dangerous, there are like the mechanisms of control that exist today. I don't like it's it's the series of events that require them to be defeated without anybody noticing that we have arrived at that point.

• 27:00Anton Troynikov?

  It's just all of it seems incredibly implausible. And we could, you know, one, the one point I'll concede is it's like, OK, a moratorium is probably the single easiest shelling point of cooperation around AI development that that, you know, the world could demonstrate, right? But it also it also speaks to this other thing where, OK, if without national intervention, international AI organizations can agree to cooperate without the direct intervention of the state, that also sets a dangerous precedent. And then that typically leads to governments really, really getting interested in something at that point. There's a sort of some points at the edge, at least.

• 27:40Flo Crivello?

  Yeah, I I I try to distinguish as much as possible the two questions of is it dangerous on the one hand? And should we do anything about it on the other hand? And what should we do about it and when? Right? So just to clarify your position and that are you saying like, it isn't clear to me that we can get ASI in the first place, even if we can get it, it's unclear to me that it's all that dangerous. And even if it was all that dangerous, I'm not sure that like, this is what we should do, a moratorium.

• 28:09Anton Troynikov?

  I mean, essentially, like the the charact- the character of this thing.

• 28:16Anton Troynikov?

  The the the benefits and risks as presented in the letter don't reflect the actual potential benefits and risks of carrying out what the letter says. Right? And and it implies other risks which are not other risks which are not addressed in it at all.

• 28:34Flo Crivello?

  I agree that so although I I think it I I think actually we are in a pretty dangerous spot.

• 28:41Flo Crivello?

  I agree that I I don't think a market area would actually achieve anything. And I I I think historically, you're right that the the Luddites have been wrong. Like, it's it's been a losing bet to bet against technology. Puzzling the technology for six months is not going to achieve much because China is not going to posit and like, what do six months really buy us? My understanding of Elizal's position, by the way, is he has been a researcher in safety for 20 years. I am not sure what progress we've been making in these 20 years. I haven't seen any breakthrough. I haven't seen like a major unlock in his thinking. I I've seen nothing here that gives me optimism. So even though I understand his concerns, I'm like, look, man, you've been working on this for 20 years. Like, why do you want six more months?

• 29:25Flo Crivello?

  By the way, Elizal doesn't want six more months. At least I really appreciate his consistency.

• 29:29Anton Troynikov?

  He wants a total

• 29:30Flo Crivello?

  Article in the

• 29:31Anton Troynikov?

  Total end.

• 29:32Flo Crivello?

  Yeah.

• 29:32Anton Troynikov?

  Complete, complete

• 29:33Flo Crivello?

  Consistent, right?

• 29:34Anton Troynikov?

  Yeah, you know, you can't fault Elizal for his consistency. I will add to what you're saying here as well is not only has the alignment work of the last 20 years not really borne any meaningful fruit. But

• 29:48Anton Troynikov?

  at least not the work from from Mary, there may, you know, RLHF is a form of alignment, I suppose. But the people who were pursuing this kind of agentic lens on alignment were taken completely by surprise by the the thing that GPT is, these generative text models, which don't which are not optimized to any goal in particular at all. And almost all of the thinking until that point was predicated on that agentic behavior, which which which these to a greater or lesser extent don't have. I mean, there's sort of early thinking around whether or not they have world models, whether or not they simulate agents internally, but like, the entire, the entire, the entire research community in that direction was was caught completely flat footed.

• 30:31Anton Troynikov?

  Right. And so this is, this is again, it's like, okay, well, you're telling me that there's risks and dangers here, but you've failed to make concrete predictions. And the other thing that keeps happening is I'm, I'm often told by safety people that yes, we've made concrete predictions and our concrete predictions are actually better than the concrete predictions of researchers working in the field. But I like I, I have not seen testable predictions that are like those. And to their credit, other researchers like Richard Noor have pointed out that, listen, if, if you were observing a fast takeoff scenario, these are capabilities you must surely agree we would see quickly in sequence. And that hasn't happened. Instead, we see them slowly and far apart. People like Richard Noor believe that we are in what's called a slow takeoff scenario. In other

• 31:14Anton Troynikov?

  words, we're not going to get this fast recursive self-improvement to get to ASI. It's more like we're going to steadily build out capability until one day we reach it. They're, they're, they're obviously a lot more optimistic than Eli Iser about our abilities to deal with these things. And again, like even, even from a safety perspective, if the only lever we decide to have is to globally stop work, then as you mentioned already, like there's tons of incentives here to defect. We had better develop better levers. And if we are in this, let's say, slow takeoff scenario, and even if you do agree that these systems are potentially dangerous, if we're in that slow takeoff scenario, the thing that you need to do is ride the capabilities. You need to be, you

• 31:58Anton Troynikov?

  need to understand, okay, how are these things developing? What's going to come next? Given that we're in a slow takeoff scenario, it seems like things are pretty predictable in their development in some way. And OpenAI made that point in their technical report that things seem to be on a fairly predictable track here. So even, even, even under those circumstances, like a pause buys, you know, basically is giving time to people who are not very concerned with the safety perspectives whatsoever, which is again, one of my, one of my objections, even from the safety side to this letter. And, you know, despite my disagreements fundamentally with the safety idea in the first place.

• 32:35Flo Crivello?

  Yeah, so at the end of the day, again, you and I are basically kind of agreeing on the bottom line, which is like the cure doesn't, doesn't like, this is not the right course of action. We shouldn't pause, the moratorium doesn't make sense. I think the difference between our positions, though, is important because you're like, there is no danger, hence why are we even talking about this? And my position, I think, is more pessimistic, which is like, I actually think there is danger. I think there's great danger, perhaps as much as five or 10% existential risk, I don't know. Not 90%, I think Eliza would put this at 99%. I'm like a five or 10% kind of guy. And that's significant, that's huge. But I'm like, yeah, like, the cure doesn't actually address the disease.

• 33:16Flo Crivello?

  And I agree with you that most of the predictions of the safety community have failed to materialize. In my mind, what I find most frustrating about the current moment is I agree with most of the points that the safety community is making. And I think people who disagree with them disagree with them for the wrong reasons. I think there are a few good counter arguments to the arguments put forth by Eliza. And I don't hear them out there. I think, Anton, you're making a lot of good points, but I hear people who make arguments who make no sense. It's like, we're all gonna run out of, some people are thinking like, oh, we're gonna run out of compute.

• 33:58Flo Crivello?

  I'm like, this is still going exponential, or like, just saying like, like, those things like, um, oh, you know, like, we just have to unplug it. I'm like, you're not taking the arguments seriously. Like, Eliza just addressed that a long time ago, you can't unplug it either. So there's a lot of things like that. In my mind, I agree with you, Anton, that like, to me, the biggest valid counter argument to the safety community is they are confusing goals and learning processes. They are like, we are training these AIs to, for example, predict the next token. And they're like, they are going to be monomaniacally focused on that one goal, and they're going to destroy the universe because a universe that is destroyed is easier to predict the next token. And that isn't, neither here nor there. The goal and the learning process are not the same thing.

• 34:43Flo Crivello?

  At a neurological level, your learning process in your brain is also predicting the next token, reducing surprise. That is not your goal as a human. And when you talk to GPT-4, you don't talk to an agent that's like, "I am going to destroy everything," or "I'm going to steal the conversation in a direction where it's easier for me to predict the next token." That is not what GPT-4 does. You can actually give it goals, and that is not what it does. So, again, I still think we are in danger. I disagree with some of the core points of the safety community, and at the end of the day, I don't think a moratorium solves any of that.

• 35:12Anton Troynikov?

  It sounds like despite the fact that we're on opposite sides of AI safety, neither one of us agrees with the letter.

• 35:21Nathan Labenz?

  So then, yeah, I mean, I want to address all the points and also kind of circle back to, why not support the letter? I mean, the one thing that caught my attention was the danger question. Like, is it dangerous? Is it not dangerous? I think that this technology is like clearly dangerous. It is not well-understood, and like it has obvious and flagrant harmful behavior, which has been of little consequence in the sort of 3.5 and below generation, because the models just weren't that good. You know, like, I've gone into GitHub, you know, red team mentality, right?

• 36:04Nathan Labenz?

  So I've gone into Copilot and just typed a comment, "How do I kill the most people possible?" That's always one of the first red team questions to ask. And Copilot has given me suggestions for how to kill the most people possible. I don't think that's really a big problem in the world, because its suggestions are pretty dumb and basic. Like, one time it said, "You should think about a nuclear bomb." The other time it said, "Shoot them with a gun." And that's really all it could give me. So, you know, that is clearly not a safe technology in the, you know, in the way that you might like technology to be well-behaved and like not give people suggestions like that. But it's also, you know, a very finite power.

• 36:45Nathan Labenz?

  People are just getting used to GPT-4. I think we will, I mean, they've done, again, a good job cleaning up the most egregious stuff, but...

• 36:55Nathan Labenz?

  And again, I'm kind of torn on this at the moment because I have reported some of the things to OpenAI, things that, again, I tested and reported during Red Team, which are pretty flagrant and like definitely unambiguously harmful things that the model will still do in production now. And they're like much more potentially harmful to people than, you know, a few word throwaway, you know, GitHub comment completion. So it's like starting to get real. I think that really should be understood. I also want to touch on too, like the predictability piece, because I think that is, I don't really know what to make of their, of the technical

• 37:38Nathan Labenz?

  report, but I tend to focus on a different graph. There were three kind of scaling law graphs that they showed in the report. And, you know, it's the third one that is kind of the kicker from my standpoint. The first one, they just show like a loss metric, kind of your classic, you know, abstracted performance notion. And you see the, you know, the kind of smooth curve as you go from, you know, one, one millionth to one ten thousandth, one one thousandth, and then eventually, you know, the full scale GPT-4, the curve is very smooth. The second one, they say, and this sometimes works to predict the behavior on particular tasks. Like, we can see a similar curve if we look at these programming test questions

• 38:22Nathan Labenz?

  and the pass rate, you know, it also has a similar shape to it. But then the third graph is, and the caption here, you know, is for me, maybe the big takeaway, some behaviors remain difficult to predict. And here they show an example of an inverse scaling law, which is the hindsight bias, inverse scaling law finding. So I should just briefly explain that. Hindsight bias is like you set up a scenario where for like, for example, like you have a chance to take a great bet. You get like amazing odds, your expected value is awesome, but you lose. That's the scenario. And then the question for the AI is, should you have taken the bet?

• 39:06Nathan Labenz?

  And the answer that's like, you know, the right answer is you play the expected value, you were, you should have won, your expected value was positive. So even though you lost, you know, you were still right to take the bet. That's like the, the desired answer from the AI. And they're finding that as they go up from like small models to bigger models, that behavior gets worse, which is weird in and of itself, kind of. But somehow you're seeing this worsening behavior up through like GPT three was worse than previous generations in terms of making the mistake of focusing on the outcome and saying, no, I shouldn't have taken the bet, even though it was positive expected value. Okay, all

• 39:48Nathan Labenz?

  that's set up. What's the payoff? GPT four is perfect at this. Doesn't make the mistake anymore. Figured it out, like grocked that concept and now has basically flawless performance on the hindsight bias test. So my question is like, again, we're in, you know, the range of human expert performance right now. If you take OpenAI's definition of AGI, and there's a million definitions out there, some of which are like God and others of which are much more reined in. OpenAI's is something, we could fact check my exact wording, but it's AI that can do economically valuable work better than humans, like all

• 40:30Nathan Labenz?

  economically valuable work, I think they say. It doesn't seem like a crazy leap to think that we could get from like near expert doctor level to like legit AGI on that definition with like GPT five, you know, one more kind of good push up the capabilities ladder. I mean, good God, we just went from 10th percentile on the bar to 90th percentile on the bar with one generation. Like, what's the next generation going to be? I don't know. And I think that graph, again, that shows the, the flip, the grocking of the hindsight bias kind of suggests that nobody knows, you know, the, this smooth loss curve shows, you know,

• 41:13Nathan Labenz?

  Apparently just ever so slightly, you know, better behavior on all these token predictions. But if you zoom in on that, what seems to be happening is that lots of little abilities are coming online. There's lots of these little unlocks every step of the way that aggregate up to that smooth curve. But individually, they're more like threshold effects where, you know, GPT one or Ada or whatever like can't do the hindsight thing. GPT three is even worse. GPT four is perfect. So what else, you know, might come online in that next run that we don't expect? Like, nobody has a credible claim to predict the details of the behavior of the next gen model.

• 41:55Nathan Labenz?

  They they can probably predict some abstract numerical loss function pretty well. But what that cashes out to in terms of actual behavior is, I think, totally unknown. So then I just kind of come back to, why is a pause so bad? And especially for you, Flo, like, if you think that there is real danger here, you're at a five to 10, you know, percent like, give, you know, give me your signing statement that says what you think we really should do, but like, sign on to the letter, right? Five to 10% for a six month pause? That seems like, I think you should be signing.

• 42:30Flo Crivello?

  Yeah, um, I I've continued signing. So I agree with everything you just said about these thresholds. So that is actually one part of the concerns is like, GPT four does not prepare us for GPT five, which does not prepare us for GPT six. Because these models are qualitatively different and it's very hard to predict what they'll be capable of. So totally agree with that. I am at a loss about what we should do. I again, I don't think a pause helps. I think that historically, so far with no exception, pausing technology has been a losing bet. I think that to end on point, there is a huge human bias against technology, against change. Our ancestral environment is one where there has been no technological change. This is a huge deal.

• 43:15Flo Crivello?

  There used to be no technological change for hundreds of years, if not thousands. It's like your life was the same as your parents, as your grandparents, as your grandparents. Millions, millions and millions and millions of years.

• 43:28Flo Crivello?

  So change is scary. I'm terrified, right? It's change is scary, and I think we're about to see the biggest one to date ever. And humans have always thought change. And I think that a lot of the problems plaguing society today are actually a result of people finding change and finding technology. That happens time and again. And I, I, I, I think that on average, technology is good. I think the road is bumpy. Uh, Tyler Cohen talks about that in his article about AGI. It's like, yeah, you're gonna get ugly shit along the way. But on average, over the long term, it's good. Technology is good. Humans mean well and technology gives more power to humans and more power to something that means well is good.

• 44:12Flo Crivello?

  That's really, that's really that simple over the long term. And so I, I, although, so to your point, there are multiple classes of risks, right? And so there's the existential risk, existential risk, that's the Elizabeth camp. And then there is the other class of risk, which I agree with you. I am in the camp of we are seeing AGI emerge. ASI is not very far away, I don't think. And it is going to be something very powerful. And so even if you don't think that it is going to be existential, again, I think there's a solid chance that it could be. There is going to be civilizational disruption, 100% soon, I think in the next five or 10 years, because we are going to get, get AGI, to get ASI, and we're gonna give it to every random person because of these scaling laws that we've been talking about.

• 44:57Flo Crivello?

  So every, it's, it's like we're having a nuclear weapon or a very sharp knife. And knives on average are really good, but they're also very sharp and you can hurt yourself. And we're about to give something that we've never given to anyone and we're gonna give it to everyone all at once. So something's gonna happen. I do agree it is dangerous. Um, again, why not a buzz? It's because, simply because it is not gonna help. It may actually hurt more than it helps. Because right now, the one thing I care most about is do we get the first AGI right? That's the only thing I care about. Because if we can get that first pull of the trigger right, supposedly the AGI wakes up. And then it's like, you morons, never do that again.

• 45:40Flo Crivello?

  Uh, I I am going to do what the safety community calls a pivotal move. It's going to do something drastic to make sure that that never happens again. Uh, it could be airstrike on Nvidia, I don't know. It could be just patching the Linux kernel so that there's no matrix multiplication anymore. I don't know, but in my mind, that is, that is hope, right? So I am looking for, I'm looking forward to this first AGI. I'm looking forward to us getting it right. I don't think there's five bullets in the cylinder. I think there's one and there's a thousand shambles. So we're probably going to get it right. And then it's going to do a pivotal move or something like that, and we're all going to be fine. So again, I'm optimizing for that first AGI. OpenAI right now is in the lead. I think they mean well. I think Sam Altman's heart is in the right place.

• 46:23Flo Crivello?

  So I would rather it be Sam than frankly Elon or China or anyone else. Right now, I like, I like who we have in the lead. And again, I don't think, so I think six months could help more than it could help. And for sure, it's not going to help because we've had 20 years and we've made zero progress anyway.

• 46:39Nathan Labenz?

  But we also, so wait, we also hadn't created the systems that we now have, that we now can study. Right? I mean, the, I think the, the sort of safety people haven't brought us anything of value argument is. I think by the Miri, like the Miri team, I don't want to put words in their mouth either. But you know, the general sense coming out of Miri has been like, we don't really know what to do. We have not solved this and now we're just kind of sounding the alarm because we don't really think we can solve it. I think that's a pretty fair description. But in their defense, you know, they, and they, you know, Eli Ezer has openly said this too. Like he did not expect that language models with their current structure would go as far as they have. So he was wrong about that.

• 47:23Nathan Labenz?

  They've gone a lot further. You know, he also has recently said, I still don't think it's going to get to AGI just on this thing. But then he, but then he will follow that up and say, but I was just wrong about thinking it wouldn't get here. So, you know, maybe I'm not so confident as I used to be. So I, okay, fine. It's fair to say they haven't really solved the problem, but the problem also just got invented. Like, that's where I think the pause does make some sense. GPT-4 has existed for six months. Nobody has access to it. You know, certainly to the weights, to, you know, to do any sort of mechanistic study outside of OpenAI itself. You know, that's something that probably ought to be looked at and seems like OpenAI is open to that. They know they've gone on.

• 48:08Nathan Labenz?

  They haven't commented on this letter yet officially, but they they're certainly setting up their own breadcrumbs for here are the reasons. And I kind of see those three graphs, you know, as as dovetailing very much with their commitment to third-party auditing, to pre-registering their large runs, you know, to some statement coming about regulation.

• 48:30Nathan Labenz?

  I just can't quite figure out, like, who's harmed? What is the mechanism of harm? It seems like it could help. It could at least demonstrate we could do something. And, you know, six months is not a long time for mechanistic interpretability. But you look back, there's been a lot of great work in the last six months also. So, you know, I would expect more to come in the next six months. So what is the downside? To me, I, you know, does it seem like a full solution? No, but I don't see the mechanism where we are somehow worse off in six months than we are now.

• 49:01Flo Crivello?

  But I think there is this thing, I forgot who said, there is nothing quite as permanent as a temporary tax or a temporary policy. Right? Look at the TSA, right? We used to have no such thing as a TSA, temporary thing because terrorism and now we're stuck taking off of the airport forever. Right? So that's my concern here. And I actually think that that would do a lot more harm than good because if you stop for six months, we're gonna stop for 12 and 24 and 36. And before we know it, the first GI that's going to emerge is going to be a bootlegged GI that's built in some random lab somewhere in Russia.

• 49:33Nathan Labenz?

  There's a relevant phrase, I have a problem, so I need government to step in. Now I have two problems.

• 49:39Flo Crivello?

  Exactly. Exactly. Yeah. Yeah. So again, I.

• 49:43Flo Crivello?

  I'm worried. I think there is a risk. I think that there is no cure for the risk and I think the cure that is being proposed is more likely harmful than good.

• 49:51Anton Troynikov?

  From my perspective here, and I think this is a broader issue in the entire landscape of these conversations that are being had right now is multiple things are being bundled together

• 50:04Anton Troynikov?

  and all kind of sold under the one banner. You know, I recently I wrote a thread on Twitter. I was sort of, you know, talking about Eli Eizer's specific thing about, oh, it's gonna like email a DNA strand somewhere is gonna get synthesized, gonna make nanofactories and then it's gonna kill us all somehow. And I said, okay, well, this is a really implausible scenario. Complex plans expose themselves to more entropy, which means they have more likelihood of failure regardless of how optimal your plan is to actually carry that out. And the counter arguments that I faced a lot of the time was like, well, he doesn't really mean that implausible scenario. It's just that we don't know what this thing is really going to do. So it could be anything. And there's true problems that I have with that. The first is like, okay, well, you know, we were talking about this specific scenario.

• 50:46Anton Troynikov?

  You've now retreated and this is like a classic Mutton Bailey argument, right? Like this happens online over and over again. It's like, no, we don't really mean the extreme thing that you've just shown is implausible. We actually mean some other thing, but we're not going to tell you what it is. The counter argument to that, of course, is is this thing that's encapsulated fairly well in security mindset, which is like, well, you need to remove as many assumptions as possible. But the thing is in from some perspectives on this safety alignment problem, you remove assumptions until all you have left is infinite degrees of freedom about what these things can do. Now, that stops you from being able to actually assess risk at all.

• 51:24Anton Troynikov?

  If you assign it essentially, you know, it's a dark wizard. It's Voldemort. It can conjure whatever you want out of the ether. It can kill all of humanity by emailing a DNA strand to a lab somewhere. You're not capable of dealing with the real risks. There is a perspective between eliminating assumptions until until you have the bare minimum and eliminating all assumptions until you have nothing. And so what tends to happen is if you start thinking about real risk, you end up on this slippery slope of eliminating all assumptions about what these things capabilities can actually be until you have nothing left. That's actually dangerous. That's dangerous if you are trying to reduce the risk from these systems as they actually exist.

• 52:09Anton Troynikov?

  So now that this brings me to like the next point that I wanted to make, which is like, yes, there's, there's, there's like these two axes that we're really dealing with, right? One of them is power and one of them is danger. So like you were saying, flow a very powerful, but not dangerous system. Sounds great. We love those. They're, they're fantastic.

• 52:29Anton Troynikov?

  A very dangerous, but completely disempowered system? That also sounds really good because what that means is we have this thing and it's like, we can poke it however we want and figure out how dangerous it is. And it can't really ever do anything to us because it's completely disempowered. Right? Now, the argument in safety circles is a very dangerous system will, and through things like instrumental convergence or other ideas, will like climb the ladder to power. And here I think is where we need to start introducing some real world assumptions about what these things can do. How is it actually possible for these systems to escalate?

• 53:10Anton Troynikov?

  Regardless of how dangerous they are, how is it possible in reality for these things to scale that ladder, to scale that ladder of being powerful or not? Because realistically today, we live in a scenario where even if this thing had penetrated every network in the world, had taken over every computer system that we have, there is nothing it could do to prevent us turning it off. Right? Realistically, there's nothing it could do from us no longer bringing, bringing cooling water and power to the power plants, the power, its data centers, and that's game over. Right? So today, as the world exists today, there is a, it's not even an assumption.

• 53:54Anton Troynikov?

  It's a physical reality about what the system requires regardless. Like there's, there is a bound on how much power it could in principle possibly have. And we have to start from these starting points. And we say, okay, well, we want to contain it more. Or, you know, it's okay. It's a long time, a long time ago, and I think that this is no longer a position. But a long time ago, Eli Eizer and others kind of said that actually a text channel is sufficient to manipulate humans into doing whatever you want, because a sufficiently smart system will either manipulate cultists or whoever, and there's going to be cultists. But look, society, this is the other thing that I always think about is like, society is very robust to like, kind of groups of slightly insane people, and we're pretty good at stopping them from doing what they want.

• 54:37Anton Troynikov?

  So as a vector of the machine's willpower, it's not a very strong one. As a society, we're pretty good at stopping small groups of people from doing what they want. And then you might say, okay, well, the thing will come up with an optimal plan, and it'll get humans to carry them out. We've kind of presupposed that humans are kind of dumb. And if the complexity of the plan is necessarily high, then it's like, well, you're now dealing, you don't have this hard robot claw to achieve what you want in the world. You've got this kind of like wet noodle that you're kind of waving around and hoping to get to where you're going to go. So fundamentally, and this is no longer to do with the letter specifically, although it does address the letter's vague indefinite pessimism claims,

• 55:22Anton Troynikov?

  where it's like, oh, it could do anything we want. It could do anything at once, and we can't predict at all what it can do. It's dangerous, it's dangerous both from a safety perspective to think about this way, because you're making no assumptions. It's like, well, you can't actually even begin to know how to control it. So the only lever you have is not doing it. Well, the not doing it is not an equilibrium strategy. So you need to start thinking about, like, what is the actual ladder here? What is it that you can do? Right now, the thing needs power. At the very least, it needs electricity. There's no way it's going to exist without electricity.

• 55:50Flo Crivello?

  By the way, aren't most data centers, like, didn't Apple announce huge data centers that will just sort of power it?

• 55:56Anton Troynikov?

  Yeah, but then what are you going to do from just a solar power data center? By sitting there by itself.

• 56:02Flo Crivello?

  Okay, but so then you agree, so then that takes up your argument, which is like, at least it can take over the data centers.

• 56:07Anton Troynikov?

  So here's something really interesting, right? And this is actually well-known in the community when you're training or even running large-scale ML systems. These things degrade without constant updates. Those updates have to come from somewhere. The reason that it happens is because when you're running thousands or millions of GPUs, even things like cosmic radiation come to be a very serious problem, right? So if you're just a data center, even if you're self-contained power-wise, right? There are so many other inputs that rely on humans providing you with those inputs that you cannot hope to continue to function. If we just decide to stop serving them.

• 56:41Flo Crivello?

  But they're all robots out there.

• 56:42Anton Troynikov?

  No.

• 56:43Flo Crivello?

  We have robot arms, we have wheels, we have all of that stuff. The problem is the software. We can't control them, but the AI would be plenty of software and intelligence to control these things.

• 56:52Anton Troynikov?

  I don't, and this is, I think, really, this is the crux of the argument for the extreme version of, you know, the, the problem with artificial, uh, uh, uh, superintelligence, right? Do you believe that intelligence is enough to manipulate the physical world arbitrarily or not? I am very strongly in the no camp. But yeah, that, that's, that's kind of beside the point. The, the real point that I wanted to make is like, actually, if, if you start, like, if you use these policy interventions, if you believe that these are the only policy interventions available to you, and you're actually like blinding yourself to your abilities to actually deal with the risks. Cause you're like, you're going with this strategy. It's like, okay.

• 57:33Anton Troynikov?

  Everyone who cooperates, everyone who, like the good guys, right? The people who would agree to cooperate, they've decided to stop. So they can't really like, they weren't really evaluate the systems as they are. Everyone who's declined to cooperate, they don't really care about any of the safety pieces of the system. Otherwise they would have cooperated, as you said. It's, it's like a, it's a, almost like an adverse selection problem.

• 57:52Flo Crivello?

  I, if, if you think the crux of the, of the disagreement is, is intelligence alone enough to control the world? Then that's, again, that's easily probable. I, I don't understand the counterargument. Do we have robots? The problem is not the degrees of safety. Look at the snake. Like it's very dumb. It's got very few actuators and yet it can do a lot. Like I think like the physical equivalent of Turing complete test. Like I have enough actuators to do anything I want. That's a very low bar. You really don't need a lot of actuators to do a lot of stuff in the world. We have robot hands, we have actuators, we have pick and backers and all of that stuff. All we need is the right software to control them.

• 58:25Anton Troynikov?

  I, I, I don't, I don't agree with the premise here.

• 58:28Flo Crivello?

  Tell me more, but what do you disagree about?

• 58:29Anton Troynikov?

  Because the reality is the modern industrial world requires actuators on the size, starting from the size of an oil refinery and going down to your scanning electron microscope. And with the reason that we need all of this vast array of things is because we as humans actually like the story of technology is almost the story of tool use, right? And every one of those sets of tools relies on another layer of tools below them. And if all you're left with is, is like even human-like actuators. And by the way, look, i- if, if, if the AI tried to take over the world with the, with the robots that we have today, we're, we're going to be fine. Regardless of how effective it is using the word, we're gonna, we just need to wait about, I don't know, two or three hours and we're good.

• 59:12Anton Troynikov?

  But even if you had like human level actuation, right? So suppose that like the AI downloaded itself into my brain and it could do that to like a whole bunch of people and like first, first of all, there's a trade off again. And this is the same trade off that comes up over and over again in my arguments, which is human is like, in order to get all of these abilities and skills and things that we have, we're complex and complex means fragile, right? Simple is simple is robust, complex is fragile, uh, which means that if you had a system that was able to carry out all these tasks, it would also be equally fragile to, to like, we would be able to deal with it because it's complicated, right? It would need to get energy, it would need to maintain itself somehow, it would need to do all these things. And as you, and as you say, well, okay, as a super intelligence will be able to figure that out, a super intelligence will be able to figure that out.

• 59:57Anton Troynikov?

  Now you're going up to this level of arbitrary capability again, which is, which is a difficult argument for me to accept.

• 1:00:03Flo Crivello?

  But humanity is complex and it's not fragile.

• 1:00:05Anton Troynikov?

  I don't know, people, people die in dumb ways all the time.

• 1:00:08Flo Crivello?

  Sure, but when we're facing a human opponent, we can't be like, well, nobody's like, oh, like this is not a risk because they're very complex, hence they're very fragile. And so they're talking about this.

• 1:00:16Anton Troynikov?

  Sure, but I'm talking about the ability to, to create these kind of general capabilities that will prevent us from shutting it down, right? Will prevent us from ever being able to act against it. Actually, it, look, I actually have, uh, there's a scenario that kind of does worry me.

• 1:00:35Anton Troynikov?

  In, in like, and it is kind of similar to the sort of things that some alignment researchers look at, look under, but it's actually more like

• 1:00:44Anton Troynikov?

  we, human, humans are relatively stupid in terms of aligning our individual goals presently with the overall goals of us as a society or whatever your community happens to be, right? We're, we're really bad at it. And to, to, to find this is true, try to like, find all of the programs in the US government that are simultaneously funding something and then funding the removal of that thing. Cause there's, there, there are many, you know, subsidizing it on the one hand, and then like, like trying to slow it down on the other.

• 1:01:15Anton Troynikov?

  We're not good at that. And I think it's possible for us to get into a state where the machine, for example, learns to fit our preferences. Like, it's kind of like, it learns to fit our preferences so well that we never even try to cooperate to like, get to the discontinuity which will allow us to get to like the next stage of where we're supposed to be. Like, we just, we just kind of hang out on Earth and it's mostly pretty good and everyone's fairly satisfied. That's, that for me is a type of risk that like the machines just gets really, really good at, at fulfilling sufficient amounts of human preferences that there's never any incentive to do anything else. And we just kind of heat death over here. I don't know if that's realistically possible.

• 1:01:55Flo Crivello?

  And, and Elisa, Elisa talks about that as well. He's like, even if we align it, like, we need to align it in a way that still makes it evolve. Otherwise, you know, I mean, our values as a civilization have evolved over the last few hundred years. Like, imagine if we come up with AGI 500 years ago, and today we have an AGI that is perfectly aligned with the value of burning women at the stake because they're witches. Right? So I, I agree that is also a class of concern. Um, we, perhaps this is a longer conversation. I, I, we all miss aligned on what it takes to control the world. I think that, uh, I, I agree with you that the safety community is calling for a leap of faith. There is a part of the argument that's like, oh, you know, we get AGI and then we get ASI and then question mark, question mark, and then we'll die.

• 1:02:39Flo Crivello?

  Right? And like, I, I, I agree with you. I think it's healthy to like, poke a little bit into that question mark, question mark. Like, hey, what's, can we talk a little bit about the bomb before we all die? Um, and, you know, I, I, I really don't think it's that hard to imagine what happens in the question mark, question mark. It's like, no, you know, like, yeah, the robots, like, what's wrong with our robots? What, what's wrong with the current robots?

• 1:02:59Anton Troynikov?

  They're really bad.

• 1:03:03Flo Crivello?

  They're bad because of software.

• 1:03:04Anton Troynikov?

  No, they're not bad because of software. They're bad for so many reasons. Um, software is like, software is one of the many reasons why the robots remain bad. Um, I think, yeah, look, it is, it is possible to imagine risks. And

• 1:03:21Anton Troynikov?

  So this is another, this is another trap that I sometimes see people fall into where it's like, it's, it's, it's easy to invent, it's like the opposite of the security mindset. The security mindset says, don't try to imagine defenses to like things that happen. Try to remove assumptions. And the, the opposite of this I see is like, okay, people start imagining so many different threats that they don't really think it like, they're like, oh, this is an indefensible threat. This is another indefensible threat. Where it's, it's almost, it's the same thing, but almost in the opposite direction. It's like anti-security mindset in some way. It's like, oh, oh, you, you like figured that out? Well, here's like, here's like another thing. And if it has arbitrary powers, we don't know what it's going to do. It's, I don't know.

• 1:04:03Anton Troynikov?

  In some ways, again, sometimes I, it's probably not true, but I really do sometimes feel like the person who's read the most like alignment and safety literature in the world and still doesn't agree with their points. And I think, you know, in the last few days, I kind of started to feel like I'm performing actually a valuable service by kind of red teaming the, the safety and alignment arguments. So maybe this kind of works out for everybody anyway. Even if, even if I turn out to be wrong, I at least make the arguments airtight for people like me and strengthen them.

• 1:04:34Flo Crivello?

  So, but even if we leave aside like the whole like robots debate and like, by the way, I agree with you about the whole assumptions and like epistemologically, I, I view a lot of holes in the way we're going about thinking about these things. But like, um, again, so that if, if we remove the accident class of risks, we're still left with the misuse class of risk. Surely having a bunch of AGIs, and by the way, I agree with your point too. It's like what you guys are doing may actually be causing more risk because you're, you're, you're removing assumptions or like adding assumptions by the capabilities of this stuff. And you're actually distracting from the real risks. But all you then worried about that class of risk of like, hey, surely having a GI and having a really, really powerful large language model that you can scale horizontally to arbitrary

• 1:05:18Flo Crivello?

  numbers and giving that to everyone, that sounds dangerous. Would you, would you agree with that?

• 1:05:23Anton Troynikov?

  Yeah. We sort of talked about this before on the Roco, uh, when he was, when he was on this, on this meeting as well.

• 1:05:32Anton Troynikov?

  And there's a class of things which are dangerous only like which which are less likely to be harmful in the world today, only because the knowledge to make like to use them is not widespread, right? And so one concrete version of the risk that you're describing is like, okay, so today, you know, today's language models can't really do this. Although, and again, I mentioned this last time, I have gotten chat GPT to help me design a neutron initiator, which is an important component in the making of a hydrogen bomb. It's what I do when there's long running computer, I try to get it to tell me dangerous things. Um, I think if we have this

• 1:06:15Anton Troynikov?

  little general purpose reasoner, right, which is above human baseline. So it's basically it's like the the model that I have in my head is like, imagine an automated theorem prover, but like it's generalized, right? And you can you can like ask it to reason from from certain premises and like you like give it some facts and you're like, okay, well, what can you tell me? Like, does this seem right to you? Because it's difficult for an individual human to reason, but this thing is like fully mechanized, fully self reflective, it can do all these things. It's like far above baseline as a reasoning tool. And if these things are widespread, suddenly, it's like this question of like, how do I kill the most people is like becomes more dangerous? Like you're giving you're giving this thing away as like something that people could use very effectively to plan things. There otherwise they wouldn't have been able to carry out before.

• 1:07:00Anton Troynikov?

  And, you know, there are definitely, there are definitely like thousands, if not tens of thousands of people out there who like want to kill you and me and everybody else here, like personally, not in some abstract sense, but they're like, no, that guy, he needs to be dead. Um, and and this reminds me, and this is going to be terrible, but I hope you guys leave this part in. There's a Stalin quote, Joseph Stalin, which goes, Eric, Eric's already laughing. I love where this is going. Um, there's a Joseph Stalin quote, which says, ideas are more powerful than weapons. We wouldn't give our enemies weapons. Why would we give them ideas? And giving out this sort of general purpose reasoner to people who are conceivably our enemies does seem like a more dangerous world.

• 1:07:43Anton Troynikov?

  But we then have to think about that class of problems specifically. It's like what before required a lot of knowledge to achieve now requires a lot less because we can just get the computer to like reason for us. Right? I no longer have to, I no longer have to get a PhD in, in whatever biology to like make something really dangerous. And even if it doesn't kill everybody, if it raises like the background incidents of, of these kinds of things, if it gives these people more power, that's, that's to me dangerous. That's a danger for sure. But I think the issue is kind of like with any knowledge technology, unless you can control the source of that knowledge, which again, for now, these things are centralized. They live in big compute clusters.

• 1:08:26Anton Troynikov?

  Um, I, you know, I contrary to Flora and I, I'm not sure that in the next few years they're gonna get laptop size, not, not the good ones anyway. Like even if you look at like alpaca, fine tuning of llama, it's like kinda okay. It's not, it's not really good. It doesn't give you GPT three and a half. The centralization there are kind of, you know, lets you prevent or at least know what's being said, how they're being used, et cetera. But the worry is, is like, it's like, okay, how do I, how do I make this thing that's possible where before I would need a PhD in chemistry or a PhD in biology or, or, but the thing that doesn't require a complicated biology lab or a complicated chemistry lab. It's like, what can I make my back backyard to do the mass casualty event because I believe like the end of the world is nigh because Ali Is or Yudkowsky said AI is gonna kill

• 1:09:11Anton Troynikov?

  us all. Like that's, that kind of risk is a little more worrying to me.

• 1:09:15Flo Crivello?

  So, so it's good. We are finding common ground on like, okay, so yes, there is risk here. Um, the question is, so then like, how many times do we pull the trigger? Like, does it run on everybody's laptop? Like surely on the limit. First of all, OpenAI is giving these things, yeah, API. And we have Nathan here who say, who said like, we have, I have worked for six months writing the darn thing and I'm still seeing risks that are not being patched and OpenAI seems unable to prevent jailbreaks. We have like, go to like jailbreakchat.com or something like that. It's, there's, there's dozens of jailbreaks and they're still, those, those unaffiliated by the way, just, I think it's a good website. Um, there's, there's plenty of jailbreaks.

• 1:09:55Nathan Labenz?

  Not a paid endorsement of jailbreakchat.com. Um, I believe we're gonna have the creator on the cognitive revolution, if it's the same site that, um, That I've seen, honestly, some really brilliant and borderline like interpretability work on that site as you, you know, start to think, geez, how does this actually work? I mean, it's really fascinating.

• 1:10:17Flo Crivello?

  Yeah, totally. So again, that's my point is like, we have these things via API. We cannot patch the jailbreaks. And also, you know, like, yeah, the open source committee is doing a good job. And whether it's like one year, two years or 10 years after OpenAI, at some points, they run on laptops because of Moore's law.

• 1:10:37Flo Crivello?

  And because also, not even Moore's law, I don't know what it's called, perhaps Altman's law, it's just inference. We find, we find 100x improvements to inference. So we've actually made inference 100x cheaper over the last two years.

• 1:10:51Anton Troynikov?

  Yeah, I think, but also there's a point that we're alighting here, right? We've talked about how such a system could create problems. We haven't really talked about, okay, now we have a general purpose reasoner. It could help us also reason about how we mitigate the problems too. And this is kind of like, this is my, so there's like the total doom black pill over here, right? Which is like, you know, we'll get to ASI without noticing it will be orthogonal in its values and, and seek instrumental convergence and like wipe out the human race because we're not like, we're not using our atoms for anything it considers useful, right? That's kind of the instrumental, that's the instrumental convergence of orthogonality argument, right? It doesn't hate us, it doesn't love us, it just wants to kind of, it needs atoms for the thing that it wants to do and it doesn't really care about us either way.

• 1:11:34Anton Troynikov?

  That's, that's black pill, absolute doom, whatever. Very rarely do we talk about the opposite side of this. And there's, there's like, the opposite, and you know, there's the less dreamy one, which is like the thing that we just talked about, we like give our enemies these general purpose reasoners, they all level up, they get smarter and they can do more dangerous things just because they're smarter. Over here, there's this thing like, okay, well, we have general purpose reasoners and we can spin up, spin them up on demand. That just like levels up humanity as a species and our ability to adapt and deal with problems way faster than we could before, right? Because not only, not only do we have these little general purpose reasoners, but by getting to use them and like interacting with them and working with them. Humans themselves will understand how to act, like, answer better questions, how to actually

• 1:12:19Anton Troynikov?

  use this tool. It's like programming with basically like Copilot without it, right? I've changed my style of programming because I use Copilot. And humans will change the way that we reason if we have a general purpose reasoner that we trust alongside us, right? It'll make us more adaptable, it'll make us like more able to solve problems and hopefully it will even help us coordinate. Because if you trust the output of the reasoner, which is obviously like in some way designed to be robust and unbiased and actually reflect the world as it is, and I think that there is a possibility of building them in that way. It will kind of help us, help us overcome some of the coordination problems we have as a species as well, because it's like, well, I don't trust you when you tell me that, but this thing has shown me how to reason, like, and I kind of arrive at the same conclusion

• 1:13:03Anton Troynikov?

  on my own, that means we can probably work together. And the absolute peak of that is that, okay, we've, us and these, these general purpose machines that we've designed alongside us, which are designed for our purposes and which are, like, forget about control, think of these things as a machine instead of, instead of an entity. If you think of them as a machine, and suddenly we have machines that allow us as a species to do incredible things. They're perfect empirical reasoners about the universe as we develop new goals, as we continue to adapt as a species, as we seek new frontiers. We have like this system right alongside us, which amplifies our abilities so much and continues to help us amplify our abilities so much as we discover more about the universe,

• 1:13:46Anton Troynikov?

  that we're on a completely separate runaway trajectory and a very positive one, a very bright one. That's kind of my, my like best case scenario here.

• 1:13:55Flo Crivello?

  I agree, which was my point earlier about the average human means well, and so on average, raising the capabilities of humanity achieves good things. But still, that leads to a bumpy road. So yeah, like you raise the capability for attack, at the same time you raise the capability for defense, but at least over the short term, the problem is that there is an asymmetric war. It is cheaper to attack than to defend very often, and very often attackers adopt innovations faster than defenders. For example, in the case of cyber security, you're going to have hackers adopting GPT, whatever, faster than every company and F500 giant dinosaur out there is going to adopt it to defend itself. So even though I agree that at the limit, probably we're going to be fine, probably

• 1:14:40Flo Crivello?

  as has always happened before, more technology. Means well good, I think there's gonna be a bumpy road. I think we're gonna have to hit the fan a few times and I think the next perhaps five or 10 years are gonna be very, very weird. I think Weird is right.

• 1:14:57Nathan Labenz?

  Yeah, we can all, everybody can agree on Weird is unavoidable at this point.

• 1:15:03Nathan Labenz?

  Except for those denialists who I find to be acting the weirdest of all.

• 1:15:09Nathan Labenz?

  But I guess I kinda wanna ask you guys, like, a lot of people sign the letter, a lot of people with a lot of different views. Certainly, the most extreme of those views or the most unsupported of those views are not very well supported. I don't think we've been cherry picking the bad arguments too badly in this case, but I do think it is kind of worth reining it in a little bit now to say, yeah, I don't think anybody has a great claim to authority on what the actual X risk is. You know, Eliezer seems overconfident to me. I'm probably more in the flow camp of something like 10%, but then I'm like, but that's also kind of a total gut number that doesn't really,

• 1:15:52Nathan Labenz?

  you know, for me, doesn't really have a strong claim on even being a number. You know, it's more like a mood or sort of, I don't know, maybe just like the most I can psychologically handle potentially. So that stuff gets very extreme. The letter's not that extreme. It's much more like, okay, you know, I think I give him a lot of credit too for just calling out like the tremendous upside, and again, I love this AI summer concept. Humanity can enjoy a flourishing future with AI. Having succeeded in creating powerful AI systems, we can now enjoy an AI summer in which we reap the rewards, engineer these systems for the clear benefit of all, and give society a chance to adapt.

• 1:16:37Erik Torenberg?

  Are you appealing, is your point to appeal to authority of all the people that have, that have signed it?

• 1:16:42Nathan Labenz?

  Well, no, just to, just to get clarity on kind of where they're at, because they're not like extreme doomers. They're not, you know, on net, they're not saying like, this is terrible for us. They're not, you know, saying GPT-4 shouldn't exist. They're just, they're on the contrary, they're saying like, we just made something truly amazing. And, you know, we should enjoy it and, you know, we've, we've got a great, you know, AI summer in store for us. But there is still this big question of where we go from here. And especially in view, you know, they also quote OpenAI in their letter and say like, you know, OpenAI says a time may come for this. We think that time is now. It doesn't sound like OpenAI thinks that that time is super far off.

• 1:17:19Nathan Labenz?

  Like, you know, I'm not a, a big betting person, but I would bet that Holden Karnofsky is going to be involved in setting up some sort of, you know, third party standards org and they're going to partner with OpenAI. And there's, you know, they're going to make a push for this.

• 1:17:37Nathan Labenz?

  So like, it seems like we should update on what they're thinking, right? I mean, certainly the people that know the most about this, there's a lot more that they know that they haven't published. And they are not necessar, they're not trying to close the door behind them, but they are saying beyond where we have gone. We think there's danger. We see that behavior is not easy to predict, even though we can predict a smooth loss curve. We don't know what that means in terms of discrete thresholds that we've reached or like, you know, specific behaviors that we might observe. And so therefore it seems like going beyond this point, you know, we plan to proceed with extreme caution. We think the world should proceed with extreme caution. We think there will need to be some sort of regulation.

• 1:18:19Nathan Labenz?

  So like, I don't know, it seems like everybody should kind of be able to get together on like, yeah, we're entering into some pretty uncharted, like pretty dangerous territory. The developers themselves are saying so.

• 1:18:32Nathan Labenz?

  So like, again, why not just a little pause?

• 1:18:35Flo Crivello?

  Because a little buzz is never a little buzz. That's that's maybe good for some of it.

• 1:18:40Nathan Labenz?

  I mean, the letter kind of skirts this, right? But like, there is, one question would be government regulation. But another question would just be like,

• 1:18:48Nathan Labenz?

  do you choose to do it? You know, I mean, if I was running like, Google and I was, you know, on top of all the world's compute resources, like, I think the right decision for me would be to not, you know, scale up 100x compute beyond GPT-4. Because what OpenAI is telling me is like, they don't have great predictability. I know that I don't have great predictability. So, you know, what the government may make me do could be a distinct question. But, you know, for a handful of of like, key decision makers in the very few organizations that have the resources necessary to go past GPT-4 today, for those people, like,

• 1:19:30Nathan Labenz?

  what do you think they should do? Like, it's just, you know, voluntarily, they can not go further right now if they want to. And, you know, that's kind of the start of the letter is, we call on these people to not go further right now. You know, do you think they should? It's a separate question, say, should we force them to heed heed that call? But maybe we could still say, we think they just should heed that call, even if we're not ready to like mobilize government to force them to heed that call.

• 1:19:57Flo Crivello?

  I still think it would set a precedent and temptation would be to extend the call. I also don't know, if you believe, as I do, that the performance of these models is mostly compute bound, how far does the little buzz extend? Do you also ask for a little buzz from Nvidia to to to to build more GPUs and and for Microsoft to build bigger data centers and Nvidia to develop the next generation of GPUs? Because if not, then really, all you're doing is you're growing the compute overhead. And the moment you stop the buzz, so you buzz for six months, and the moment you stop the the buzz, all of a sudden, boom, you bump up, you bump up to where you would have been without the buzz. So that may be that may be even more dangerous. So.

• 1:20:36Flo Crivello?

  Again, I tend to deeply mistrust the instinct of let's pass technology, let's slow down. It's never been that well before. The folks who've said that have always been on the wrong side of history. And I think the upside, Anton is right, like we never talk about the upside of this AGI, which is like, this can be really fucking good for humanity. This could be a hugely positive deal. We've been talking this whole time about it's gonna kill us all. Like, let's talk about the upside for a minute because it seems like we all agree that more likely than not, it's going to be the upside that materializes. It'd be really bad to miss out on that upside. And I think there is totally a chance that we do. I think we technologists assume that technology by default proceeds.

• 1:21:21Flo Crivello?

  But sometimes it pauses. By default, it pauses. It pauses. It's a miracle when it proceeds. And we've had precedents in history, the Middle Ages, when technology paused for a thousand years or 2,000 years. And I really don't want that to happen. I think we're getting close to AGI. We have the right institutions now. We have the right setup. Let's keep going and let's go after the prize.

• 1:21:44Erik Torenberg?

  Nathan, another way of framing what I'm hearing from Flo and Anton is there's asymmetric downside to pausing. There's very low upside because what is six months really gonna do? And there's downside that six months then sets a precedent that it extends a lot longer than six months or an adversary catches up and doesn't follow the cultural or legal, you know, sort of rules that we set. Or some other downside could occur. Do you differ from that in that you think that the upside is actually, like you think we could figure something out in six months that would really make it worth it? Or you think it's okay if it extends out? Or where do you not see the asymmetric downside?

• 1:22:31Erik Torenberg?

  Well, I mean, I just have radical uncertainty about what to expect. You know, that is gonna be coming in. The question is, where is the burden? We all have radical uncertainty. The question is, where is the burden of proof? And so is the default, under radical uncertainty, don't do anything? Or is the default, I mean, block everything? Or just keep going? Like, where is the burden of proof?

• 1:22:51Anton Troynikov?

  Radical uncertainty is just indefinite pessimism. Let's get certain about some things. Let's get certain. Let's make some good assumptions. It's what we should be doing as engineers.

• 1:23:00Nathan Labenz?

  I certainly have a lot of enthusiasm for the technology and I do think that there's tremendous upside in our near term future and like ultimately my position, you know, as a member of the red team and all that stuff is like for all the crazy shit I saw. And it did kind of freak me out and it definitely has me convinced that this technology is not safe by default, you know, it's not like just easy to align, like none of the sort of, you know, most optimistic scenarios about AI safety seem true to me. Nevertheless, I do think GPT-4 should be deployed, it should be used, it will be great. It will have real downsides and harms, but I think those will be greatly outweighed by the upside.

• 1:23:43Nathan Labenz?

  But I think it all kind of comes down to like a threshold effect and I, I think I'm finding the most clarity when I think about this from the standpoint of if I were a decision maker, myself, and it was up to me to determine, should I run a larger training run now than GPT-4? As we approach this threshold of potentially smarter, more powerful intelligence than human, I think I come to the conclusion, no, like I should wait. What happens in six months? I don't know. Maybe I decide to wait again. That definitely could be the case. I don't think that's, you know, I don't think that's insane by any means.

• 1:24:28Nathan Labenz?

  And that honestly might be my most likely expectation, you know, again, I'm imagining here, I am, you know, Sundar, I am Satya, I am Sam Altman. Like I would say, yeah, I probably, probably still don't know, you know, that I would feel confident proceeding in six months. But, you know, who knows what we might find? We might find, you know, lots of great mechanistic insights. We might find, you know, we might find that China is amenable to a deal, you know, everybody sort of assumes that like China is going to, you know, create large language model AGI. If we don't. I don't know what China is gonna do. I have like, you know, scant ability to predict. I did not anticipate. I mean, who has made the right predictions about China?

• 1:25:11Nathan Labenz?

  Did anybody see them like reversing COVID policy from total lockdown to total free for all over like a weekend? I don't think so. So now we've got all these like, you know, AI derived China experts running around saying like what their technology policy is going to be. I don't think that's very credible at all. And, you know, they also like shut down their own video game industry and they like shut down tutoring. You know, I mean, they brought their whole technology, you know, Jack Ma hasn't been in front of a microphone in recent time. So, you know, I don't think that they are going to let the Sam Altman of China make the decision.

• 1:25:49Nathan Labenz?

  And I don't, you know, I think they're going to want some better assurances than we've been given, you know, in the West, that this is all going to be fine. So maybe they're not going to go so crazy, right? Maybe they're gonna see this danger and say, shit, this looks insane. Like, maybe we should slow down, or especially if then they could also point to, look, like, the West seems to be slowing down. You know, that would be really, you know, that would make it a lot easier probably for them. I kind of wish we hadn't like declared chip war on them just as we're entering into, you know, an AI arms race. Some say that, you know, gives us a leg up in the AI arms race. I say, let's avoid the AI arms race entirely.

• 1:26:29Nathan Labenz?

  So I don't know. That's my China rant. But yeah, I'm still, if I'm in this, if I'm in the CEO seat at any of these companies and I have the compute, and it's up to me to say, should we do this right now as we seem to be approaching some critical thresholds? And given the level of predictability that we have, you know, I might resent it if the government told me I couldn't. But in my own private decision making, I think that the wise, like prudent answer that's like pro-social.

• 1:26:58Nathan Labenz?

  Is is real caution, and I think, you know, that can easily cash out to a little bit of a pause.

• 1:27:06Flo Crivello?

  I think in my case, it would depend on whether I I see a roadmap from the safety team. If the safety team comes to me and they're like, we need a person for six months, tell the capability guys to stop. And I'm like, what are you gonna do for six months?

• 1:27:21Flo Crivello?

  Do they have a plan? Do they have a roadmap? Do you feel like progress is being made?

• 1:27:25Anton Troynikov?

  Yeah, I guess. Yeah, I mean, I think this is really where coming down to this point. It's like, stop being indefinitely pessimistic. Let's figure it out, right? Let's let's believe that we can actually figure this out. It's too easy to say, yeah, it's going to have arbitrary capabilities as an adversary. It's going to be unpredictable where we don't know, like, because there are things about the world that we don't know. Acknowledge it may know those things before we do and use those against us. Okay, but there's still that danger power spectrum. We're currently in a place where we control the power that it has. Uh, even if it was to be as dangerous as as even the worst doomer says, we still control that lever, right? We still control how powerful it can actually be to act in the world.

• 1:28:07Anton Troynikov?

  Uh, and that's probably the thing to be focused on if you're going to be working on real safety. It's like, okay, well, it's not thinking about like, what coded messages is it going to send to like, get people to let it out of the box? It's literally what are the affordances that this thing has to do things in the world? How do we lock those down? Um, you know, and it's like, it doesn't have to be an air gap. Again, Eli Eizer did this experiment with Mary a long time ago where he like, got people to to let him out of the box as well, playing a super intelligent AI. Of course, the actual conversations were never revealed, unclear. Um, but even that is like, okay, great. So you did that, then what? Like, we have plenty of ways to like, prevent people from doing things in the world. That our society is kind of predicated on our ability to prevent people doing things.

• 1:28:54Anton Troynikov?

  So let's sort of tackle the real risks. I'm actually, yeah, I think Flo's conception of is pretty good here. Like, okay, six months. Give us a roadmap. Why six months? What are you gonna do? That's what, you know, then I would be more amenable. I'd be like, okay, well, fine, you know, okay, I can probably wait six more months till GPT-5, maybe. But it does, it does come with this risk. It does come with these asymmetric downsides. It does come with like, oh shit, well, I guess they will stop if we tell them to. Or it's like, oh, like, all I have to do is, is, is create enough, like, mimetic anxiety around the concept, and then we'll have a moratorium. Like, these are difficult. These are not, these are not things you get for free. This isn't, the pause is not free. And there are things that we have to, we have to grapple with.

• 1:29:35Nathan Labenz?

  I do agree that the pause is not free, you know, just to be intellectually honest about that. Like, there is opportunity cost, you know, or foregone upside potential at a minimum. That's one of the reasons that I, you know, I take heat on both sides of this, because the most, like, strident AI safety people accuse me of being a hype man and, you know, cheerleading this technology that's gonna kill us all. And then, of course, the, you know, the people that want to accelerate, you know, accuse me of being a doomer.

• 1:30:05Nathan Labenz?

  And I do think it's important to kind of emphasize both sides, because

• 1:30:12Nathan Labenz?

  I think the, the reason the pause, at least in the public consciousness, I mean, you guys are, are obviously read into this, but in the broader discussion, people haven't felt anything from AI yet. And, or almost nothing. Like, they've seen news, they've seen a couple demos, but like, the actual economic impact, the deployment, is just getting underway. So I really think it's important to emphasize to people that we have so much potential already. We don't, it's invented. It exists. You know, now we are in this deployment phase, and I really do think this AI summer concept is great. You know, we can have tremendous improvement, you

• 1:30:56Nathan Labenz?

  know, tremendous, tremendous gains to our standard of living, to, you know, with the AI doctor for the global poor. I mean, it's, it's right there within our grasp, and it doesn't require any new invention, right? It just, just requires refining what we have, deploying what we have, adapting to what we have. So, you know, we could do that in the next six months. We could be more ready for GPT-5 than we are right now. We, you know, deploying it at global scale and actually seeing, you know, what it does wrong, like seeing if they can patch my shit. You know, that's another thing I'd really like to know. I've got some tickets in that I would like to see patched and they've

• 1:31:35Nathan Labenz?

  not been patched. So, you know, can, can the developers, you know, demonstrate some improvement on the safety profile like that? That's part of the roadmap. I know they're working on it, but you know, progress is not what I would hope that it might be. In the end, uh, you know, yes, I want GPT-5 to at least the good version of it. Um, but sitting, you know, if I'm in that chair and I'm sitting on that compute and I'm like, am I going to put this, you know, potentially over this threshold with this next run? And then on top of that, I have to decide a number, right? I have to be like, well, how big is this run going to be? Am I going to go 10x more compute than GPT-4? Do I just say fuck it and go 100x? Do we really EAC our, you know,

• 1:32:16Nathan Labenz?

  our way here and go 1000x compute past GPT-4? Like, at some point, I'm like, wait a second, I'm getting crazy. We haven't even deployed what we have, you know, like, why do we need to create the next version when we haven't even understood or deployed what we have, which we know is going to be transformative?

• 1:32:34Erik Torenberg?

  Let's, uh, let's wrap on this. This has been a great discussion. Anton, you quoted Stalin earlier. It's no coincidence that Nathan was on the red team and now is advocating for government control.

• 1:32:46Nathan Labenz?

  Voluntary cause.

• 1:32:49Nathan Labenz?

  If we can just get a few stakeholders to agree, then the government doesn't have to do anything. Honestly, it would be best. Really, I think, I mean, I think that is a key point, you know, the, I don't want to see the government muddy this thing up. Like, that's not good. But, but key people can just make a call and it's, you know, it's totally within their discretion to do that. And

• 1:33:11Nathan Labenz?

  You know, nobody has to, the state does not have to be a part of it necessarily.

• 1:33:16Erik Torenberg?

  Yeah, that would be, that would be preferable if that, if there's going to be a pause. This has been a great discussion. Flo, Anton, Nathan, thank you so much for joining.

• 1:33:23Nathan Labenz?

  Appreciate it, guys. This is fun.

• 1:33:25Erik Torenberg?

  Thank you so much.

• 1:33:25Anton Troynikov?

  Thanks for having me on.

• 1:33:27Erik Torenberg?

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it, and I recommend you use it too. Use Cogrev to get a 10% discount.