E44: The AI Safety Debates with Zvi Mowshowitz

Nathan Labenz sits down with Zvi Mowshowitz, the writer behind Don't Worry About the Vase. Zvi is an information hyperprocessor who synthesizes vast amounts of new and ever-evolving information into extremely clear summaries that help educated people keep up with the latest news. In this episode, we cover his AI safety worldview, an overview of the AI discourse, and who really matters in the AI debates. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics Do you have questions you want us to answer, topic requests, or guest suggestions for upcoming episodes? Email us at [email protected] Also, be sure to check out our recent release on YouTube: The AI Scouting Report Part 1: The Fundamentals TIMESTAMPS: (00:00) Episode Preview (05:00) Zvi’s Introduction to AI (07:04) Weekly 10,000+ words / Weekly newsletter (12:34) Language models (18:25) AI Worldview (27:30) Probably of Due (33:10) Inspirations for Content (39:00) Audience for Writings (45:25) Impactful figures’ impact (48:55) Path of the river (55:39) Different camps in AI discourse (01:13:55) Acceleration Front Argument (01:20:08) Large Language Models Today (01:27:00) Spendings in AI (01:36:03) Principles / Virtue Ethicism (01:43:30) Human vs Non-human Universe (01:47:32) AI Safety & “Doomers” (02:02:10) Expectations of Human/AI Relationship (02:10:30) Future of Online Laws and Ethics (02:19:10) What do we do next? (02:34:50) Sources for learning (02:42:08) Conclusion LINKS: Don't Worry About the Vase: https://thezvi.substack.com/ TWITTER: @labenz (Nathan) @theZvi (Zvi) @eriktorenberg (Erik) @cogrev_podcast SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky (www.omneky.com) for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. MUSIC CREDIT: MusicLM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Zvi Mowshowitz?

  A lot of people talk about alignment, talk about it as some sort of weird dual end, where if you figure out how to align a system, suddenly everything is aligned, and if everything is aligned, then everything is fine. And I think this is a serious way people fool themselves into thinking this problem is a lot easier than it sounds like it is. But we're going to tell them things like, make the most money, right? Make the most people happy. Get me the most dates, or whatever it is. And the moment you say most on top of anything, you get effectively, you know, power-seeking behaviors, resource-seeking behaviors, by things that, again, are better optimizers than you are. And I think Sam Altman understands the problem pretty well, but that it seems clear he hasn't

• 0:41Zvi Mowshowitz?

  cultivated a culture of safety amongst the people he hired. I think one of the biggest barriers, right, to coordination is people assuming they can't coordinate, and therefore not trying. We need to engage with the Chinese, and we need to talk to everybody else as well, and we need to explore, could we coordinate on these ways? Which first requires us to understand that we need to coordinate on these ways.

• 1:05Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, my guest is Zvi Masiewicz, author of the influential blog, Don't Worry About the Base, online at thezvi.substack.com. Zvi is an information hyperprocessor with a background in math, trading, economics, game theory, and game design.

• 1:50Nathan Labenz?

  Over the last three-plus years, first with COVID and now with AI, he's carved out a unique niche for himself as a writer who can rapidly synthesize vast amounts of new and ever-evolving information into extremely clear, comprehensive summaries that help educated people keep up with the latest news. We had a wide-ranging conversation, first, talking about Zvi's longstanding interest in AI and his general AI safety worldview, and also comparing notes about how we are each using AI in our respective workflows today. Spoiler: we do both use LLMs for a number of different purposes, but neither of us use it much in our core writing work as of now. We then discuss who really matters in today's AI debates.

• 2:34Nathan Labenz?

  In other words, who exactly should we be trying to influence with our AI takes? I found Zvi's perspective on this question very interesting. Then, Zvi gives an overview of the current state of AI discourse, summarizing the positions of these so-called effective accelerationists, the AI ethics and AI safety camps, and the, in my view, unfortunate rivalry between them, and of course, the doomers, or as Zvi prefers to call them, the worried. Along the way, we cover Zvi's personal ethical framework, how much value he sees in possibly, but possibly not, sentient AI agents, whether the current AI moment was inevitable, the promise of various approaches to AI safety, the impact of the specter of US-China rivalry

• 3:16Nathan Labenz?

  on AI discourse, and whether the AI safety movement has been a huge success, or perhaps has merely further accelerated AI capabilities progress. As always, if you're finding value in the show, we encourage you to share it with your friends. And also, please send us your questions for an upcoming AMA episode, your guest suggestions, especially if you know entrepreneurs or researchers outside the United States who would make great guests. And of course, your general feedback is always welcome. You can email us at [email protected] or DM me on Twitter, where I am at @LeBenz.

• 3:54Nathan Labenz?

  And finally, don't forget that my new AI scouting report is now available on our YouTube channel. We'll be promoting this more in the coming weeks, but already I've been very pleased to see that a number of early commenters have said that it really has helped them solidify their understanding of AI fundamentals.

• 4:11Nathan Labenz?

  With that, please enjoy this conversation on AI safety and AI discourse with Zvi Mowshowitz. Zvi Mowshowitz, welcome to the cognitive revolution.

• 4:21Zvi Mowshowitz?

  Thank you. Great to be here.

• 4:23Nathan Labenz?

  So yeah, I'm excited about this. You are a leader when it comes to attempting to make sense of the AI discourse, which is along with everything else, going exponential these days in terms of its volume. So I'm excited to, you know, kind of get your worldview on all of this and try to get your sense of kind of the lay of the land, what really matters right now, and hopefully, you know, identify the signal that really matters in the, you know, increasing noisy environment that we're in. I usually don't even do this when I'm talking to like, you know, people who have just published a paper or a project or a product. But because your role is more kind of meta analyst, I thought it would be good to first just give you a second to kind of introduce yourself,

• 5:05Nathan Labenz?

  give us a little bit of your history on kind of how you got interested in AI, the perspective that you're coming to it from, because I think that will really help inform people as they listen to the rest of your commentary.

• 5:14Zvi Mowshowitz?

  Yeah, I want to say none of the things that you said in your introduction seem less strange over time. They will continue to feel bizarre, and I think for a long time. But um, so I've been thinking about AI one way or another for decades now, because I was introduced to the boom debate between Oleg Yudkaskiy and Robin Hanson, which is how I got into the rationalist discourse in the first place. And so, you know, I've been expecting AI on the horizon and wondering about what we can do about it and expecting the default outcome to be quite bad if we could actively do something else for a very long time at this point, you know, over a decade. But I had made a deliberate decision not to focus on that, not to make it the thing that I was having shower thoughts about

• 5:58Zvi Mowshowitz?

  or, you know, bothered me so much because I concluded, you know, other people are better equipped to do this. You know, my competitive advantage lies elsewhere. You know, my math is not strong enough to engage in the kind of theoretical maths that they were working on back in the 2000s and 2010s, early on. And so I instead, you know, I mean, first I was a competitive gamer before I started worrying about this. Then I was a trader of various types, started to try to start a few businesses. And then in 2020, COVID happened.

• 6:29Zvi Mowshowitz?

  And then I started writing about COVID. I'd already had a blog and I was writing a bit about my thoughts. But then I started writing just to, because that's the way I process the world. So you know, you write down what you're thinking, you try to explain to somebody else. If you can explain to somebody else you actually know what you think. And it forces you to like check your sources and justify your reasoning and I need to process what's happening. And then people found it useful and I kept doing it and it kind of snowballed. And now I'm a professional writer and when COVID ended, I jokingly said on Twitter, "Weekly COVID posts are replaced by weekly AI posts." As if that, what a nightmare, this is going so fast. And everyone said, "Yes, please." And I was like, "Oh." And then I tried it and I've been writing 10,000 plus words a week ever since.

• 7:11Nathan Labenz?

  So yeah, the 10,000 plus words is a good guide for the uninitiated. My general sense of what you're trying to do in these kind of periodic roundup posts is sort of create the thing that if people were patient enough to not read anything else and then just come read you once a week, they would kind of come away without having missed anything important. Is that kind of how you conceive of the project as well?

• 7:35Zvi Mowshowitz?

  I consider that as sort of one of the pillars of the project is same with COVID, what I was doing before is a one-stop shop where I put out the links, but if you wanted to ignore AI entirely and the AI discourse and AI risk and AI money utility, except for using it for your own purposes and then every once a week you check back, you see what I have to say about the parts you care about, you can figure out for something you need to pay attention to, you can click through, you can explore further, or you can not. And the other big thing is to explain the thinking, explain, build up the world model over the course of weeks and months and many words. And help people think about what's going on, not just lay out, okay, here's what I think.

• 8:16Zvi Mowshowitz?

  Here's what you should think. I think that's, you know, here's, here's, here's how it is. And that's not as helpful.

• 8:22Nathan Labenz?

  So how do you just briefly approach the challenge upfront of just ingesting all the information? I mean, it probably goes without, you know, it goes by assumption that you're a pretty fast reader and just naturally given to it. But even so, you know, there's so much stuff, right? That even you can't probably consume everything you'd like to consume at this point. I saw a tweet that you posted recently about the AI X-Risk, you know, pro versus con with Tegmark and Jan LeCoon and their debate partners. And you asked like, should I watch this? So that was very interesting that you're kind of crowdsourcing, you know, what you should watch, a reflection of just how much stuff there is. But maybe more broadly, like, how do you approach identifying what you want to take in in the first place?

• 9:05Nathan Labenz?

  And then obviously, you know, working through toward synthesizing that for the audience?

• 9:09Zvi Mowshowitz?

  Yeah, I find audio content is especially expensive to consume, right? Because I can consume it about 1.3 and still be able to actually process the information or arguments of people who are speaking like detailed, complex stuff. And for really tough stuff, I call it the Tyler, I want to go at 1.1. But that's still like, compared to reading the transcript, that would be at, you know, at least two, probably more than three or four times that speed. And I can then deliberately speed up and slow down as there's like explicit things I want to think about and I don't constantly have to process the new information while I process it, it's much better. So when I have something with audio content, I have to think carefully, do I want to consume this?

• 9:52Zvi Mowshowitz?

  And I have to make cuts, especially with lots and lots of three hour podcasts coming out now from the computer vice institute, 80,000 hours, lots of other people. And for a while, I was on every podcast, no demand. And a lot of actual good content, you have to make choices. For reading, my procedure is I have Twitter, where I set up an intensive set of lists. Remember, I have a rationalist list and an AI list, and I have my follows. And I consume those, and then I pop out windows for here are some things that have content I want to consider to do today. If it's a small thing, I use the AI to do that for the one, just do it immediately. If it takes more than two minutes to handle it, I'll pop it out. And then at some point later, I'll go through all the AI links, and I'll go, okay, what does this have to say? How does this incorporate into what I'm doing this week? Which section would this fall into?

• 10:35Zvi Mowshowitz?

  And then I build the thing step by step as I have different sources to incorporate. And I also have a Feedly, so I have a lot of RSS feeds. And that's where I get most of my other stuff.

• 10:47Zvi Mowshowitz?

  And then, mind, these two will alert me to anything that isn't going either of these two, so I can then go find it and then proceed there. Plus, there's some things that appear in less wrong at the alignment form.

• 10:58Nathan Labenz?

  That's fascinating. So we've got some aspects of our history here are very similar. Like, I also kind of got initially very interested in AI from the sort of overcoming bias, Eli Ezer and Robin Hanson, days back in like, primarily 2007, I think, was kind of the heart of that. And similarly, also, kind of felt like, you know, at least as Eli Ezer was articulating the work to be done at that time, such theoretical math chops seemed required that I basically felt like that, you know, wasn't really for me. And obviously, the situation has changed quite a bit. I'm very different, though, actually, on the mode of consumption. For me, I can listen to pretty much everything at 2X, and I kind of slotted

• 11:43Nathan Labenz?

  into my, you know, downtime pretty naturally. And I feel like it's more like the actual time looking at a device that feels scarce. And that probably also just reflects that I'm not as fast or as fluent of a reader as you probably are.

• 11:56Zvi Mowshowitz?

  Yeah, I think different people are very different. Tyler Cowen will read at several times my speed. Otherwise, there's no way he could read what he writes, what he writes about, what he really says, because when I'm reading, it's insane. I read most faster than most people, but I definitely have trouble with listening to the art of processing. I've always had it. I've had problems learning foreign languages for that reason. It's just something that takes more of my energy. And so, it's a relatively less efficient form of consumption. And like, every time someone says, "I listened to you," or "you're writing," this strikes me as absurd. Like, why would you listen to a podcast with this when you could just read it? But sometimes it will.

• 12:33Nathan Labenz?

  Is there any role of language models in your workflow today? Like, do you use a tool that helps you summarize, or, you know, are you dabbling in, like, creating first draft type of content for yourself, or do you just do it still, all's the top to bottom?

• 12:51Zvi Mowshowitz?

  The writing is all me. I have found The L M's are very, very bad at summarizing the things that I'm talking about. They're very bad at extending your writing in this type of form. I know what style I want. I don't want to transform into a different style. The spell checker basically does most of what I want it to do. I don't think that like having it check for errors would be very convenient. I experimented a bit with this and doing even writing some Python code and I haven't been able to get anything particularly useful on that. I probably should experiment more, but I don't think they're quite there yet. I think other people will build tools. I'll eventually find those tools close. I'll start experimenting with those tools, but it doesn't mean I don't use L M's. I absolutely subscribe to G P T four and I use G P T four and I use Bing and I use the flexibility sometimes.

• 13:36Zvi Mowshowitz?

  But the main thing I use them for are for finding out about the world, asking questions about what's going on, asking questions about things I don't understand. Checking intuitions, checking to see like if the L M will share my intuitions about a question. And also like coming up with like more examples of something, right? If I have like intuition, like today I was trying to write a book about debate. And so like debate strikes me as something that's kind of like democracy. It's one of these, it's the worst thing except for all the alternatives. And so I asked you, name 10 more things that are like that. And it came up with some beautiful examples and it started to lose steam by the end, but like you've got five really good ones. And so, you know, stuff like that is really useful. And just it's better than Google for, you know, like, well, what was the source of this book? What's going on with this particular, can you narrow this down?

• 14:21Zvi Mowshowitz?

  Can you extract this particular piece of information? I'm often wanting things that aren't that easy to Google specifically. And also I'm just confused about something and I want to understand it. It's not useful and for coding. I still don't code very much because I have something, I think going on with my productivity was through the roof when I started using it. That was just obvious.

• 14:38Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:41Nathan Labenz?

  Yeah, that all resonates with me as well. I think pretty much down the line, I don't really use it for my writing. I am starting to experiment a little bit with. Actually, the mobile app, because the transcription is so good that I am finding there's at least some utility in, I actually was doing this the other day, walking my kids down the street in their stroller, trying to get them to take a nap. And, you know, as they're like getting close to nap, you know, I'm talking to the mobile app and, you know, kind of telling it what I want to do, what I'm trying to create, you know, here's kind of my thoughts, it'll take a shot. It is typically not close. But then I find like

• 15:22Nathan Labenz?

  just transcribing, you know, recording again and go just scrolling through its output and just giving it raw commentary on its output and saying what I want more of, less of, do and don't like, et cetera. Over like two to three rounds of that, I'm having some successes where I'm getting to like 80%, you know, kind of there. And it does feel like when I actually do go back to my computer later and want to write, that I'm in a better place than I would have been if I had just, you know, sat down blank slate, or even just because my typing is only so fast, right? So it's like just getting a lot of stuff out that then I can edit. I'm not sold on that workflow yet. I do think it has some

• 16:05Nathan Labenz?

  trade offs and there are some things that kind of the LM does that like you wouldn't have done, that you maybe kind of wish you, you know, wish hadn't been the final product that kind of still creep into the final product. But I don't know, it's at least kind of starting to work for me a little bit in that way.

• 16:19Zvi Mowshowitz?

  You know, all the discussions about AI and how much AI is useful and how much of AI productivity is the question of bottleneck. It's like, where is the limiting factor? What is actually slowing you down? You know, what would help you and not help you? In this case, it's a place where like your writing experience is giving you a bottleneck and being able to flesh out your thoughts, get a rough idea of where you want to do. And that's just not a bottleneck for me at all. Right? So that's, for me, like the bottleneck is getting the information. It's knowing what I want to say, how I think about something and the actual like process of that. When I start writing. It's almost like this, just if anything is free while I'm finishing the part of processing the information.

• 17:03Zvi Mowshowitz?

  And so having it, like, take this kind of first stab, wouldn't actually speed up my process at all, because the part where I'm doing that is actually doing work anyway. And it's also very quick. So I think a lot of AI is like, people say, well, it's not very useful. A lot of people, they haven't figured out how to use it. And a lot of that, they haven't asked, they haven't thought carefully about their own process. And exactly what parts of things are easy and hard, don't try to do exactly what everyone else is doing. Do the thing that's useful for you. And then iterate on it and also learn what kind of front engineering gets you the thing that you want. I haven't done any, any of the detailed, like, paragraphs or pages long things, but I figured out some very basic tricks that work for specifically getting me more of the type of things that I want, much the type of things that I don't want.

• 17:46Nathan Labenz?

  That it all resonates to me. The, um, the other use cases, I'm right there with you too. I mean, coding, good God. Similarly, you know, code some, but not full time every day. And the ability to just import the right stuff and, you know, write the, the right kind of boilerplate and get me, you know, man, that really helps a ton. So I love that. Okay, let's talk about your worldview and then we're really going to dig into all this stuff that's going on in AI discourse. But again, just want to kind of set out your position a little bit. Could you, this is a big question, but could you, you know, kind of take a couple of minutes and characterize your overall AI worldview and specifically, you know, as the safety debate has really heated up, emphasis on that.

• 18:26Zvi Mowshowitz?

  Yeah, there's a lot of different aspects to how one thinks about this. And a lot of the problem with the debate is that whenever you're debating somebody, you never know, like, which of the 20 things that are unclear, they're actually disagreeing with you about, and often it'll manifest as a disagreement somewhere else. You don't understand that where they're actually disagreeing with you is something they haven't mentioned, because it hadn't even occurred to them that someone might think differently about that. It's so obvious to them. And at other times, there will be like 17 different disagreements that don't really have a common cause, or don't have a common logical cause, but are motivated by the desire to get to an end result in various people, or other things of that nature, or they're just sort of going off a vibe and don't really have opinions on most of these questions.

• 19:10Zvi Mowshowitz?

  And they don't really have a coherent model. And then like, you have to get them to think about the thing at all in order to have a disagreement, so you can then figure out what part of this is useful. So for me, I would start with, you know, LMs are this incredibly useful tool. In their present state, they are almost entirely positive. I think that almost all of the fears and worries about misuse or negative effects of LMs are overblown, if anything reversed, and that the effects of current level of tech are just almost uniformly going to be positive.

• 19:45Zvi Mowshowitz?

  However, I do think that this changes as the AIs approach human, approach to surpass human-level intelligence. I think that this is a very different environment for us to live in. I think it's a very different world. I think that lots and lots of very hard to predict things start happening. And that if we're not very careful, and almost by, pretty much by default, the outcome is the death of every human on the planet. Or at least the loss of control over our future such that

• 20:15Zvi Mowshowitz?

  if you ask most people, including myself, they would see the resulting future to have very little or no value compared to certainly the potential futures or compared to what you currently might imagine a normal non-AI future to be. And so you have to draw this big distinction between the AI is a tool. Right now, it's a tool, right? It, it has some amount of the ability to reason things and figure things out and like mimic human intelligence, but it's not like that. And we can use that to greatly enhance our lives and experience and our progress. And the point where the AI starts to become a rival optimization engine, rival set of capabilities, rival intelligence that can eclipse our own, whether or not that then there's a curse of self-improvement as Elliot D. Caffey talks about.

• 21:00Zvi Mowshowitz?

  And then suddenly, you know, very quickly after we have something that's human or near-human, it suddenly is deeply superhuman and we have no idea what's going on. And then the entire solar system's atoms get rearranged. However, the superintelligence would prefer, and that likely doesn't involve us. Or if it's something that looks more gradual, looks more slow. I think these are, they're all very, all possibilities are like, not just like possible, but like substantial double-digit probabilities of various outcomes.

• 21:32Zvi Mowshowitz?

  And that all of these scenarios offer ways for us to come out of it very well, and ways for us to come out of it not at all, if we're not very careful about it. And you have to think carefully about how these things work, what the problems are going ahead, and that almost everyone is thinking very poorly about at least some of the problems involved in this, even if they think well about some of the other problems. And there probably aren't any people who are thinking very well about every step of the way. And that includes me, right? I'm almost certainly thinking poorly about some of these steps, and that will be exposed over the coming months and years, hopefully, as we learn more and I think more and I change my mind.

• 22:13Zvi Mowshowitz?

  But essentially, the AI safety problem is if you build a more capable optimization engine, something that's more intelligent, can do more things than we are, that has inherently many advantages over us, it'll be copyable, it'll be faster, it'll be able to, you know, instantiate in various ways, we have to rewind and reform the state, it can have essentially limitless memory, et cetera, et cetera, you know, input huge amounts of data, orders of magnitude more than any human could possibly hope for, it's already happened, et cetera, et cetera. So if it can mimic the parts of what we have that's missing well enough, suddenly, its capability is the sky's the limit.

• 22:58Zvi Mowshowitz?

  And then it's going to be a tremendously powerful optimization engine that's going to optimize whatever gets pointed out to optimize, even if we try to keep it as a chatbot. We say, oh, we're not going to give it goals, but we are going to give it goals. We know this, we run this experiment. And so it's going to have goals, it's going to have targets, it's going to optimize according to some set of priorities. And we have no idea how to make it do this in a way that we control what those priorities are, and we have no idea what priorities, if we could control it, we could put into it, that wouldn't ensure disastrous outcomes, even if we had that ability, either individually or collectively. So all of this are problems we have to solve.

• 23:39Zvi Mowshowitz?

  And we have to solve it knowing that we don't know how to solve these problems very well in current systems, despite the fact that we are smarter than the current systems and the current systems are not dangerous and are not engaging in purely adversarial actions. But when we start creating these more intelligent, more capable, stronger optimization engines, on every level, we're going to face optimizations for things that we do not want, that respond in adversarial fashion, that require a security mindset, where any way for them to fail is the way they definitely will fail. Every time we try to fix problems, we are correcting for the thing that fools us into thinking we solved their problems, or that gets around whatever our solution is.

• 24:22Zvi Mowshowitz?

  And fundamentally speaking, we're trying to get and keep control of something that is smarter than us, and more capable than us, and more efficient than us with a competitive advantage over us. And that is a very unnatural thing to want to happen. And if you think that this is impossible, I am deeply confused. You think it cannot possibly go badly. I think you're just not thinking about this clearly at all. And if you think we will probably be fine, then I am confused why. Right? I think a lot of people are like, well, we'll figure it out. We will solve these problems and it'll probably be fine. And I think we have to solve a lot of virtually impossible, not impossible in the impossible

• 25:04Zvi Mowshowitz?

  difficulty level on a game sense, problems in order to not have this go badly. I think there's probably solutions to that, but these solutions are incredibly hard to find. And as Eliezer Yudkaski emphasizes, maybe they have to be solved on the first try, because the moment you transition from a non-dangerous system to a dangerous system, that's exactly when a lot of the things that were working before will stop working, your techniques to align the system, as we call it. And it's exactly the time when, if they're not, if the system isn't aligned, you are in deep, deep trouble and you might not, you, meaning the human species, might not be able to recover from a serious mistake. And we also can lock ourselves into various dynamics and equilibrium and world states,

• 25:46Zvi Mowshowitz?

  where even though we figured out how to align in an important sense individual AIs, and we've been able to figure out how to make them act according to something that we told them to act according to, how did the resulting competitive dynamics end well for us?

• 26:04Zvi Mowshowitz?

  And those are more problems than I think, you know, people are not thinking clearly about, people have good solutions to. A lot of people who talk about alignment, talk about it as some sort of weird Boolean, where if you figure out how to align a system, suddenly everything is aligned, and if everything is aligned, then everything is fine. And I think this is a serious way people fool themselves into thinking this problem is a lot easier than it sounds like it is. Because yes, that's part one of the impossible problem, is you have to impossibly figure out how to make the thing do the thing you want at all. But then you figure out what you want. And collectively, we have to coordinate on figuring out what we want. And we have to make this in a enforceable, actually instantiated thing, and we have to not screw it up in an impossibly stupid way. Because the history of humanity is the history of people constantly making incredibly stupid mistakes and having to fix them by trial and error,

• 26:49Zvi Mowshowitz?

  even when everybody means relatively well.

• 26:51Nathan Labenz?

  So let me just try to give the super high-level summary of a couple of those things. I guess if I were to try to describe your overall kind of expectations, like first-order approximation, it almost seems it's like uniform probability across everything, like basically radical uncertainty, where there's some chance everything could go great, you don't really quite know how. There's like significant chance, you know, things go very bad or totally catastrophic. And the details of that are also kind of fuzzy, although you gave a little bit more definition, I think, to, you know, how that might actually play out versus how things could go well. But is that basically right, just kind of probability-wise, radical uncertainty?

• 27:37Zvi Mowshowitz?

  Yes and no. So I would say that one of the things that people do is they ask, what's your P doom, right? What's your probability of doom? And the answer to me, the first question is like sort of, where does that matter, right? So I'd say that like, well, if you're in the single digits or below, that's important, because then it starts to be a question of, well, what's the P doom of not acting or slowing down or other alternate possibilities? What are the, what are the gains to be had? And you start to think, well, maybe we're getting close to a point where we might want to start taking some chances on that level in order to get the benefits. Or you could argue even that's crazy. And in many other contexts, almost everyone on Mirko would agree.

• 28:18Zvi Mowshowitz?

  If you think your P doom is super high, it's in the nineties or ninety nines or whatever it is. Then you have to start thinking about what actions might make sense, given that you can't really hope for just to have a good outcome by proceeding. And so that starts to change your outcome. But if you're in anywhere in the middle, there are very few actions that make sense at ten percent doom, but not ninety percent doom. Or vice versa. Because all of the orders of magnitude of payoffs and costs are just bigger than this one order of magnitude of difference here. And so, you know, it's not that you have radical uncertainty is that like it doesn't really pay to be that person. Being that precise is not as valuable as you might think. And that's one reason why people end up with these like kind of arbitrary probabilities.

• 29:00Zvi Mowshowitz?

  There are also just so many different things in the process to model, so many different ways it can change. You hear a lot of people say five percent, ten percent. You don't see that many people closely in the middle, but that's actually where I am. I, I think fifty percent when I have to make a guess. But I understand that like as the rules of the maze, my numbers should be shifted around constantly and it's not that clear. I wouldn't say all these things are equally likely. It's more that if they're all likely enough, we have to consider them as plausible scenarios. And we have to like be willing to spend substantial amounts of time and effort and money and sacrificing a futility to make sure that there are good outcomes in those cases. But if I had to, you know, guess what the most likely outcome is, I would say doom is, is less likely than non doom. Although again, both are substantially lies.

• 29:42Zvi Mowshowitz?

  And that if I guess that we lost, you know, I'd say a lot of it is, well, we just failed the alignment problem. You know, maybe roughly half of it is we just got the alignment problem early. Maybe half of it is that like we succeeded at the alignment problem as we were talking about the alignment problem. And then we specified something that got us doomed anyway, because we didn't understand about the dynamics implied in what we were doing. But I don't consider these questions to be worth the detailed time and effort to try and get numerical answers from. Because I don't see any actions that change. Right? Like, and again, the question is like, are these real things that we really have to deal with? Or not? And the answer to me is, well, all of them, yes. Right. And that's much more important than, well, okay.

• 30:28Zvi Mowshowitz?

  Exactly where do we go wrong more often? Right. To me, it's like an interesting, but like not very useful question.

• 30:35Nathan Labenz?

  Yeah, okay. I think that all makes a lot of sense to me. I think I'm pretty much right there with you in terms of not spending too much time trying to narrow my, you know, confidence intervals on how likely are, you know, various extreme outcomes. It just seems like the lower bound on that, or the lower end of that interval is high enough that it is of concern to me. And it's like, it should be of concern to most people, kind of regardless of exactly where you end up. So it resonates with me and strikes me as just generally wise approach to not get, you know, nerd sniped into kind of false precision. The kind of simplest, I'm going to try to do again, the simplest version of why this is something that people should really take seriously.

• 31:17Nathan Labenz?

  And then I do want to get on to kind of the more discourse side of this too, but just your view. Simplest possible form seems to be when something is more powerful than you as a general purpose optimizer, that's inherently dangerous. And even if you get to set it up, you know, it still remains dangerous. Is that a good one sentence summary?

• 31:42Zvi Mowshowitz?

  Yeah, they're good. We're going to give these AIs, even if we manage to give them the gold and ideas that we want. Mac, what's called maximalist goal, right? We're going to tell them things like make the most money. Right. Make the most people happy. Right. Like, you know, get me the most dates or whatever it is. And the moment you say most on top of anything, or you want the most probability, you want the close, you know, the number of nines on your success on anything you could do suddenly, or the thing is simply trying to be selected for more instantiations. Or, you know, we're just seeing more of the copies of the things that successfully convince humans to instantiate the more, or that manage to gather resources, or any of these dynamics like that.

• 32:24Zvi Mowshowitz?

  You get effectively, you know, power-seeking behaviors, resource-seeking behaviors, by things that, again, are better optimizers than you are. And by default, those things are going to end up with the resources, and those things are going to end up with the power. And that's not going to go well for us.

• 32:39Nathan Labenz?

  Yeah, makes sense. Certainly, I have never heard one good reason to dismiss that as a concern. So let's leave that there for now, and we'll certainly touch back on some of these aspects as we kind of go through the survey of the AI debate and debate debates, as you put it. For starters, I kind of wanted to ask, who you think matters as you create content? And obviously, anybody can, you know, come to your blog, you know, among many other points of confusion. I feel like a lot of people are kind of confused about who are they even trying to reach when they put stuff out in the first place? And so I've got my own kind of quick model of like, who matters and why? And I want to just bounce that off you and see how you would react to it, think about it maybe differently. Seems to me like the most important people right now, if you could choose who you would

• 33:24Nathan Labenz?

  want to influence, would be the leaders of the leading AI developers, as they are the ones deciding, you know, what to do and not to do, and how to approach various problems, you know, in actual work, making actual systems, which seem to be getting, you know, quite powerful quite quickly. So, you know, leadership of OpenAI, leadership of Anthropic, leadership of DeepMind, you know, maybe a dozen global companies would seem to be like the most critical decision-makers right now. And then my tier two would maybe be like policymakers on the one hand, and sort of general AI research community on the other. Those are maybe, you know, if you think there's maybe 100 decision-makers across those top

• 34:05Nathan Labenz?

  10 or so companies, there's maybe then, I don't know, 10,000 each of kind of the policymaker and, you know, researcher, you know, somebody who could really make an advance in the field kind of sets. And then the broader, you know, circle beyond that would be maybe like...

• 34:22Nathan Labenz?

  The discourse, you know, like all the people that are sort of echoing stuff back to each other, you know, public intellectuals, your sort of, you know, venture capitalists, that sort of whole, you know, people who have a lot of clout on Twitter, but aren't necessarily making object level decisions in the short term. And then finally, beyond that, there's kind of like the general public. And they, you know, kind of feedback on to who are going to be the policymakers and like, you know, that kind of on some distant level controls, you know, will companies be allowed to do stuff or not? How would you compare and contrast how you think about it versus that starting point?

• 34:57Zvi Mowshowitz?

  I think this question is incredibly complicated. And you can't think about it in isolation. You have to think about it together with questions like, what needs to happen for things to go well? What, if it goes wrong, causes things to go poorly? How do these things influence each other? And these questions are very complex and difficult to answer. So if I had to choose like one person to align my PEL properly on the margin, you know, to edge towards my view, I would definitely pick Sam Altman. Right? Sam Altman is more important than Joe Biden in this sense. And then, you know, Dennis Obis, Dario, Eda Prophic, you know, potentially the CEO of

• 35:40Zvi Mowshowitz?

  Google and Microsoft, you know, Zuckerberg and others at Meta become prominent because like they are the highest on the totem pole that like clearly don't take this seriously. And it would be great if they did take this seriously. It would substantially make the situation better. And then like, how important are they versus how important are the politicians? Like, how much do politicians actually drive policy? How much are they willing to stick their necks out? How much do their opinions, do they have real opinions? Do they matter? You know, if we convince Joe Biden, what would happen? I don't even know if it would be that big a deal. Or how likely he would be to understand the issues involved because they're they're very complicated and technical. And, you know, he has infinite demand on his time and is very old.

• 36:25Zvi Mowshowitz?

  And I'm sure his amount of compute left is shrinking in some sense, like as certainly available for this. And then, you know, we have a system with a lot of these veto points and negotiation points. And it's very unclear who causes what to happen. One of my disagreements with Tyler Cowen is Tyler puts a lot of weight on academic consensus and publishing papers. To try and convince a bunch of academics to then try and convince potentially a bunch of like sort of behind the scenes policy people. Then maybe convince a bunch of lawmakers and regulators to do the things that would be better. And I think it's possible that that's an important vector. But it's confusing to me and unclear whether that vector is actually open to us or how effective it would be.

• 37:09Zvi Mowshowitz?

  And I don't see how to run experiments to actually prove it one way or the other. Similarly, for the public, right? Like, the public is currently doesn't really know what it thinks, hasn't really understood the situation very well. But tends to be pretty worried about AI when you actually show it AI and ask it basic questions. And how much does that matter? How much does that drive legislation? And then another key question is, how much do various different paths lead to good interventions that are well selected? And designed to actually ensure good outcomes? Versus they cause misaligned actions? So like one worry you would have is the pop- the pop- the population gets really scared about AI. They demand action.

• 37:51Zvi Mowshowitz?

  And then we prevent people from using AIs on a variety of current productive activity in ways that make our lives worse, but don't make us any safer from the real threats. And so that's a reason not to go down the public line, right? If we convince the public to be scared, we're not going to invest this very, very specific complex message that we need to control the training of advanced frontier models. Because that is where the danger lies. That is what could go wrong. They are just going to latch on to whatever their minds can see. Because they are not going to spend endless hours on this. They are not specialists, right? And I would do the same if you talk to me about something that I'd spend very little time on.

• 38:34Nathan Labenz?

  So how does that? Play into your writing. It sounds like, I mean, almost by definition, you're not writing for the general public. Anybody who's reading you, you know, that's enough to qualify as being kind of part of the public intellectual set at least. But do you actively think about like, an audience of OpenAI leadership, Anthropic leadership, or do you just kind of write what you think and hope that that, you know, works its way to them?

• 39:01Zvi Mowshowitz?

  I think in a certain way, you write for yourself, always, when your writing's gonna suck. And so you write the things that I would want to see if I hadn't put the effort into making this, the things that explain things to me. And then the question is, you know, who else are you targeting? And I try to become more and more careful about like, who am I targeting? And who matters? Because one of the things that I do think the number of people who actually like, have a major influence on events, who sculpt what's going to happen, is relatively small. Right? Not that other people collectively don't matter, but that like mostly they are not individually worth trying to target in that sense. And certainly, I know of the case of, I'm gonna target people who can handle very dense, very complex amounts of information,

• 39:44Zvi Mowshowitz?

  massive amounts of information, and actually want to process it and make sense of it, and like, can engage with it on its, on the level it deserves to be engaged with. If you can't handle that, I'm counting on people who can handle that to then get the information to them in some number of steps, by putting distillations into a form they can process, or are willing to invest the time to process, right? Like, it's not a knock on them. I, if you gave a 10,000-word-per-week, complex, dense thing on most topics, I wouldn't read it, right? Even if it was very, very good, because I just don't have that kind of time. So, in my mind, like, there's a handful of public intellectuals who are sticking their

• 40:25Zvi Mowshowitz?

  neck out and making like impactful, meaningful arguments about this. They show up time and time again. I try to engage with them specifically. I talk to them, and I talk to people who are reading them, that people pay attention to them. More specifically, you know, I do hope that Altman and Isabas and Abadi are reading, what I have to say, but...

• 40:48Zvi Mowshowitz?

  I think even anyone who works at one of those labs, right, who can then pack the information along, who can then spread it to other people, who can then pass the core concepts and the most important parts on to people like that and also people who work around them, matters a lot. One of the things I worry about most of OpenAI is that I think Sam Altman understands the problem pretty well, but that it seems clear he hasn't cultivated a culture of safety amongst the people he hired, such that a lot of people who work at OpenAI don't understand safety the way that Sam understands it and the culture of the people he hired drives your company's actions. It works. And so he has much less ability to steer in these ways than Dario does at a profit where he did in fact do this, even though I'm not convinced that Dario

• 41:32Zvi Mowshowitz?

  had a better understanding or worse understanding of what it actually takes to be safe at the end of the day than Sam does. I think these are serious things to consider. I don't know as much about he might, but I've had conversations with people who work at all three organizations on, you know, as engineers. And there are definitely people who care deeply about ensuring good outcomes with all three and have at least one reader in all three and hopefully they forward things on. But yeah, it's a combination of also people who work in alignment and putting the outside of the labs, people who are in the discourse, who are out there being people with the arguments, public intellectuals who actually matter. But there might be a million public intellectuals in some sense out of seven billion people. But the vast majority of them don't care about AI.

• 42:16Zvi Mowshowitz?

  And the vast majority of the ones who do care about AI still aren't really providing any inputs to the way AI goes. They're more of what I think of as a lesser tier with respect to AI. So like, I have this metaphor of like on level one, you have people who are generating new ideas, new ways of thinking about things, like new synthesises. And then level two, you have people who are capable of processing things from people who are on level one. And then level three, you have people who are only capable of processing things at level two because again of the amount of time investment and expertise they have in this particular place. And it's sort of because they couldn't do something else. Definitely move up to a, you know, higher tier, lower number if you want to put in that effort in many cases. Doesn't mean it makes sense. And so then, you know, basically you want to go down the pyramid.

• 43:00Zvi Mowshowitz?

  So the question is, who is going to be able to explain this to somebody down the line? Or who is going to be able to persuade other people on the same level as where I am? Those are the people I'm most talking to. But I do think there's like a handful of people whose viewpoints like matter a lot. And if they were to change their perspectives, change their way of looking at things and make arguments in the other direction, it makes a central difference. And we've seen a number of those people actually make this change, right? But not necessarily people who are previously out there making lots of the most impactful arguments, but it's often people who just by nature of their careers, by nature of their experiences, they command to raise respect. Like most recently Douglas Hofstadter, right, offered Godelastra Bach, who pretty

• 43:44Zvi Mowshowitz?

  much everybody I know looked up to for that I'm a strange loop, all of that work. Now, he thought that super, super intelligence, like, you know, full AGI was very far until super recently. And he has not flinched from being loud about now that I think it's near, I look at what's about to happen and this horrifies me, this scares the hell out of me. I don't know what to do. And, and a few months ago, you had, you know, Yashia Bengo, you have Jeffrey Hinton. You know, these are a substantial portion of the voices out there. And, you know, on the other side, you have a handful of people who are cited majority of the time when people cite reasons not to be worried. And sometimes their arguments are good and sometimes their arguments are quite bad. But if like even one of those

• 44:29Zvi Mowshowitz?

  people, you know, of the like five or 10 people were to come around, I think that would make a significant difference in my point of view. And if a lot of them were to come around, like it would be a complete game changer. I don't think that, you know, most people are doing anything other than looking to a very small group of people for guidance, either directly or indirectly.

• 44:49Nathan Labenz?

  Very interesting. So it almost sounds like you've mentioned you have private, you know, conversations, discourse with some of these and then others I'm sure you're not in touch with. But it almost sounds like you have a list somewhat explicitly of like, these are, you know, say 10 people who

• 45:06Nathan Labenz?

  If they were to change their minds, would make a huge difference to how things go. Would, uh, would you be willing to venture like actually outlining a list of some individuals that you think are kind of most cruxy for the broader path of history?

• 45:19Zvi Mowshowitz?

  So I think there are also a lot of people in positions of like governmental authority or power who would be tremendously impactful if they were to change their minds. Certainly if, if any senator were to come out like pretty strongly with good views, well explained, I think that would buy a tremendous amount of credibility. Certainly if someone like Biden were to do so, it would be huge. Like the Prime Minister of the United Kingdom came out as concerned about this, and I think that definitely made a significant difference. I think there's like a broad category of many people who are not currently having a voice in the debate, who if they were to suddenly like make it clear that they cared about this pretty deeply and became loud, would matter quite a lot. Certainly hundreds, possibly thousands on that level. But in terms of people who are being like very loud on the other side of the debate, I, I hate to spend more time on him than we should, given his behaviors.

• 46:03Zvi Mowshowitz?

  But like, for example, if he on the, right, like he's been very, very, very outspoken, very, very loud, he works at Meta, he is the third godfather of AI, the other two have joined the cause. The third one were to very profoundly and loudly say, oh, I realize I was being stupid. I was wrong. Well, and we could all welcome that. And he started making relatively good arguments and making them for a, you know, more nuanced, balanced position, or even for a, you know, this is dangerous, you know, anything in what we look, I think is accurate position. I think it would be complete overturning of the test board. Like just that one person on his own. And there's a number of other people who, you know, really make a big difference because in many cases, because they are one of the few people who are being very loud in a credible

• 46:47Zvi Mowshowitz?

  way, because of their previous reputations about accelerationism and the need to like actively not take safety precautions or the inability to take safety precautions. Right, and that's why I pay attention to them and I try to address them as much as possible. It's probably to convince those particular specific people themselves. And say, you are making a mistake. Like you are, you are wrong. And I respect you enough that I think that if I were to point this out, you might change your mind. And in some cases I've written, in some cases like I'm a clue, I said, okay, that's not gonna happen. This person is not interested within the arguments. And trying to figure things out for themselves, they have. Made their mind up for whatever reason, for better or worse. It is what it is. In other cases, I still hold out a lot of hope. I think you have to try. Because.

• 47:32Zvi Mowshowitz?

  It's a lot of how things can go well. Also simply people at the labs often like being going to stand up. I think Sam Altman. Did in fact do a lot of this in a positive, very positive direction recently, right? He's come out in favor of strongly. Very good regulatory principles and ideas. And of the dangers that are out there. He, there are other things he's doing that are not as helpful, but this alone was tremendously helpful. I think that like Jack Clark and Dario at Anthropic. Could be making much stronger statements that I think they would endorse in private. Even now. About the situation, I think would help a lot. I'd like to see Dennis Asamas speak out more. I'd like to see, you know, the CEOs. Of Microsoft and Google speak out more.

• 48:17Zvi Mowshowitz?

  Right, like these are people who would, I think, be tremendous game changers if they emphasized the actual. Actual problems involved. But like the list would go on and on, right? Like I couldn't name any number of people who would be tremendously influential. And they haven't proved like any of the standard people who are just like above or below, like people who lots and lots of people listen to. Just who aren't particularly weighing in on us. I think you'd see a lot change.

• 48:39Nathan Labenz?

  Interesting. So yeah, I think one big question I've been very uncertain about. Over time is like. To what degree is this just gonna kind of? Play out how it's gonna play out versus? You know, is there a sort of more contingent? Model of history where, you know, the, the local decisions that.

• 49:00Nathan Labenz?

  You know, individuals make and the, you know, the, the very small scale dynamics actually could make a huge difference. This one seems tough because it's like, on the one hand, you do have these massive trends of kind of web scale data, web scale compute, you know, a whole kind of field that is pushing on this. But then on the other hand, you know, you do have these kind of decision makers who do really seem to matter. It seems like you do think it is ultimately kind of a path of the river, history really matters to what outcome we get.

• 49:28Zvi Mowshowitz?

  Yeah, I think strongly the path of the river will determine the outcome we get. I think there are, like, I have a lot of model uncertainty about this, right? We might live in worlds where the paths of the river are all almost entirely good. We might live in a world where the paths of the river are almost entirely all extinction. We might live in a world in which it's entirely up for grabs and it's very hard to tell. Certain aspects of the situation are very, very difficult to stop, right? We're not going to stop, you know, Moore's law until the laws of physics stop Moore's law. We're not going to stop algorithmic improvements and hardware improvements in some general sense. We're not going to be able to contain the things that have already been released, right? We're not going to contain the profit motives. We're not going to prevent there from being massive investments into AI, into AI applications.

• 50:14Zvi Mowshowitz?

  Unless we're willing to take very strong actions, a lot of these ships have sailed. We have very tough game theoretic problems to solve, but certainly the path from there to here has been very detail dependent in many ways, right? If we sat around until a very small number of people realized because that transformers live because that transformers. The people who founded all three major AI labs that are pushing things forward did so specifically because of events that are directly downstream of Alexey Katsky and his warnings about the dangers of AI, which resulted both in a substantial advancement in AI capabilities faster than we would have otherwise had.

• 50:56Zvi Mowshowitz?

  And people who have a much better than we would have otherwise expected understanding of the problem being in positions of authority over all three labs. And so there are good things about this and there are bad things about this. But if a very small number of people, potentially even one person, had made potentially different decisions about what to emphasize, the discourse will look very different today. The state of AI will look very different today, you know, a different timeline, a different game board.

• 51:24Zvi Mowshowitz?

  And I think the arguments for whatever's gonna happen is gonna happen are something along the lines of either we've already passed the points where we can make these important distinctions and it's kind of too late, you think is not impossible in some sense, but also a sense of there are very competitive dynamics you can't avoid, that the pressures are too strong, that the humans just don't have the coordination capacity, don't have the in-practice ability to make the sacrifices necessary to change the eventual outcome that it's just begun. And, you know, always one can say, "Well, not with that attitude."

• 52:04Zvi Mowshowitz?

  Right? Like, if, if, I think one of the biggest barriers, right, to coordination is people assuming they can't coordinate and therefore not trying. You see this especially in China, where I think that, you know, we have, we definitely don't have strong evidence the Chinese are in any mood to cooperate with us on pretty much anything, on any level, AI or not AI, certainly regarding, you know, world transformational technologies. But we also do not seem to have taken the first step of asking nicely, or put out an indicator of interest. We need to talk to China, we need to engage with the Chinese, and we need to talk to everybody

• 52:46Zvi Mowshowitz?

  else as well, and we need to explore, could we coordinate on these ways? Which first requires us to understand that we need to coordinate on these ways, because there's no "we" to talk to them. There's no "them" and there's no "we" until we understand collectively in some sense what has to happen. But certainly, that's not determined. I see lots and lots of decisions made every day that like raise or lower slightly the probability of things being steered in any particular direction, and the regulations on this and the ways people look at this are incredibly fluid. I would even say that like if you look at the history of media surrounding AI, that the decisions of individual script writers and directors regarding a very small number of properties have a substantial impact on our debate. Imagine if "Don't Look Up" hadn't been made.

• 53:32Zvi Mowshowitz?

  It's very easy to imagine a world where that project wasn't greenlit. And yet it clearly had a substantial impact on the way people looked at things, even though it had nothing to do with it. Yeah, that's sort of the song.

• 53:42Nathan Labenz?

  Yeah, the Terminator also comes to mind there as uh. If there had just been no Terminator series.

• 53:48Zvi Mowshowitz?

  Right. Well, what if the Terminator wasn't very good, right? What if he's giving it to a bad director and hadn't hired Arnold and it sucked? Because it was a normal action movie that happened to have this weird plot. And then nobody thinks about it. Instead, you know. We have lots of people who are going, you know, well, this could be like Terminator. And people are going, well, Terminator is a terrible metaphor, terrible metaphor. And other people are going, actually, the Terminator scenario itself is actually highly plausible if you've got Pantravel. And other similar related things that it's actual, you know. There's a lot of things that media actually gets a lot of questions remarkably right remarkably often. And it's surprising to me on reflection, I'm watching Pursuit of Interest right now. And they get a tremendous number of things correct. On a technology that didn't exist.

• 54:34Zvi Mowshowitz?

  And they couldn't possibly have understood.

• 54:37Zvi Mowshowitz?

  They also get plenty of things wrong, of course, and there are some things that are like intentionally wrong for television purposes. But it's amazing how much it's like, no, this is how L L M's perform. This is what big data is about. This is the kind of influences you can draw. This is the kind of capabilities that emerge when you didn't expect them. This is the kinds of issues you have to think about when you're dealing with the emergence of these kinds of systems for the first time. And what are the ethical considerations and what are the, you know, who do you want to have power over this thing? How do you limit it? How do you align it? What's the alignment tax you're paying? And they don't talk on the same, they don't use the words we use. But they ask a lot of these really hard questions and they come up with like highly plausible ways that something might play out or the ways that, you know, characters think about them might answer them. In a cinematized world.

• 55:21Nathan Labenz?

  You said a minute ago, there is no us and them until we kind of, you know, figure out what everybody thinks. China being, you know, another dimension of this entirely, just zooming in kind of the local, you know, Western. A I discourse. There definitely are some camps. So I kind of wanted to get your. Survey of the camps, how you see them, you know, which are kind of. Maybe can, you know, be merged or could at least work together versus, you know, which are sort of irreconcilable. I guess I would break it down for starters into five camps. Four, four or five. On the sort of, you know, what me worry end, you have the EAC, Effective Accelerationists, which, you know, you can maybe try to steel man that for me if you can.

• 56:06Nathan Labenz?

  So that's nothing to worry about. Then there's like the, you know, current concerns are most important. That's often called the AI ethics community. They often seem to be accusing people of existing in a category that I don't really think anyone necessarily exists in, but that would be the AI safety hype marketers. That's sort of the accusation that people are making bad faith arguments about risks because they want to drum up interest in their own products. So that, I would say, seems to be an accusation from the AI ethics folks against maybe the next camp, which is like the AI safety folks. Basically, you know, that seems to be kind of the centrist position from my standpoint of like, hey, we should definitely take this seriously. It seems like, you know, it could be a real problem.

• 56:50Nathan Labenz?

  As you mentioned, all the leaders of the key developers are seemingly in that camp today with maybe Jan Leakun from Meta as a notable exception. And then finally, you've got your kind of doomers, you know, we're so screwed that, you know, as one person memorably put it, you know, time to start spending more time with your family. Is that how you would break it down as well? I mean, that's kind of the typical, you know, summary that anybody who is trying to summarize this would give, but is that how you also see it? Or do you conceive of those groups a bit differently?

• 57:24Zvi Mowshowitz?

  I conceive a bit differently. There's, there's a lot of similarities between our models. A lot of this is like when you go to Europe and there's like 17 different parties and like mostly they all lie on the spectrum and they're just like, but a bunch of them are just like, here's the pirate party who just believes in ending copyright, right? It has completely nothing to do with anybody else. And like the position on, you know, economic redistribution is I don't care, I'm just giving you my copyright material. And so, or they might actually care as well, but also be really strong about copyright material. So when I look at these groups, I think that, you know, we start with the accelerationists. I think that's, you know, they call them EXL. So I think these people are very strongly in favor of just full speed ahead. And then.

• 58:07Zvi Mowshowitz?

  The question is like, what's going on there? Like, why are these people have that belief? And, you know, I mean, I'm effectively EXL in most places, the same way that like people talk, you know, the famous, well, you're an atheist for every god but one. I just don't make an exception. And so they're just not making me the exception. Like, they're not seeing what my, they don't understand what makes AI different from all other technologies. And why this is something we shouldn't do, whereas everything else that had this kind of promise, we would do. Right? And so like traditionally, people say like, okay, AI and bio risk, like, you know, creating bioengineered plagues and other like risky biological research of particular types and trying to create artificial general intelligence. Well, might not want to do those two.

• 58:52Zvi Mowshowitz?

  But you want to do pretty much everything else, right? You want confusion power. You want your new medicines, you know, you want all your new techs. You want to build all the things you want to.

• 59:02Nathan Labenz?

  Want our VR holodecks.

• 59:04Zvi Mowshowitz?

  Yeah. In, you know, can we into space? Yes, we can, et cetera, et cetera. We're for all the things, which is not for this particular thing because we have concern. And they're like, no. Everyone always has reasons to suspect any given tech. There's always talk of it took our jobs. There's always talk of this will destroy the nature of the family, or this will destroy our way of life. Or humans will somehow be killed by this horrible new thing. We've heard it all a thousand times before. And I think some of that is in fact a legitimate, like, way of looking at the problem from one direction, like a way of grasping the elephant. And a thing to be aware of when thinking about the problem. But basically what's going on is people either

• 59:49Zvi Mowshowitz?

  don't understand why this is different, or choose not to understand, or don't want to understand why it is different. Or just have some sort of weird, like, they generalize so thoroughly as this lining optimism. Or they see no alternative. And so they're presenting a front where they think we should do it anyway. So like Ruin on Twitter is, I think, a very good accelerationist thinker in the sense that he fully acknowledges that it's very easy for us to get us all killed. And there was tremendous danger. But he advocates eyes open, do it anyway. Right? Like you must, like the spice must flow, as he puts it, right? Like you, there's no choice. There's no, there's, there's no plan B. And I think others are, in fact, of this rough opinion.

• 1:00:33Zvi Mowshowitz?

  But they don't say it out loud. Right? This opinion of, well, you know, if we don't build AGI, then our civilization has cut off all these other avenues for progress so thoroughly that we are doomed. So even if it might kill everyone, or it might cause various other catastrophes, that's just a risk we're gonna have to pay. These are problems we're gonna have to solve. We must boldly go and make it work out, right? Boldly going is never safe. Technology, not say, intelligence is just the least safe technology in the universe, right?

• 1:01:06Nathan Labenz?

  Yeah, we're proof of that.

• 1:01:08Zvi Mowshowitz?

  Yeah, most dangerous thing that's ever happened. You know, why should we think to create more of it that isn't us? Is it a safe thing to do? That's crazy. What are you thinking? But also, one could argue it would be crazy not to do it. Right? There's so much available promise, so much value, so much wonder, so little alternative. It's the future. Go do it, or they just think that, you know, it's inevitable. Right? They go, I think some people do genuinely buy the China argument of better us than, or various versions of if not me, then somebody else. And that somebody else will be less responsible with it, or have worse values, or both. And so there's just no use in holding back. And I think there is various people in this group of various levels of being genuine about it,

• 1:01:53Zvi Mowshowitz?

  of understanding or not understanding the danger, of, you know, facing, internally facing down what actually is going on and what, and whatnot. And I, I try my best to give people the benefit of the doubt. Right? To, to treat as many people as possible as actually understanding things as well as possible, while also being as, as good faith as possible and wanting good things. And I do think most of our race is definitely want a good future for the human race. And want a good future in terms of what they see as utility. And I will stubbornly cling to this as much as possible. I think there are a handful who, you know, I would say there's kind of a fixed category that's beyond EXL, which could be called the extensionists, right?

• 1:02:37Zvi Mowshowitz?

  Could, they're, they're, they're, they're machinists. But the people who in the free body problems said that this world belongs to Trisolaris. Except for AI. They think that AI is the future. They call it speciesist if you say anything otherwise. They think the AI is high value. They're more, they'll be moral patients. They'll have more complexity. They'll think we're just more tricked bots. They're, they're what matters and we might die and that's okay. And to them, I say I disagree. Right? I just completely disagree. We have to give them our different moral codes and I think we have the right to have preferences. And it all works.

• 1:03:12Zvi Mowshowitz?

  Right? Some of these people think you're not allowed to have preferences, right? Like objectively, in some sense, is better. So you're not allowed to say, I'm human and I, and I have children and I have grandchildren. I want that to continue. Because that's a, you know, it's, how do you justify that? I'm like, I don't have to. Right? You could have bond arguments about that, but I don't have to. And so, you know, there are good accelerationists and there are less good accelerationists. And like, the accelerationists I respect, you know, most acknowledge that there is a problem. You know, advocate looking hard on solving that problem, but think that like slowing down doesn't help you solve the problem. Introduces more other problems. Or is just impractical and just think, well, we might as well be forthcoming. Right? So you divide the accelerationists into the ones who are, you know, useful, you know, good faith accelerationists who want to solve our problems together.

• 1:03:57Zvi Mowshowitz?

  And the ones who think that, you know, the way we solve all our problems is we just keep going forward without worrying about our problems. And our problems never materialize. As opposed to every problem in human history. Right? We just, we've had tons and tons of problems in human history. We solve them by actually solving them. We solve them by realizing there's a problem that if we don't do anything, it's going to be very bad. And then figuring out what to do about it. So these problems have been very, very hard. Some of these problems have been like, well, it's easy, but if you literally just treat it like it doesn't exist, it's going to be a real problem. So then, you know, the safety hype manufacturer, they don't exist. I'm going to flat out deny the existence of the safety hype. I think that there was no motivation for this. It does not actually make sense.

• 1:04:37Zvi Mowshowitz?

  Doesn't benefit anybody and the people who are, you know, people who say that they're talking their book. I think it's a combination of they're talking their books, right? Instead, they're trying to explain why the opposite outcome should happen. So they attribute bad motives to the other players because they have those motives themselves. Or they're just so cynical and so second on mindset where everybody's always talking their book, whether even if they aren't. But they can't imagine the idea that anybody could possibly say anything that wasn't profit motivated. There was a post today, in fact, for example, on Pirate Wires. It was like, well, everyone is just talking their book from all points of view. Everybody is just saying whatever their incentives say to say. Because I think this person just can't imagine that there are people on many sides of this who are saying what they believe.

• 1:05:23Zvi Mowshowitz?

  Or advocating for what they think is good for the world, which would also be good for them. Like, I don't get to say, well, you know, I think the world's going to die. But in the meantime, there'll be really good companies, so I should support the acceleration of them.

• 1:05:38Zvi Mowshowitz?

  I also don't get to say, well, you know, I think it's all going to be fine and create a paradise. But, you know, I can get more subscribers if I say we're all doomed. So I'm going to sacrifice the future of humanity for readership. How would that make me better off, right? If the world's going to be wonderful, I want that wonderful world to live in far more than I care about anything else. I think it's all just silly, right? If you want to stop this emitting technology, you want to stop it because you're afraid of it. You want to stop it because you think it's a big deal. It's a problem. I don't think that means that everybody who comes to do a position is doing so for logically great reasons, right? Like any other position, a lot of people come to that conclusion because they had a vibe. Or the people around them were worried.

• 1:06:22Zvi Mowshowitz?

  Or they've hallucinated or come up with a different problem that's actually kind of solvable. Or they're imagining a highly unlikely scenario as almost inevitable, just like every other point of view. They do exist. So like, I think then, you know, jumping ahead, I'll get to the middle afterwards. We have the doomers as such, but I think this is a wide variety. So like, you sort of enter this space, not when you say there's an X percent chance. Like, a lot of people, there are people who are like described as doomers, like Paul Christiano, for example. Who then you ask them what their chance of doom, they have like 10%, 20%, like something reasonably low. But they talk about doomers because they are saying, well, if we don't treat this problem as an impossible level problem, right?

• 1:07:07Zvi Mowshowitz?

  If we try to play like we're still playing guitar here on medium, we will definitely utterly die unless we get super lucky. Like it would be very, very bad. But they think, well, with an extraordinary effort, we can set up a deal to be impossible. We can make this happen. But we understand that we need to shout from the rooftops just how dangerous the situation is to get that to happen. So that we can then solve our problems, right? Same way that like, if I didn't think that we were capable at all of coming together and making these extraordinary efforts, I wouldn't be at the team, I'd be much higher. Right? A lot of that is, we will in fact do things that are very impressive. In some sense, we will make an extraordinary effort. We will figure things out that I don't know how to figure out and we will solve our problems.

• 1:07:52Zvi Mowshowitz?

  And then there are people who actually think, you know, like, oh, yeah, Zierkowski, you think the probability of doom is, yeah, more than 90%, more than 99%. And need to do whatever gives us any chance at all of success. Or alternatively, like the problems are so incredibly hard that if you don't have a mindset that we are super doomed, none of the things you do will be useful, right? If you, you can't solve our problem by treating it as a normal problem. You have to treat it as an impossibly hard problem. Have any hope of your solution actually moving us towards the real solution. And if you try to solve an easy problem, we're just not helping. Or maybe even at all. So then there's people, people who talk about AI ethics.

• 1:08:33Zvi Mowshowitz?

  I think that's a very strange middle ground where I think some people are like, whoa, there's, there's only people who are like, whoa, there's, there's problems of AI because I'm against AI. And I'm going to cite every problem I can about AI. Or who genuinely just see disaster on every turn because they're anti new technology. They're anti-growth, they're anti, just like intelligence in general in some sense often. And they just see the downside of everything. And so they tend to emphasize these ethical or mundane concerns over the doom concerns, but they would endorse the doom concerns probably as well if they thought it was useful in solving the technology. They just don't see it as especially useful in solving the technology.

• 1:09:15Zvi Mowshowitz?

  Or they don't see it as directly addressing like the particular harm that they worry about as directly. So they don't pay as much attention. And then there are people who like see the short term harms as real and see the long term, the longer term harms as irrelevant. And it's not clear to me again, like if they are coherently thinking about the long term harms. I think a lot of them are, you ask them, is there in fact a danger, a existential risk? Would respond with new or no answer, like just no opinion whatsoever. Just, we can't think about that right now. That's a distraction. Is there a real answer? They actually haven't thought about it. And it's unreal if they don't care. Like we'll cross that bridge when we come to it, if we come to it, for now.

• 1:10:00Zvi Mowshowitz?

  Like, I really think life is about bias. Right? There's some people who really do think that that's what civilization dedicate itself to. Is fighting bias, fighting misinformation, all of these little micro causes compared to the things that we have. And one can argue whether or not the thing they think, how important that is. But I think it's a lot less important than whether humans live or die at all. And I'm not sure they agree, right? They, I think some of these people, well, if we can't solve these problems, we don't deserve to live or something like that, or even, I don't know. I don't want, you know, there are lots of different people over there. We're trying to characterize them. I think of the AI, you know, the AI ethics people

• 1:10:43Zvi Mowshowitz?

  as the people who put what they call ethics above, you know, what actually, the things that matter to them, right? Above survival of humanity, the general, generally getting good outcomes for people, generally ensuring there'll be like things of value throughout the universe. These are things that I care about a lot. And I also care about most of their concerns. I think those are important concerns. But they see my concerns as a distraction from their concerns. I see their concerns as

• 1:11:20Zvi Mowshowitz?

  Essentially a distraction from my concerns to the extent that they use that as a reason to dismiss my concerns. But we don't have to do that. We can say, we have to solve both these problems. And in fact, these problems have complementary solutions. Any solution that helps solve my problem, that helps us keep us alive, will help fight bias. Any solution that helps us get the AIs to do what we want them to do, that helps keep them under our control, will prevent the AIs from spreading misinformation, will prevent them from manipulating their users, it will help with all of that. Right? And so, and when they do their work on their thing, it's at least not harmful. I welcome that work. I welcome all of that work.

• 1:12:03Zvi Mowshowitz?

  But when you do things like argue the distraction thing, it's, it's, that's when we get into a fight. We get into a fight because you choose to make it one. You choose to think of this as a zero-sum game, as opposed to we all benefit when we do all of this work, is the way I look at it. And if you have a proposal for mitigating near-term harms, I will compare the costs and benefits and I will hopefully support that. That's a good idea, but it can't be our focus. And if we only have so much room to pay attention to these things that we have to prioritize, and I know where my prioritization is, and I hope that I, you know, that will also help with other things that we have to do. And so, in general, I would say you have the accelerationists, you have the people

• 1:12:46Zvi Mowshowitz?

  who are, I call them the worried, the people who are worried about AI killing your person, because doomers is kind of pejorative and we don't wanna, you know, I don't call you fakers, right? Why are you calling me doomers? But, yeah, I'm willing to own it on occasion. You know, we have people who are worried, which makes it clear that like you only have to be so worried to be worried, right? You know, 10% is worrying enough. People who think we need to take this kind of action that won't happen on its own. We have the people who, you know, actively strive to distract in order to complain about other AI risks and people who like...

• 1:13:16Zvi Mowshowitz?

  Yeah, and then I would say the AI safety hype market factor, again, they aren't real. Instead, you have a pretty bad faith faction of the ethics people and a bad faith faction of the accelerationist people who are claiming this to try and win an argument in ways that don't really make sense.

• 1:13:34Nathan Labenz?

  Cool. I think that's pretty much all in line with my perception as well. Couple just point level follow up questions on a few of these camps. On the accelerationist front, is there any decent argument, you know, what's the most compelling thing you've heard that would give somebody reason to believe, actually, yeah, there is a, even if you don't think it's the majority case, there's some decent chance that this could just go really well and actually turn out to be easy and never really was anything to be worried about in the first place?

• 1:14:05Zvi Mowshowitz?

  Well, you give me the hard question, right? It's not just that, like, there's a, an easy hard question, which is, what's the best argument for why we should go boldly? Right? Which is, it won't be any better if we wait. Right? Basically, there are arguments that basically by waiting, you create what's called overhangs, where we have the ability to do something much more powerful, we choose not to. And we encourage bad actors to be the ones who are first movers, because the good actors just rain themselves. And we risk, like, there being too many people who have capabilities in question. And sooner or later, someone's going to do the thing we're worried about. And the only way to stop, you know, a bad, an irresponsible person with an AGI, or creating an AGI, is to create

• 1:14:48Zvi Mowshowitz?

  the responsible AGI first. Or that only by getting close to AGI can we possibly be able to do the proper research that will let us figure out how to make it safe. And so the time we spend putting our thumbs now, just give us less time later. We should instead accelerate right to the edge, because we can judge where the edge is, which I don't think we can, but like, we can much better judge where the edge is. And then we can take our time, then we can put in our Manhattan Project, our huge amount of effort, whatever you want to call it. Then we can solve our problems and ensure good outcomes. I think these are like, not facetious arguments. I think these are reasonable arguments. I think they're overcomp. I think they're, they're not what carried today. But I understand why someone would advocate for that. The argument, well, why should we think that everything is just

• 1:15:32Zvi Mowshowitz?

  going to be easy? People make various arguments of this type. A lot of them are content free, right? Like, like the, Andreessen literally said, you know, well, the good guys always win. Right? And we're the good guys is implied, right? So win, which means the AI must be aligned. Like, if that's simple, like, we'll figure something out. And obviously, that's kind of nonsense. But you can say things like, like, a lot of people say, well, you know, well, like, the Yom Kippur is like, well, you know, or some other people like that often say things like, well, you know, if it was sufficiently intelligent, it would be smart enough to realize that its goal was stupid, or that, like, morality is correct, and that it would therefore, like, do good things because it was smart. Right? Like, Scott Aronson has expressed views like this, for example, I believe.

• 1:16:18Zvi Mowshowitz?

  I've seen it a number of places.

• 1:16:21Zvi Mowshowitz?

  I think it's a bad argument. I think there is no reason to think that a more intelligent thing would be inherently moral, or would deliberately and consciously, like, reject its goals in favor of goals that we like, or that, like, you know, any of the sort makes any sense. I think that what we'd be hoping for is, like, something like, it turns out that if you kind of vibe on, like, Alm's style, on the things that are good, and you, like, enshrine a bunch of messy principles that don't really have logical coherence, well, that's kind of how humans act normal and nice to humans, right? One could argue, right?

• 1:17:03Zvi Mowshowitz?

  And then they sort of generalize this thing, and they realize they're supposed to, like, be in a sort of consistent, reflexive way, friendly and positive, and cooperative, and care about other people, and then they actually internalize that because that's the best way, like, having positive incremental goals is the best way to get positive final goal, to get positive final outcomes that will be approved of, that allow them to achieve their goals in general. Therefore, it's good to be as humans that are, like, pretty friendly in ways that, like, correlate with intelligence, or you could argue, correlate with intelligence as well, but yet even smarter than us, can take harder for longer, process more data, get wiser, like, and again, has more experience, can process more data.

• 1:17:45Zvi Mowshowitz?

  This thing will then, like, converge on some sort of place that humans would converge on if they had more thought in terms of, like, what is, what is good in life, what should be done, what could be a good and positive answer, as long as we, like... Give it sufficient images in the right direction, but like you don't have to be very specific. It'll just sort of, it's an attractor state, right? Like the, the, the good outcome is in the trap. That like, we don't have to get a point of measure zero, right? On a giant graph of exactly the thing that does the thing that we want. We don't have to specify the goal. We don't have to understand what we want either. You just let the AI figure it out and the AI will be vaguely well meaning and the AI will cooperate.

• 1:18:24Zvi Mowshowitz?

  And then the dynamics amongst many of these AIs that are like positive or the way I created a positive will lead to a good outcome. And so all we have to do is not like intentionally make this terrible. Right? Like obviously you create chaos GPT and you set up these bad names. Like most people would agree with that. But you know, no one would be so stupid as to. So like, you just do the ordinary best thing, which is like some combination of our OHF and constitutional AI and you know, basic ingraining of good principles and then you do various tests. And then when it starts to exhibit bad behaviors, you correct it. And though it won't suddenly, it won't then learn how to do this via deception.

• 1:19:05Zvi Mowshowitz?

  It won't learn how to pretend it's thinking what it's not thinking. It won't, you know, secretly record data in places you're not looking so that it can survive various iterations. And it won't like pass things on to the successor states and do all these tricky things. It won't figure out things we ever thought about, but do these things. But just do the obvious basic thing, because that's what most intelligences on that we've ever known have basically done. And that's like my steel man case for like, why is it relatively easy? Is it like, yeah, it's kind of what naturally happens. Just like, it'll just sort of happen. Like would be the, the optimistic case. And to be clear, like, no, right? That's not how this works. It's not how any of this works. I don't think it's like completely impossible.

• 1:19:50Nathan Labenz?

  Yeah, I was gonna say, it does, you know, large language models do seem a heck of a lot closer to that than I would have expected five plus years ago, right? Like if you just asked me, okay, we make some AI and it's like an AI that's breaking through and is like becoming really useful. And people are all, you know, attention is turning to it. And you don't tell me anything about the nature of that I, how, that AI, how it was trained or anything. And then you said like, you know, how much do you think it will kind of, you know, understand or be able to kind of act in accordance with human values? I think we're way on the high end of my expectation with LLMs, which is not to say that I disagree with you, I don't think in the big picture,

• 1:20:32Nathan Labenz?

  but just that, holy moly, like there's some sense in which we're like very lucky, I think, with the nature of LLMs, right? A much more kind of alpha zero type lineage probably doesn't produce anything along the lines of what we do seem to, you know, kind of learn in some abstract sense just from all the language.

• 1:20:52Zvi Mowshowitz?

  So I think that this says we may have been lucky and we may have been unlucky, and it's really hard to tell which until crunch time. In the sense that, you know, you can tell a lucky story where these things are like much better at grokking like more or less what we mean and more or less what we care about in these kind of fuzzy, messy ways than we would have expected. They're not going to be ruthless optimizers because like it's just not the nature of the mind that we've created in some broad sense. And so, sure, we're getting, you know, human-ish things, maybe at the end of the day, in some ways, and then if we scale them up, they'll still exhibit these characteristics, and then we're in a much better state.

• 1:21:34Zvi Mowshowitz?

  However, the tough side of that is that you get these things that are impossible to predict or understand, that are very squishy, that are very inscrutable, and that are not being particularly coherent in many ways. And that like if they go off the rails, it'd be very hard to like prove or detect they went off the rails in a meaningful way. They're going to like misunderstand us in what we actually care about importantly. And we have no hope of actually injecting like be that thing if we figure out what it is. Because they just can't take exact things at all in that sense. And then maybe that's just not good enough, right? Maybe, maybe even the best version of that just doesn't get us what we want.

• 1:22:18Zvi Mowshowitz?

  And will start to break down pretty fast. One thing is to note that sort of there are often these very rapid transformations, where the best solution to a type of situation or problem, the very like nature of it changes and you're best served by throwing out all your previous work. And then putting in a lot more work on this more intelligent, more work level. And do a better job. And so for example, you have deception, or you're telling people what they want to hear. You know, having a direct like, what, how are you going to respond when you see this? Like by directly modeling exactly how these mental processes work is one of those things. Right? And there are many things like that. And I've been trying to write out this argument in more detail the last few weeks. I'm trying to get it right.

• 1:23:03Zvi Mowshowitz?

  But essentially, right now, with its current level of understanding, just trying to sort of messily represent human preferences and reflect them is the best thing I can possibly do with its current level of capabilities. Right? It's deception wouldn't work, but doesn't really understand what you care about very well. Whereas if it understood very well, like it's going to tell you what you want to hear. Absolutely, the thing will tell you what you want to hear. It does not, there's no secure truth optimization going on here, except insofar as we punish it for finding out that it's wrong. And if you had an AlphaGo style problem where it's just optimizing, if you knew exactly

• 1:23:45Zvi Mowshowitz?

  what you wanted, you optimized for exactly the right thing. Like, maybe you could win, right? Outright win. Get something great that would then do the things you wanted. So it's kind of a, right? If you get a food, or if you get like a very precisely defined optimization target, and you hit it correctly, you win, and if you miss, you lose. And like, there's not that much middle ground, and there's not much room for you to correct an error. Right? Like, if you said that you made a mistake, you're stuck with a mistake. And if that means you still get some value out of the world, you get some value out of the world, but you can't get back the part you lost. Right? You just, what we've lost is lost. Dead is dead.

• 1:24:27Zvi Mowshowitz?

  Whereas of these messy L M's, like, you don't lose that way. You don't have this really scary thing where it suddenly decides to like, perform this really rigid optimization in a way we never approve of. But you also don't get the good thing either. But you you don't get this precise like ensuring that things go well. You just have this kind of messy situation. And this leads back to my concern over competitive dynamics. Where if you unleash a bunch of like messy, humanish things that have messy, humanish like alignment. But they're also better optimizers than us and they're more capable and they're more efficient and they have all these advantages. Yeah, that they don't immediately turn around and kill us on purpose or anything. But then actually do it sound much good. This is our.

• 1:25:12Zvi Mowshowitz?

  Right, like that's not efficient condition. For this to turn out well. And that's assuming kind of the that our current methods don't break down. Like our current methods hold. In a separate way, so if you tell the story that I told. Which is actually, you know, you give them a bunch of feedback as to vaguely what we want and they kind of. Generate some processes that kind of internalize the things that we want and that generalize and they turn into a genuinely. Like. About as aligned to us type of aligned thing. Well. If you suddenly like created a bunch of things that were. You know, it would have quickly outcompeted us with numerous advantages. That had about our morality, you wouldn't survive very long.

• 1:25:57Zvi Mowshowitz?

  Like certainly we wouldn't have every grandchildren. But like this idea that like there is much hope in that level of. Approach. I haven't really understood why. Right, even if it succeeds. And also I just don't think it happens automatically. Right, I think that what happens is. The AI suddenly does a grocking thing, right, where like when it goes from ten to the X to ten to the X plus one or ten to the X plus two. Cycles. It suddenly realizes that it can do a completely different thing to get the outcome it wants. That it's being rewarded for, it does that other thing instead. And we find out that all the things we did before are suddenly useless. I think that's by far the more likely outcome.

• 1:26:34Nathan Labenz?

  Yeah, grocking as a phenomenon, I think, is one of the most under appreciated and under. I mean, people don't even know about it broadly. Uh, but that that paradigm has shaped my thinking probably more than. Any other over the last couple of years of the, you know. Moving from memorization to generalization and the unpredictability of that. Uh, it seems like, yeah, just such a critical, critical point. Going back up the stack then, the I that was my one question on the EX, but the. AI ethics and safety.

• 1:27:07Nathan Labenz?

  It seems like, from my perspective, probably really dumb that these camps, you know, kind of are so at odds with each other so often. And I would say there's blame on both sides of that, not to blame present company at all, but there is kind of, I'd say both sides kind of say, your, you know, concern is a distraction from my concern. And that has kind of, the battle lines have kind of hardened along those lines. If there's any hope that that could change, I mean, we're starting to see a little bit of thaw. And I guess my understanding of why there's been maybe some thaw and could be hoped for more, working together in the future, is maybe that as such focus and attention and

• 1:27:49Nathan Labenz?

  resources come to this area in general, maybe there's sort of an omnibus spending kind of thing that everybody can kind of say, yeah, you know, I, much like we have, you know, in these, these kind of big tent, big budget bills that get passed, right? Like, well, I don't care about what you're doing. You don't necessarily care about what I'm doing, but we're both kind of doing something that this is going to fund. And so that could be good. Maybe that kind of brings camps together into a, hey, what matters is we invest in this, we invest a lot. And, you know, if all of a sudden there's going to be, you know, huge governmental funds poured into it, then there's kind of enough for everybody, you know, to get funded and do their thing. You don't look

• 1:28:32Nathan Labenz?

  too pleased with that scenario or hopeful for it?

• 1:28:35Zvi Mowshowitz?

  I wish the primary intervention that made sense was to pour a bunch of funds at this. Right? That would make things so much easier because then, yeah, absolutely, we can all agree, you know, 100 billion here, 100 billion there. I get my money, you get your money.

• 1:28:47Nathan Labenz?

  Pretty soon, you're talking AI safety.

• 1:28:49Zvi Mowshowitz?

  Well, pretty soon you're talking about money, hopefully. But like, I don't know if you're talking, the problem is I don't think you're talking AI safety, right? Like, I, I think it helps. But, you know, I don't think this is a case of for money at the problem. And then like, we hide the cost of getting that money back from the taxpayer, but just having it be any more spending though. I think you have to impose actual costly restrictions on what companies can do, on what models can be developed and played and used in various ways. And some of those restrictions help with one problem, some of those can help with the other problem. And you can require these people to pass certain tasks and to go through certain hoops. And again, like, there's competing for a kind of limited resource here in a much more clear way. This makes it harder.

• 1:29:32Zvi Mowshowitz?

  I do agree that there's no reason these characters to be distinct. If they are, there's no reason they're to be each other's throats. I think a lot of the reason why, you know, the AI doomer, right, side of this is so against the ethics people is one, they know the AI ethics people are going to try and completely sideline the doomer people. They're going to try and ignore all these concerns, get everybody to ignore all these concerns. They see that in advance, right? It's the, well, you know, if, you know, AlphaVille is going to attack Betatown, and Betatown doesn't like AlphaVille. Right? Like, it's just, that's just how it is, because you see the conflict coming. We're fretting over limited resources. I'd be fine if P Street did that. Well, it's a problem. Part of it is

• 1:30:17Zvi Mowshowitz?

  their arguments are incoherent because they imply our problem, and then they just deny it. Right? To a large extent, we see this and we get really frustrated and mad. It's like, well, yeah, of course, if these people who are like, well, you know, the AI will never be able to fool people enough to take over the world. The misinformation from AI-generated content next year is a serious problem in our election. It's like, really? Right? Like, you can have either of these positions and it's fine, and it makes some sense. It's really hard to have both in some important sense, right? Like, you think that, like, these much less capable systems that we fully understand and that we can manage in real time, recovering from our mistakes, are these dire threats that require these responses?

• 1:31:03Zvi Mowshowitz?

  And yet you can't understand by scaling them up a lot would present the next example of crap. And that's just, so, like, at one point, you know, someone was like, you know, it's like being on a bus, and the bus is about to go over a cliff, and someone worries they're going to be late for work. Well, yes, you're going to be late for work.

• 1:31:22Zvi Mowshowitz?

  But it's very hard to reach that conclusion about noticing the bus is about to go over the cliff. Right? And if you know the bus is going to go over the cliff, maybe you should have other problems, that you should worry about more. But at the same time, like, you know, once we steer this bus away from the ledge, you have to get you to work on time. It's important. Right? Like, it matters. And, you know, I do think that's like an example of a metaphor that's like, oh, too dismissive, and pisses these people off for no reason.

• 1:31:51Zvi Mowshowitz?

  And we have to move on to to things that are are more positive, right? Like, I don't want that to be the go to. And so I cited as an example of both like how it's easy to interpret the problem from the side of someone worried and how it's easy to like be somewhat unfair to those people who are on the other side at the same time. But there really is this inherent logical contradiction going on a lot of the time. Where they just don't want to see it, uh, because they can't imagine it. Like they they they can imagine things in front of their face, not the things in the future. And also, like, we've seen this time again, where like these types of concerns hijack the political process, regulatory process. And then, like, we used to call our concern AI safety.

• 1:32:35Zvi Mowshowitz?

  And these people came in, they just started using AI safety to mean don't say bad words. And now we have to find new words for what we're saying. And like, they pretend they're us. They pretend that they're doing the thing that we're doing, and they're just not. Right? And it's like, doesn't mean their thing is bad, or their thing is not necessarily. Which is incredibly frustrating. And so, yeah, we end up with each other's throats in bed, and I wish we didn't. As I said, like, I think there's room for us to all cooperate on this, right? Because we all want the same things. But also, I just think that they're objectively wrong about the near term concerns, like the AI ethos. And also that, like, they're wrong about their preferences in many cases, right? And so, like, you would have this agreement over, like, what is specifically going to happen?

• 1:33:19Zvi Mowshowitz?

  Like, if you unleash this AI generation, like, what? You know, are people going to be fooled by these deep things? Are they going to think that their girlfriend cheated on them because they saw an AI generated picture? Are they going to think that Donald Trump was in fact, you know, killing Godzilla? And that Joe Biden was being, you know, impaling innocent young virgins on stiffs? Like, I don't know. And then, like, they'll be fooled and they'll vote for Trump and it'll all be terrible. And like, no.

• 1:33:47Zvi Mowshowitz?

  It's not gonna happen. We're gonna be able to deal with that. Is my opinion. And like, that's one disagreement. Another disagreement is they think that certain things are really, really important. But I don't think they're as important relative to the potential utility gains that are involved, right? We have this ongoing disagreement in general about progress and about technology and about growth and about like flourishing of humanity. Many of us, right? And the ethicists, people who call themselves ethicists these days, right? Medical ethicists show like this, bioethicists show like this, you know, and so on. Their attitude is harm is what matters, right?

• 1:34:30Zvi Mowshowitz?

  The specific harms that I am worried about trump your, they call it greed or desire or wants. But like they trump the good that can be done. I think it's just not true. Right? Like, I think these harms should be considered. They should be mitigated. We should do our best to avoid them. But, you know, when we don't, if they have one of the people who like, you know, don't do challenge trials for COVID vaccines, right? Like, could have saved so many people. It'd be funny to a few people who would have gladly volunteered. We didn't do it because the ethicists told us no. And the ethicists are constantly like in a medical context, telling us no. And imagine that will save us, that will save lives. Right?

• 1:35:14Zvi Mowshowitz?

  Where everybody involved is in agreement, is consenting. They don't care. They think it's not okay. And like this is usually less egregious than that, but it's the same idea, right? It's, they think that there are the deontological rules that matter more than the utility involved, right? Or than other rules and other principles. And I think they have the wrong rules. Right? I am following a set of preferences that's very different.

• 1:35:46Nathan Labenz?

  I'm sure you have a reconciliation for this, but I think one place where people might be a little confused about your views is obviously in that, you know, last moment, you sound quite consequentialist, utilitarian, whatever. Earlier, when talking about sort of AI descendants as a path and like how much value you would ascribe to something like that, it sounded like you were not willing to.

• 1:36:10Nathan Labenz?

  You know, go so far on utilitarian thinking as to say, well, if the robots have a lot of utility, then that it's good enough for me. So I wonder, what is the, could you kind of unpack those underlying principles a little bit more? It's like a certain kind of utility that you care about the most? Or how do you think about that?

• 1:36:28Zvi Mowshowitz?

  So I'm a virtue ethicist, first of all, just flat out. If I have to select one of the big three to describe how I think. I do think that like, if you can't express your view in all three at the same time, your view is ill conceived and you haven't actually figured out what you want. So like, I should be able to define a utility function. It's just going to be really complex. And like with the human amount of data and compute, virtue ethics is the correct way to express what I want. Similarly, the ontological rules, right, are very, very useful. Again, because we have limited compute, limited data. And, you know, it's very useful for ourselves. And these calculations come intensely complicated and involve lots of terms we just don't know.

• 1:37:09Zvi Mowshowitz?

  But we have radical uncertainty about many aspects of the universe and like, we lack the data to do the calculation problem. And like, the utility of the ontological implementation is higher, right? So we should use that. But my recognition is very simple. I care about myself and my descendants, my fellow humans. And I don't care on the same level about the AI. And it doesn't mean that I think they could have no value, but I don't think you can do math. But I notice that you can't just do math. Right? You can't, like, certainly there are people I know who would say things like, well, you know, each chicken has this much value. Each worm has this much value.

• 1:37:52Zvi Mowshowitz?

  If you simulate this many worm and this many GPUs, it might generate this much value. You have them all have orgasms all the time and it would have this much value. And I'm like, no, you've obviously done something. Right? Like you're, you're, you're taking your, your metric and you're treating it like it's real. Right? When it was supposed to be an approximation, a way of figuring out how to handle situations. And you're taking it out of distribution, right? This is exactly what we're worried the AI with in many cases, right? We teach it what is good by giving it examples in a distribution. And it learns in that distribution how to optimize for the right outcomes. And then we take it out of that distribution by either treating it with problems that weren't in the training side or are likely in the training side.

• 1:38:37Zvi Mowshowitz?

  Or just like giving it a lot more capabilities than anyone in these situations that we were modeling had at the time. And that inherently makes every situation radically different. Or you know, for other things. And then we say, okay, what happens now? And the answer is you get something weird. You get something that doesn't make any sense usually often, or that we wouldn't prefer. Because the data that you had doesn't actually tell you the answer. Right? Like utilitarianism is a good way of correcting for the fact that like, you have to do math and two of something is more important than one of something. And 20 of something is actually sometimes 10 times as important as two. But not always.

• 1:39:17Zvi Mowshowitz?

  And it's not that simple. And also like,

• 1:39:21Zvi Mowshowitz?

  human preferences are not inherently very coherent or like very consistent. You can try to make them as consistent as you can, figure out what you actually prefer on reflection. But if you just say, okay, you take a system with a contradiction, right? It's well known. If you take any system of math and you start with a contradiction, you can prove anything you want.

• 1:39:43Zvi Mowshowitz?

  Right? So like if you start, if you're a utilitarian and you write down an inexact equation and an approximation, and you start manipulating those inexact equations a lot, you're going to get nonsense results. Visually out of distribution with enough steps. And one of those nonsense results is like, well, maybe we should simulate a bunch of worms having organism. This is dumb. Right? Like we all know this is dumb. Right? No matter like, and I can accept now that.

• 1:40:13Zvi Mowshowitz?

  If you build a bunch of AIs and they wipe us out, that universe, so I think, I think most of those universes, right? The majority of them are pretty close to zero utility. I evaluate them, I was like, the AI is not good, the AI is not bad, it doesn't love me, it doesn't hate me, I don't have love or hate it. It's just using my atoms for something else. And that's probably unfortunate, but like, whatever it is I care about is not by default there. Right? Like, I don't really care about something that's optimizing for a W24 metric. I don't really care about something that's optimizing for many of the things that might be somewhat technically complex, but which are particularly interesting. Something that is like effectively pure power seeking is not very interesting to me, is not very valuable to me.

• 1:40:58Zvi Mowshowitz?

  And also, I really don't think you have to take this sort of disembodied, non-positional utility perspective and say that's what you have to prefer. I think that's not okay. Like, you know, if we live on our world, and the Volkans live on their world, and the Klingons live on their world, we're allowed to prefer humans to Klingon, you know, we're allowed to prefer humans survive and the Klingons die, and the Klingons live and the humans die, even if under some abstract utility, there's no real difference. We're allowed to care about ourselves. In fact, I would say it is a key part of my survival, that like, you need to care about yourself and those around you

• 1:41:42Zvi Mowshowitz?

  more than random other people, or the entire thing falls apart and doesn't work, and nobody gets anything to them. Right? If, if, if parents don't care for their children more than they care for other people's children, this doesn't go well. Right? And so like, in some sense, you can realize that like, there's a contradiction, and there's a conflict, and there's a kind of a mistake, but... It has to be that way. It's so important. So I did, I don't, I don't talk philosophy in my blog. Because I know that like philosophers who have spent lots and lots of time on this could rip anything I say to the ground. I understand this. But at the same time, I have fallen in love with these things and I believe strongly that you are allowed to care about yourself and care about things you care about.

• 1:42:26Zvi Mowshowitz?

  And define your identity function and what you think are important things to zip. And I've made my choices and again, these people called themselves ethicists. I've made very different choices. And I'm not saying you're not allowed to have those preferences, legitimately, if they have those preferences.

• 1:42:44Zvi Mowshowitz?

  I can say I think they're wrong. In the sense that I actually think they, they, they do not in fact refer the world they are claiming to prefer. On reflection, the world that they're claiming to disprefer.

• 1:42:55Zvi Mowshowitz?

  And that if they actually reflected and reconciled their positions, they would have to change their minds. But maybe they might do. I don't know. I, you know, it's a different kind of disagreement. Right, pardon me.

• 1:43:09Nathan Labenz?

  Let's imagine all the humans are gone. And then there's some AIs or no AIs and then within some AIs, you could have like, you know, unfeeling, no, you know, qualia, no subjective experience, AIs that are just, you know, paperclip maximizers or whatever. And then on the other end, you're on the other hand, you might have some AIs that have, you know, some sense of feeling and subjective experience and maybe they can, you know, do even some of the things that we could do, like they could appreciate some art or whatever. But they're, you know, obviously can be still quite different from us, presumably. It does sound like you distinguish between those, like you would rather have the art appreciators than the non-art appreciators and you'd rather have more than fewer.

• 1:43:53Nathan Labenz?

  But then at some point where people start to push on, well, what if there was enough of them or, you know, uh, what if they, you know, what if there was a trillion trillion of them? Then at some point you just kind of get off that train and say, well, you're not, I'm not going to allow you to push the multiplication problem to that extent. Because, you know, I just feel like that's out of distribution entirely. And so I kind of fall back to a more brute force fact of my own ethics, which is I care about myself. And so, you know, you can't push me that far. Fair.

• 1:44:23Zvi Mowshowitz?

  I certainly think I have a preference order over the non-human universes. I am confused about how much I prefer those universes and how much probability of survival I would trade off to get the better universe. If I fail versus the worst universe like that, I think that's unclear stuff. But I do think that, yeah, it's it's not that multiplication can never do it. It's not that like you couldn't convince me, who was officially convincing K's autophlection that like there's a situation in which I prefer certain long-distance futures, other long-distance futures. But I also have a sense in which like, it's not my role to prefer.

• 1:45:05Zvi Mowshowitz?

  Uh, the system doesn't work. If people do that, like we're supposed to prefer that the things that we care about survive. And there's no reason why, in theory, we couldn't spread those things out to, you know, spread throughout the universe in some sense. But, you know, I don't, I don't think this is an area where anyone can figure it out. Right? I don't think that anybody really knows what does and doesn't have certain types of properties or what like these completely out of distribution universes, right, with these completely out of distribution things, what they should think about them. How they should prefer different configurations to other configurations.

• 1:45:45Zvi Mowshowitz?

  And this is one of the dangers is that like we're, we're potentially setting the configuration of the universe in devotion relatively soon. With no idea what we actually prefer. Right? And what matters to us for reals at this scale? And that's really scary because if we don't know how we each an AI, we don't even know. That doesn't mean that like we haven't figured out what we think, like we just legitimately are deeply confused. And our feedbacks are not going to reflect anything that we want here. Then we're going to sort of iterate on that somehow with some sort of system. It's, it's a giant disaster waiting to happen. I wish we, if I had a long reflection, I would take it, right? If I had the opportunity to suddenly like, you know, have me and the people around me like not age and live for a hundred thousand years.

• 1:46:30Zvi Mowshowitz?

  And I kept looking at everything and talk about this stuff for a long time and just do philosophy. That sounds great, but we don't get it. Right? Like we have to solve all our problems first before we get to there. It's an important sense, uh, before we can do that. And so we have to like have these squishy messy. Views on how to do these things and like muddle through the best we can. But yeah, I noticed that I prefer not to die and I prefer my children not to die and I prefer them to have children of their own. And I prefer that the human race continues in general. And I think I am very much committed to that preference and it is good that I have that preference. And hell, you're welcome to try and talk me out of it, but I don't expect, I don't want anyone to succeed.

• 1:47:16Nathan Labenz?

  Yeah, I won't attempt that here. Uh, certainly I'm not the best person to try as I think I again, mostly uh, very, pretty similar in my position. Including even just, I would say probably weighting virtue ethics more heavily in my own thinking than, you know, a pure utilitarian consequentialist, whatever. Popping it back up then, right? I was asking my point level questions on these different camps. And we're getting to the doomers, AKA the worried. I think people who are coming to this relatively recently and who are encountering this, you know, line of thought relatively recently. First of all, have no sense for just like how long it has been around and kind of how deep it has run. I wonder if you could kind of give a little bit of a just short story from your perspective of kind of,

• 1:48:01Nathan Labenz?

  what the AI safety, you know, evolution has been over the last 15 years since you kind of first got interested in it. And I think it's especially interesting to, to help people understand how much kind of care. And I sometimes call it AI safety respectability politics has gone into just trying to be, and not always succeeding, but you're trying to be the sort of people who can be trusted or can be respected or like, don't seem crazy on other dimensions. So I'd love to just kind of hear that story from your perspective a little bit.

• 1:48:36Zvi Mowshowitz?

  From my perspective, you know, I hadn't given AI any thought for a while. And then I read these series of blog posts by O S U Kowski and Robin Hanson. And they were engaging in this debate over, you know, what would it look like in our distant future when there was artificial intelligence? And I had these arguments again, as I thought about the implications. You know, I realized that.

• 1:49:02Zvi Mowshowitz?

  If this technology were to exist, you know, everything would change.

• 1:49:06Zvi Mowshowitz?

  And that, oh, he was basically correct. That by default, if you built these super powerful, yeah, if you built something smarter than a human,

• 1:49:17Zvi Mowshowitz?

  then it would be able to build something smarter than itself still because humans were able to build that thing.

• 1:49:22Zvi Mowshowitz?

  And it would be able to scale up its resources quickly, pretty quickly, by default, again, by default, or not automatic, not necessarily automatically, but by default. And if you didn't have this thing able to preserve some property of what it was optimizing for in a kind of like known groupable way, that thing would change in a weird direction and also was actually remarkably hard to specify anyway. And that this thing would take control of the future, right? We didn't use these, these are the terms that we use now. It's almost hard to remember exactly what terms I was using in the head back then. But it was a long time ago. It's 15 years, something like that, before now. And

• 1:50:03Zvi Mowshowitz?

  at the time, we were trying to solve this, you know, the people like I was with, Cassie, were trying to solve this problem by creating mathematical systems, stringing together various different problems together would allow a system, while it was self-improving,

• 1:50:18Zvi Mowshowitz?

  to preserve provably its utility function.

• 1:50:22Zvi Mowshowitz?

  Such that whatever you put into it as the thing it was trying to accomplish, it would still accomplish that thing. Right when it was done. Because we didn't think it was an L M. What was it? Go fun, a go old fashioned AI, right? It was basically a logical enterprise. And if you could preserve that, well, that was not an easy problem. And then you had to like actually make it so that like preserving it, that you had a thing that was worth preserving that would then lead to a good outcome as well. And so, you know, at one point, there was a meeting that I was a part of where like I was with a bunch of people from what then called, uh, Singularity Institute, now is called MIRI. And we, we out, and there was, and you know, it was outlined, okay, here's the sound things that need to solve.

• 1:51:06Zvi Mowshowitz?

  Solve these seven things, then you can solve the alignment problem on a kind of GoFi. And also you've kind of built the AI as well, like incidentally along the way by doing so. Certainly, or like you're pretty close, you have to understand a lot of the principles of all that in order to do this. And then you can build it and it's saved and everyone's, right? And then I was told at the end of the meeting, like, and so before I was told the seven things, like, forget where they are. Like, it's an info hazard for you to know what they are. Like, you might tell somebody else, you might act in a way that reflects what they are. And like, we don't want this stuff getting out, but it helps to a build they asked. I actually did forget.

• 1:51:47Zvi Mowshowitz?

  I remember one of the six things, one of the seven things. Right? Which is like, a pretty non-dangerous one to remember, but like, forgot the others. And, you know, throughout the years, they did very hard work to try and figure out how to make progress on this type of agenda. You know, this Asian foundations trying to kind of make agents that acted predictively, acted in a way that you could prove things about, rely upon. They would work under iteration and improvement. And this was deep math, like hard mathematics, right? Hard enough math that when I asked if I should work on it, I was told.

• 1:52:22Zvi Mowshowitz?

  And they worked on this, but it was a very small number of people. And this effort, like, basically didn't make much progress. But we, we got a number of things written, we figured out some things out, some very interesting advanced things. We're not that, never got that close to a solution. And then, over time, you know, as we better understood, like, all the ways we couldn't align an AI, all the things that made the problem harder. Like, we didn't make progress in that sense, but we discovered all these different reasons why AI was really, really hard to align. Right? And we explained them really well, we made better decisions during that anybody else. Which is one of the reasons why, like, and all this time, like, it was often said, why don't you go into academia? Why don't you, not me specifically, but like, Eliezer and people like that, why don't you go get PhDs?

• 1:53:06Zvi Mowshowitz?

  Why don't you go write formal papers, why don't you submit them for review? And, you know, the answer to this was that, you know, these people are not inclined to. Listen to and understand and, you know, fairly judge and then, you know, use arguments that like the field evolves too slowly in this type of way because it takes decades to have any influence and by the time it's too late. And like they're all because of we treat it as theoretical, we treat it as like you don't have any evidence. We're not getting things in. And the one place where they touched on the theory was in decision theory. Right. So like decision theory is an academic discipline. Which asks, okay, how do we decide what's the best thing to do? What's the best decision to do? And academic decision theory basically has two decision theories.

• 1:53:50Zvi Mowshowitz?

  They have evidential decision theory. And they have causal decision theory. The causal decision theory is like the common sense decision theory you would have, which is calculate what's the expected value of doing X, what's the expected value of doing Y? If X is higher, do X. If Y is higher, do Y. Right. That makes perfect sense. It's a sensible thing to try. The problem with this is it's vulnerable, right? It doesn't, it has no way to commit to things, for example, like without making physical like punishing. It has to make like credible pre commitments in order to commit to anything. Like, otherwise it will just like defect when it's time to like go into the room and like either cooperate with the police or not. Because that benefits it directly.

• 1:54:33Zvi Mowshowitz?

  So like CDT agents, right, like have these set of weaknesses, right? They won't pay part for hitchhiker. They won't cooperate in the prisoner's dilemma. They can't coordinate with each other very well. Right. And they also respond to threats. And there's some other problems as well, but basically they have lots of well-known, very bad.

• 1:54:52Zvi Mowshowitz?

  It's also advocates for evidential decision theory. Evidential decision theory said, do the thing that will make you happiest to have learned that you just did it. Right. Which is a very, very strange sounding proposal for having decisions. And there are some obvious places where this does really, really dumb stuff. Right. Where like you look at like, what the hell are you doing? It also solves some very weird problems pretty well. And so like, there's basically a lot of arguments back and forth about how this problem is better here. This problem is better there and how this is obviously wrong and that's obviously wrong. They're both obviously wrong. And so a bunch of people working on the rationalist puzzle. Right. Worked out this thing that's called logical decision theory or functional decision theory.

• 1:55:32Zvi Mowshowitz?

  With various iterations, which is act as if you are choosing the output of the logical process that you are using.

• 1:55:40Zvi Mowshowitz?

  Which basically means that things are correlated to your decision, right? Things are made, their decisions are being made the same way you're making your decision. You should act, you should act as if you had to make those decisions as well, right? Even if they're in the past, the future, or separate, or separate in space. And obviously, like, you have no physical control over them. You still do this because that will lead you to make better decisions. It will lead to better outcomes. And so this is just vastly better. Like, once you understand it, once you work it out. And like, there have been papers, and putting like a hundred page, like, academic length, like, academically formatted, like, properly written papers, that are very, very good, address the previous literature, address the concerns, go step by step, do all the things, and they just got ignored.

• 1:56:22Zvi Mowshowitz?

  And then to me, like, oh, this is the place where our contribution was, like, very clearly, incrementally making progress on the existing scientific literature. And providing a just clearly superior solution to a problem, rather than like, warning about its obscure, esoteric long-term danger. And it just got ignored. I mean, yes, we could have all pursued, like, a bunch of PhDs and tried to, like, establish a certain credibility, at least to tap our lie, to turn, to listen. But it makes sense that we didn't, right? It's an important thing. We just, like, instead, choose other systems of knowledge. And meanwhile, Eliasson builds up, you know, together with people who decided to, to, to work with him, his entire rationale is seen on the principle that, okay, I tried to warn people about AI. But they couldn't think well enough to understand my arguments. They understood my arguments better.

• 1:57:07Zvi Mowshowitz?

  This is what he was thinking. I'm not endorsing, like, this, this entire thought entirely, but... He thought, well, okay, we need to teach people to think. And once we teach people to think, they will be able to understand my arguments. He wrote the sequences and generally dedicated the entire community figuring out how to think better about things in general, in the hope that we think better about this one thing in particular. I think it basically worked, to a large extent, but not to the extent that Eliasson needed it to work, in order for his plan to succeed. And so, it's not about the success, it's about the failure. But it greatly enhanced my life and the lives of many people around me and thus their thinkers. So, I'm very grateful for that. But basically, you know, I, I'm thinking about this AI situation for a while, and I'm thinking to myself, okay, we have to solve this very hard problem, keep it making progress on,

• 1:57:51Zvi Mowshowitz?

  but also, like, how close is AI really? You know, no reason to panic, go about your life, it's fine.

• 1:57:58Zvi Mowshowitz?

  Then, you know, DeepMind happens and then OpenAI starts to happen. And then, you know, the origins of OpenAI, you can variously look at Sam Altman and Elon Musk and the interaction with GPT-1, and concerns about Google and concerns about, you know, what Larry Page said to, you know, about being the speciesist and the other claims about exactly what happened. But basically, this company gets founded by people who are worried about the power of AI. The same way that GPT-1 founded DeepMind, because he was worried about the power of AI and the dangers of AI because of Billy Henson. This is what they say in science.

• 1:58:39Zvi Mowshowitz?

  And so, this entire time, we've been talking about info hazards and holding back what we thought were ideas that might enhance AI capabilities. When it turned out, the very fact that AI was like a danger was a dangerous idea all along. Right? All of our ideas about capabilities, yeah, it would have been somewhat exhilarating to share all of them right away, but like, this was so much worse than all of them combined. But it turns out, right, it got people excited, like, ooh, look, this might kill everyone. I better get involved in that. That was like really unfortunate, that's like what we saw. And then they started going down this line of these things that we didn't have any ability to control or understand.

• 1:59:19Zvi Mowshowitz?

  Like even in theory, you just throw a more computer at them and then like, couldn't stop it. Right? Like, and even had organizations and various people like invest in the effort to try and get voice, like to try and like be able to influence it for the better. Which I think, you know, didn't have zero effect, but mostly, like, I think was a mistake. Very clearly a mistake. That we should have been more outspoken about how much we didn't want to go down this road from the beginning. But, you know, that's one end of the bridge now, right? Here we are. And, you know, the concerns I've been expressing, you know, for a very, very long time now,

• 2:00:01Zvi Mowshowitz?

  15 years in various forms, they're not mainstream exactly, but they no longer sound crazy.

• 2:00:10Zvi Mowshowitz?

  They're no longer something you can just miss out on. And they now have a lot of very credible people, very, very much authorities and experts and key players in the industry endorsing them. And that's a tremendous place to have gotten it sometimes. And now, you know, I also, we're in a world where like the alignment problem is much more fluid and like there are real systems to be worked on where you can make what appears to be incremental progress. And a lot of the question is, does this incremental progress do anything? Right? So like, a lot of the debate, one of the debates that we haven't discussed in the discourse is the extent to which, if you solve how to make the current systems do what you want,

• 2:00:53Zvi Mowshowitz?

  if you figure out how to align them for practical purposes, now, to what extent are you developing useful techniques and methods that then scale up and then will work in the future when we have dangerous ones? And to what extent, even if the current techniques don't work, are you learning key information that will then allow you to develop the techniques in the future that will evolve from the current techniques and those techniques? Or is this solving an easy problem that does not actually help you with the hard problem? Because the hard problem is completely separate. Or something in the middle where like it's completely separate track, but doing the easy track first gives you insight into what the hard track might look like. Right? You learn a thousand ways not to build a light bulb helps you build a light bulb, but you can't actually build a light bulb that way.

• 2:01:38Zvi Mowshowitz?

  And like, if you think you do, then you die. But if you know you can't, maybe it's very helpful. There are a number of ways to go. There's a lot of arguments about it and a lot of it's very technical and a lot of it's very detailed and has a lot of branches and a lot of people think about it very differently.

• 2:01:53Nathan Labenz?

  If I understand you correctly, I think maybe one of the things I think maybe I see most differently or interpret most differently is to what degree has sort of the Eliezer school of thought won versus, you know, lost by shooting itself in the foot, by, you know, by becoming sort of accidentally accelerationist? I'm guessing that maybe hinges on a core question around like, to what degree are AI systems of sort of roughly the capability that we have now kind of inevitable, given the data and compute inputs already coming to exist? Because I guess my theory right now is like,

• 2:02:38Nathan Labenz?

  it sure didn't seem like it took all that long from the time we got web-scale data and web-scale compute to get some, you know, pretty good working architectures, namely the transformer, you know, and derivatives. And it's hard for me to imagine a counterfactual history where if deep learning kicked off in 2012, you know, and that's whatever, 10 years into Google or something, and then 2017, we get the transformer. It's hard for me to imagine a 2023 where we're not like pretty far down that path, you know, kind of even if Eliezer had never been born or whatever. And so I guess I look at it as a pretty big win

• 2:03:18Nathan Labenz?

  that those that are doing the development at least have like a certain healthy fear of what they're creating. Because it seems awfully easy for me to imagine a counterfactual world where the technology continues to progress and like the leaders are just, you know, totally uninterested in any sort of concerns, just don't want to hear about it. But it sounds like you see it more the other way, if I understand, like more that if if there were no Eliezer inspiring these people to get involved, then what would your expectation be? Like that we just wouldn't have this technology at this point in time or something different?

• 2:03:53Zvi Mowshowitz?

  I think it's unclear, but the way I think of it is if there's no DeepMind, there's no OpenAI, there's no Anthropic.

• 2:04:04Zvi Mowshowitz?

  If DeepMind sort of didn't have OpenAI, they would be moving quieter and probably slower and in like fundamentally safer rates. Right? Like Google was kind of a system that they kind of like giddy up and like deploy powerful systems to do real things, it could be as possible and loudly as possible. There wouldn't be this like orders of magnitude expansion of money. Now, I don't think you keep the bid on this forever. Right? As the amounts of money required to do interesting things continuously goes down, at some point, some open source person or some relatively less funded lab makes GPT-2 style thing at some point, it doesn't cost that much money. People see the potential, they start scaling up, it happens a number of years later, is my baseline scenario, and then we're off the races again.

• 2:04:50Zvi Mowshowitz?

  You can also argue that like, some amount of safety concern is inevitable, the same way that some amount of capability is inevitable, because like, these arguments are hard to find once you start seeing things. A lot of people who woke up to the risks of AI didn't come like in the last year, right? Didn't wake up to them because we said, do. Yeah, ChatGPT was enough. Like Douglas Hofstadter and Geoffrey Hinton and a bunch of other people, they don't really cascade off, right? Like, they knew thirty years ago that if this happened, right, humanity was in deep danger. Because it's just obvious to them, right? Like, it's like you see all the quote things back in the fifties about how this is obviously a danger, Alan Turing, right?

• 2:05:32Zvi Mowshowitz?

  But what you, what they did think was, this is just not going to happen anytime soon. Right? We're not, we're not going to succeed, right? I'm chasing the male, but I mean, I'm chasing the male, the mailman, but I'm not going to catch him. They didn't think about what would happen if they caught him or they were about to catch him until they were like, oh my God, I'm going to catch him. Now what? Uh-oh. It's actually really bad.

• 2:05:54Zvi Mowshowitz?

  I don't like this. Now what? And like, we gave them a better, you know, it made it easier to think, wow, but like, how much better? I'm not sure, not clear. Whereas

• 2:06:06Zvi Mowshowitz?

  the key concern is that like, just because you're thinking that safety is a thing, just because you understand that you have to solve a problem, doesn't mean that you appreciate the problem in a way that allows you to actually try and solve it for real. And this, I think, is where this is fair. Like, if we had you cowsky in, like, true, like, if we had full, like, Miri-style security mindset, enable awareness of the true level of danger that I think we're in. Like, which is, yeah, less than, oh, yes, it is, right? But if they understood just how impossible the problems are in front of them in terms of like game level difficulty, not in terms of like can't be at all.

• 2:06:51Zvi Mowshowitz?

  And understood how important it was now that we're entering this realm where like potentially you're just as likely dangerous, to proceed with the proper amount of caution and understood things like it might kill you during the training run.

• 2:07:04Zvi Mowshowitz?

  Not just after you deploy it. Understood like the ways in which you had to structure these systems to minimize the chances that that things would happen. Understood that your solutions had to be like really, really robust and thought out carefully. It couldn't just be sold up kind of vague pushes in the right directions. But that like the chance of that working is very low and like it's very hard to test to see if it's gonna work without doing something dangerous. They haven't thought hard about how to do that either. It just generally like had a much, much more radical approach to safety.

• 2:07:35Zvi Mowshowitz?

  Then I'd say, yeah, the trade was worth it. But I don't think we're seeing that. Particularly, right? I, I think that, you know, Anthropic was the one that claimed they were doing that like pretty consciously. But if you look at their investor pitch deck. And you look at what Jack Clark and Dario are saying and not saying. It doesn't reflect that level of paranoia. It doesn't reflect that level of caution. And they may feel that with the competition, they don't have a choice. They're not gonna be right. But it's still, they're not doing the thing that makes me feel better about it, right? And there's a question of whether or not like someone who's marginally better, it makes it a three horse race than if it's two horse races, making things better or worse. Like even if they are in fact better in that way, and it's not entirely obvious.

• 2:08:20Zvi Mowshowitz?

  It's always the question of, you know, is a little bit of safety just a way to fool yourself in thinking that you're not gonna get yourself killed and getting yourself killed? Or is a little bit of safety good? You could also look at it as if it weren't for our obsession with safety, these things wouldn't be useful. Right? Cause like any work on alignment is likely to have immediate utilitarian impact on the mundane usefulness of the current models. If we didn't have our OHS operating really well on current GPT, we wouldn't have ChatGPT. We wouldn't have our current explosion of investment. What we have, we'd have this kind of like pretty useless thing that gives like

• 2:09:05Zvi Mowshowitz?

  not very useful outputs and that will often say things that are pretty racist.

• 2:09:10Zvi Mowshowitz?

  Right? And then like, what do you do with that? Like, Google has had this problem for years, right? Where they were able to make pretty strong chatbots and probably the intelligence, pretty strong generative AIs. But they couldn't align them well enough to make them the consumer product they were willing to release. And so they had to win cheap, right? In those cycles. And so you could argue that like, by focusing on these alignment questions, we enable the capabilities that enable the explosion and enable these orders of magnitude of investment. Otherwise, we should be sitting here. So I think it's all very complicated. I try to think too hard about the counterfactuals from the past because we can't unwind time.

• 2:09:54Zvi Mowshowitz?

  That's the part of the memory that's not realistic. We have to move forward from the game board that we have. And so, you know, you hope that these people are sufficiently aligned and filled, sufficiently concerned, can be made so, can be worked with. And you hope that the game, you know, the actual structure of the problem is that it can be solved.

• 2:10:13Nathan Labenz?

  So what do we do now? Like, you know, many parts to that question, obviously. But, you know, do you think that the Overton window is now wide enough open that everybody can just be honest and say what they think? Do you think that things like ELIZERS, you know, we should be willing to call in airstrikes on rogue data centers is a good thing to put out there? Or sort of still too inflammatory and hard for people to hear? That it's just like counterproductive in its own way?

• 2:10:40Zvi Mowshowitz?

  So, so I don't want to be too dismissive of concerns about that type of statement. But, you know, it was July 4th yesterday, right? And one of the things people were celebrating was free speech. The way they celebrated was to point out there was this woman in England. It got sentenced to 13 months in prison, I think it was. It was just over a year for being a troll on the internet. Right? And like, she just said mean things on the internet, like social media. No one, nothing bigger than bad happened. It was just being an asshole on the internet. And like, she apologized in court with her lawyer for being an asshole on the internet because it took over a year in prison. So like, for being an asshole on the internet, men of guns came to her house, put handcuffs

• 2:11:24Zvi Mowshowitz?

  on her, dragged her to prison, and locked her up for a year. Right? Like, we enforce all sorts of laws, and all of our laws are enforcing the barrel of a gun. This idea, there is some sort of radical transformation because it's this different thing that we're trying to enforce, is kind of silly. It's kind of unreasonable. What's going on is that Eliasser is owning up to the implications of what it means to legislate, what it means to prohibit, right? What it means to prohibit something from happening is that you will use force, if necessary. You will start by asking nicely, maybe with economic persuasion, but if necessary, you will in fact use men of guns.

• 2:12:08Zvi Mowshowitz?

  Or if you are facing a point of men of guns, you will use men of tanks, or bombers, or whatever it takes, if necessary. If there was an armed body against the United States, we'd absolutely bring out the equipment. You know, and that, you know, what happens every time people defy the law and just won't back down and start shooting at us, we escalate. We have to. It means to have laws, and it means to enforce things. And so, you know, proposing is law. Proposing is regulation. Proposing is rules. And, you know, these are rules and regulations that like, are much milder than we would apply to many things that are far less dangerous. Like, this is not an outlier in terms of the degree to which we are contingent on people's

• 2:12:53Zvi Mowshowitz?

  freedom of action, certainly not, you know, freedom of speech, or freedom of privacy, or anything like that. We're talking about restricting a very specific, the very specific uses of a very specific type of very technical hardware that can be manufactured only under the most difficult conditions of anything that man has ever created. It really wouldn't be very difficult to track. We have turnip school control on already. But we're basically prohibiting this thing from going to China as it is.

• 2:13:19Zvi Mowshowitz?

  And what we're proposing is that if you want to use that, you try and potentially create a more powerful intelligence optimizer than a human,

• 2:13:30Zvi Mowshowitz?

  right, to go into the realm where that's possible. That you need to get permission to do that. And maybe we don't want to grant that permission under any circumstances, anytime soon, right? And I think that is 100% in the opportunity window. 100% very reasonable. And yes, it implies that somehow, in theory, if Russia were to smuggle enough A100s, or whatever their two systems down the line successor is, into a data center in Russia,

• 2:14:05Zvi Mowshowitz?

  and they were thought to be training an Alan that might kill us all, we would do what you would do if you thought that was an existential threat. You would take it out. But that's not going to happen, right? It is an absurd thing to say. Because the whole point of what I was trying to say is, if you make it clear what you are willing to do, you don't have to do it. Right? If you make it clear that like, under no circumstances will we tolerate this very specific action. You will take whatever countermeasures are necessary. And they believe you? They don't do it.

• 2:14:45Zvi Mowshowitz?

  Because it would be crazy to do it, right? Why would you go down that path, right? Nobody wants that. Whereas if you don't commit to such actions, then they do do it, right? If you just voluntarily say, oh, I'm not going to build the thing, but you know, if you build it, I'm going to hand stop you. It doesn't work, right? So, you know, the hope is that simple economic...

• 2:15:12Zvi Mowshowitz?

  Rules and regulations and trackers and other such influences are able to act on this very, very complex supply chain and this very, very advanced set of systems such that this is not something anybody can just do. And in fact, if you follow the logic, right, of the Chinese can't have these chips and they can't be allowed to train advanced AI systems, well then, why are we letting anybody in the world train them on our data centers and rent our compute whenever they want to do this? Doesn't make any sense. Why are we encouraging open source models that are powerful enough that China can literally just copy and run?

• 2:15:57Zvi Mowshowitz?

  Doesn't make any sense. Right, like, and yet, at some point, in capabilities, and you have to draw that line, you can't simultaneously think these things are as important as we think they are and not put restrictions on them. Even if you don't think they're that difficult, right, there are many other good reasons to try and do this anyway, but certainly if you think there's an extension risk. You know, everybody understands, I think at this point, the only way to prevent these things from coming into existence that doesn't impede tremendously on our liberty and on our economic value and our economic success and that might actually work is to target

• 2:16:40Zvi Mowshowitz?

  chips, target concentrations of chips, uses of chips, target the training of intense training runs, large compute runs, frontier models. Exactly where we draw that line, you know, the 10 to the 23 flops, the 10 to the 25 flops, the 10 to the 6 flops, and we try to define exactly what capabilities they're allowed to have, we try to use evaluations to try and see what can we do in a gray zone where it's permitted but only under certain conditions, it tends to have a gray zone, how do we adjust this over time as algorithmic improvements occur? These are detailed questions that we need to sort out, that I don't have the best answers to. But I don't see an alternate proposal on the table that could possibly work, right?

• 2:17:23Zvi Mowshowitz?

  To not put some limit on how many flops you can use in a training run or some similar restriction and to force that restriction absolutely.

• 2:17:35Zvi Mowshowitz?

  Unless you have specific permission and only get a specific permission under very, very clear detailed circumstances at best is the only way that we have. Any other method of control will be circumvented, right? You can try to say, okay, don't turn them into agents. You can't use our UDTs. It's a hundred lines of Python code. That's not going to work. You can say, don't use them for this particular application. Okay, those are just words. How do you intend to do that? It's not going to work. Right? You can do any, there's nothing else that anybody is proposing that has any chance of working. And, you know, we are very fortunate in this sense, right?

• 2:18:17Zvi Mowshowitz?

  We are very fortunate that the thing that enables AI training runs is in fact the most complex, hard to factor thing that has ever been created in the history of the world. And that basically friendly countries that value US economic dominance control the entire supply chain. It did not have to be true, but it is true. And given that, we have a unique opportunity to exert real control over what happens next that gives us a chance. And we have to take it.

• 2:18:50Nathan Labenz?

  Would it just quickly get your thoughts on like any other promising paths to safety? I'm a big fan of mechanistic interpretability work on its own merits, even if it's not, you know, a great contributor to safety, although it seems promising to me in its own way. And then also kind of want to get your take on, so what do we do from here, right? If you want to see those kinds of rules put in place, you can, you hear fairly similar things from the likes of a Sam Altman, as you noted earlier, but then what do you do about that? Do you follow Tyler Cowen's advice and start going to the journals? It doesn't sound like you're too bullish on that. Do you like start writing more Eli Ezer style fiction, you know, go back to the Terminator and try to scare the voters into electing the right people?

• 2:19:34Nathan Labenz?

  Like what's, you know, do you kind of go lobby at some agency?

• 2:19:38Zvi Mowshowitz?

  I don't see anything that I love as an alignment approach. I think it's super optimistic, but I see a number of things that we can try that would at least find us ways not to align an AI that would alert us to dangers, that would help us understand the problem at a minimum and maybe, you know, lead to something that would lead to something that would lead to a solution or something like that. And maybe we'll get really lucky when we think we'll work out better than I expect. Interpretability.

• 2:20:04Zvi Mowshowitz?

  I think it's one of these dual technologies that, you know, because interpretability also helps you utilize the AI and helps you make AI work okayable. So it's not entirely safe to work on that, but I do think it's worth working on. I do think it's a good idea. And I do think that like a significant chunk of our resources should probably go towards various forms of interpretability effort. So for, it hasn't gone that great. We've made some strides, but like we are not very good at understanding G P two, using G P four. And the way I think of this as, okay, you, interpretability starts helping you, when G P T five can understand G P T six. Right? We're G P T five and a half. The, the, the whole goal is for something to understand something smarter than itself.

• 2:20:46Zvi Mowshowitz?

  Because otherwise, you have a chicken and egg problem, right? Like you don't, you don't get to trust the thing that's telling you the information. So if I can interpret something that's more complex than me, that's amazing news. That's the target we have to hit. Right now, we haven't hit a vastly easier target, right? And we're not anywhere that close to hitting it, but we can try. Um, and in general, like, you know, I talk about what is the thing we do, the regulatory open door window. This has to go hand in hand, obviously, with a ton of interpretability and other safety work. Because if we don't figure out the solution to the problem, eventually, algorithmic improvements will catch up to us.

• 2:21:29Zvi Mowshowitz?

  We don't have infinite time, no matter what we do. We have to solve these problems as best we can. We're gonna have to let's we take our shot.

• 2:21:37Zvi Mowshowitz?

  All we're trying to do is buy time. Like, we understand this. And like, maybe we'll get super lucky and the algorithmic improvements will, I guess, curve out. And then we'll be able to stay here indefinitely. But even then, like, every, you know, every, every day, we make the conscious choice not to break those limits.

• 2:21:54Zvi Mowshowitz?

  And that, again, won't last forever. And also, there are other things to worry about. We have to eventually do this. So we have to solve these problems. In terms of what we, you know, individually.

• 2:22:08Zvi Mowshowitz?

  can do. I definitely think it's a case of some of us should do one thing. And some of us should do the other. But some of us should work on, some of us should work on. Safety directly. So we should try to solve these problems. Other people should work on, you know, lobbying and trying to talk to people with decisions. Other people should write papers. Other people should write blog posts. Other people should try to just. Orient, understand the world, communicate as best they can. Other people should work within the labs. You know, it's, it's not, we don't know which of these things is the best approach. I am in fact going to make an effort to get some of the papers ready. Uh, because. You know, I don't have that much hope that it's going to work. But.

• 2:22:53Zvi Mowshowitz?

  It's, we will learn in the attempt. Right. If we make this serious attempt, at least we will learn why it fails or how it fails. And we'll be able to demonstrate, oh, look, this failed. And here's it failing.

• 2:23:05Zvi Mowshowitz?

  And we will also probably. Think well. In the course of doing that exercise. Right. It wouldn't be as efficient as thinking well without that exercise. If you were optimizing something else, it's pretty good. It's a reason people write papers and figure out things eventually. I do despair about the speed of the enterprise. I do despair of. You know, Tyler is asking for. Economic style papers because he's an economist. But that will only convince economists.

• 2:23:33Zvi Mowshowitz?

  Right. And then you have to like, repeat this process 20 times. Because different people have different perspectives. And a lot of them are going to be less amenable than Tyler's. Like Tyler's is relatively. A good fit, right? Like you should be able to argue. Carefully. In his mode. A lot of the outcomes. And a lot of people believe that like the sort of. The need to do something is vastly over the term, right? It's not. If you had to prove a model went to journals that boom is going to happen, I think you'd be in a lot of trouble because like it's the kind of thing they just reject inherently for just not being scientific or something. Or like not having the right data or evidence, but like I think there's other models that you can in fact like

• 2:24:16Zvi Mowshowitz?

  get pretty into the mathematical model and it leads other people if you wanted to. I just don't think it's really a natural exercise that you would do if you were just trying to seek the truth. But if you need to go through this process, you can do it. Right? And maybe it's maybe it's actually a good idea and like again, division of labor, I think there's a lot of people who want to help with this problem. Who have a variety of skill sets who are kind of throwing up their hands. Right? Like the point that most frustrated at Heli is is at the end of every podcast, people will ask him, so what do I do about this? If I'm terrified, but I think we're all going to die if we don't do something and I want to do something, what's something that I should do? And like, he doesn't have anywhere to send money.

• 2:24:59Zvi Mowshowitz?

  He doesn't have anywhere to work on a concrete problem. I mean, he had to like, you know, try to orient, try to think about the problem. These are good things to suggest. Right? Try to figure it out for yourself, but like we do have to be better. Right? And I do think there are now a number of AIs that like are rapidly expanding that are like have their have net positive impact and expectation and like have their hearts in the right place. I do think you have to worry about dumb money. But if you go in to a charitable ecosystem or like an ecosystem of work and you start throwing around money without understanding it. And a lot of people do the same thing, the incentives get all worse. And like bad, bad actions drive out good.

• 2:25:43Zvi Mowshowitz?

  So like you can't just steal that stuff without thinking about it carefully. You have to orient, you have to get your own independent opinion. You can't just trust whoever is telling you things or that those bands go badly.

• 2:25:55Zvi Mowshowitz?

  But absolutely there should be support for a lot more alignment efforts. And in fact, we got very good news today. I don't know if you saw it or the podcast came out, but OpenAI made an announcement. That they're devoting 20% of their required compute so far. There's going to be a lot less than 20% of their compute over the next four years to level these scaling up their computers as fast as possible and it doubles every year, but still 20% of their existing compute over the next four years to working on alignment. To try and generate a more automated, you know, more advanced alignment process. I have deep skepticism. For the strategy they are pursuing. I think that as something that could possibly hope to scale to solve for human level or beyond human level intelligent alignment.

• 2:26:41Zvi Mowshowitz?

  It's pretty hopeless, right? Or not necessarily hopeless, but none of the things they're going to, they're announcing they're going to work on. Address the things that make that problem hard.

• 2:26:51Zvi Mowshowitz?

  The hope is that they might make it easier to try and address the hard problem because you will be able to iterate faster on the problems when you built the tools. And the compute will potentially allow people to try things that aren't listed there. It's basically, you know, they said like build a human level alignment researcher to help with their alignment work. And as I understand it right now, until, it happened today, so processing the announcements, it's not, you know, free. This doesn't quite mean actual human level alignment researcher. Like if that was true, it would be like, was commonly proposed as the worst idea. Which is get the AIs to solve alignment for you. Which basically means put the AIs first human level problem as the alignment problem.

• 2:27:34Zvi Mowshowitz?

  Which is like one of the hardest places to ensure that your AI is actually aligned. To test to see if it's doing the thing you want, to see if it understands what you want. It's like a great way to get absolutely killed. Right? It's a great way to absolutely get a misaligned system. So much more likely than so many other ways. But what they actually, I think, meant was simply have a human level way of evaluating whether a given output seems aligned. And humans are pretty bad at this.

• 2:28:05Zvi Mowshowitz?

  So getting an AI that's good, as good as a human, a generic human, doesn't seem like a crazy thing to ask for. However, I still think it's a bad idea and I'll be, I haven't written about this, I don't understand it yet. Right? Like, I understand things by writing about them. But basically, the, the only way to align for real on human preferences is to get humans who are thinking deeply and well about their preferences. And you deeply understand the question at the bottom. You can't learn from training data.

• 2:28:38Zvi Mowshowitz?

  And human feedback, things that aren't in the train data and human feedback, that are less, that are more sophisticated than what you're being given, until the point where you can pull out the complexity. So like, if you have a very large amount of data and a very large amount of feedback, you can draw inferences and correlations and build up a model of all of that, where you is, you know, the thing under gradient descent, that is more powerful than the thing that's driving the individual components. But there's still an upper limit, if you're trying to decode what was going on, that if it wasn't in the thing that you were decoding, it won't be there to find. So you need, you're going to need not just like

• 2:29:22Zvi Mowshowitz?

  the equivalent of the random person that you're paying a normal hourly wage to go chat with the chatbot, or to go just check for some offensive language or something. You need someone who is pretty intelligent, who is thinking carefully about all of the implications of what they're saying, and is going to give you pretty sophisticated feedback.

• 2:29:46Zvi Mowshowitz?

  And as a parent, this echoes very much in like, when you're observing and trying to give feedback to your children, right? Like, you need to be very smart, very, very careful about exactly what feedback to give to your kids. And with humans, it's much, much worse, because they're operating in much less compute, and like, infinite words of magnitude less data and less cycles than AI. So every little bit is going to have so much power to reshape their world and their, their way of thinking. But you have to think very carefully on so many different levels about, okay, what caused that output? What is this person thinking? What does this person understand and not understand?

• 2:30:31Zvi Mowshowitz?

  What are this person's incentives? What can I say in response to that? That will cause, you know, their update to be the update that I want. Because one of the things that I increasingly discover is that by thinking about AI, you think about humans. Because we are neural networks too. And we also gather in data and use our compute to try and update our connections to have the best output possible and achieve our goals. And a lot of it rings very true, although obviously also a lot of it's very different, but it can help you really think about yourself and others and understand, right, how to use the analogies tools to get a better model and to get better outcomes. And so when you're dealing with a child,

• 2:31:16Zvi Mowshowitz?

  you have to think very carefully on multiple levels about, okay, what's the important salient things here? What do I have to do? How would he respond to various different things I might say? What would that lead to? What would that, what vibe would that cause in the conversation? Same with Alan, associate with vibes, people associate with vibes. You know, what might this cause him to be curious about, not curious about, if I like that? Right, like what are the, think five moves ahead, think 10 moves ahead. You know, try to figure these things out and you are on a limited compute budget too. You have to do all of this instantaneously. You have to do all this continuously. And you're doing this life at the same time. I like to bet you're fat. So,

• 2:31:59Zvi Mowshowitz?

  you know, when you're dealing with an AI, you have some more really because you have so many more individual data points you can afford to be less precise in some sense. But if you're trying to train something that's going to be much smarter, more capable, then I really do think you have to be very, very precise and careful. Because otherwise, good heart's law is going to bite you. Right, it's going to optimize to the actual procedure that's being followed, not to the thing that you're trying to mimic. And that difference is going to be very, very bad for you.

• 2:32:30Zvi Mowshowitz?

  Right, partly because of manipulation, but probably just because you are going to be wrong about what you want. This means if you develop like AI to help you with the AI in this sense, right, like you have a serious problem. Because it's not, you are going to good heart's law, I mean, it's a big subject of good heart's law already. And so like you get meta good hearted.

• 2:32:53Zvi Mowshowitz?

  And many good hardening is not good, right? Like if you, if you have a telephone effectively, right? It's like, I tried to explain to, like, Spie tried to explain to Nathan, right? Like, what Spie wants and cares about. And then Nathan watches, watches, you know, Alice. And Alice, like, tries a bunch of things and Nathan's like, no, no, no, Spie wouldn't want that because of that. So Spie would want that because of why? And then Alice tries to teach Carol. And like, Carol's not gonna know what the hell's going on.

• 2:33:25Zvi Mowshowitz?

  Right? Like, Carol's gonna have no idea. Like, this is not, it's not gonna preserve. Like, it's not gonna be, like, distilled carefully and preserved. So I don't know that there's that much hope in doing this, but it does provide the ability to iterate on various techniques and try things and test things very quickly. You can learn a lot by doing that. Now, I don't know. We don't have a solution. So it's, it's easy to pitch in a hole and like shoot down anything anyone proposes. It's always, that will never work.

• 2:33:57Zvi Mowshowitz?

  But what will, right? The answer is I don't know.

• 2:34:01Nathan Labenz?

  Well, that's probably, uh, you know, an appropriately sobering point to, uh, wrap up for today. I think, you know, for long form, but still only 1%, uh, you know, as long as what it took you to read to create it. Your blog and the regular updates there that just kind of run down all of the news and give, I think, some of the clearest, you know, again, most concise analysis is one of the go-to places on the internet for keeping up with everything that's going on with AI. And I appreciate all the hard work that goes into it. Last question, I think, for me today is just, what other things do you recommend that people look at?

• 2:34:41Nathan Labenz?

  You know, that could be other safety approaches that we haven't talked about or particular individuals or groups that you think, you know, are kind of deserve more attention, respect, resources, support of any kind? Uh, and then what about just other sources that you think are particularly enlightening? You know, if somebody wants to not just read you as a consumer, but kind of get toward your level? What are the places that you would recommend that they, uh, go that you find value in?

• 2:35:08Zvi Mowshowitz?

  Yeah, so as always, you have to think carefully about what you care about. And what makes sense for you, don't just copy, right, what someone else is doing. But I would say, you know, I cultivated my set of sources, you know, because that's the set of sources that's helpful to me.

• 2:35:29Zvi Mowshowitz?

  But if you want to get involved in orienting around these problems more seriously. So first of all, you can ruthlessly steal from them, right? Like you can literally just follow my rationalist list, you can follow my AI list. You can look at my follow, my people I follow, and you can follow them if they seem to be AI oriented. There's also a bunch of people who are not AI oriented there because I have other things I follow as well. But again, like, if I was starting to orient, I would look at the people who are in AI that you think are processing information well, or offering good ideas. Look at what their information sources are and put or make this openly accessible.

• 2:36:09Zvi Mowshowitz?

  And then I would chicken choose based on my own reading, which of these are helpful to me, which of these are not helpful to me. Similarly, I would look at your AI's blogs, right? So like if I'm, if you're reading my weekly blog, whenever you see something like, you can decide whether or not that source is something you want to systematically follow. Same way I do when I see a new source, or whether it's a one-off. And, you know, you build up a series of assets that way. So the next question you have to ask yourself is how technical do I want to be? Right? So like, there's several different types of AI developers you might want to follow, right? There's what I call mundane utility, right?

• 2:36:51Zvi Mowshowitz?

  The standard capabilities, procedures of like, look, AI can do this. And for that, it's a very different, almost often not very overlapping set of people you would follow. And there are any number of ways you can get that, like any, any level of specification. Then you have the sort of discourse following, people who are engaged in making arguments of each other. And

• 2:37:14Zvi Mowshowitz?

  Think carefully about how much of this you actually need. Would be my actual advice, like I cover it, right? But if you aren't covering it, like,

• 2:37:25Zvi Mowshowitz?

  you should only follow that to the extent that either you want to like evaluate the arguments, or you want to make the arguments and get involved in the arguments. And try shape the discourse, and that might or might not be why you want to get involved, right? If you want to work on technical stuff, you probably don't want to devote too much effort to the discourse and vice versa. So you probably want to choose an area of specialization to a large extent. Like I've chosen to specialize somewhat in discourse because that just sort of, someone had to in this situation. You know, a few weeks of that concentration is probably enlightening, but after a while it defeats. So you've got your AI debate, which is the nominal topic of today, and it's debate and discourse.

• 2:38:07Zvi Mowshowitz?

  And then you've got the AI actual work on the alignment problem, right? And then you've got like the policy questions which are associated strongly with the debate. So the question is, you should decide where you want to focus your time and effort and to what extent. Like you might want to cover all of it, but you, these are just, you just want to focus on one of these two things and not the other. And, you know, we definitely need more alignment research. We need more people who are trying to solve the actual problem. And so there's another, the other reason you might want to be paying attention is, you know, you might want to contribute money, either in investment or contributions. Or you might want to otherwise know like which things deserve support or deserve local support,

• 2:38:49Zvi Mowshowitz?

  including like anything from alignment effort to regulatory effort to an academic paper or a place of creating them.

• 2:38:58Zvi Mowshowitz?

  There's a wide variety of efforts now to advance AI safety in this sense. And, you know, there are some AI organizations, some similar areas where you can find a number of projects. You can also often find links to these things through the arm and form, people's Twitter posts and stuff like that. And there are formal places like Yandex Brands where you can try to get involved more explicitly directly if that's what you're interested in. But again, I would say, don't try to jump into that. Don't try to just say, okay, I'm gonna just write a check tomorrow because, you know, just it's in the right area. This is a lot more of a complex situation. There is no equivalent of, you know, doctor's orders or some other like generic good people you can definitely give money to.

• 2:39:42Zvi Mowshowitz?

  And maybe it's not optimal, but you have to worry about what they're doing. So you need to orient first, unfortunately, if you're doing any of that. And then see what probably gets you excited and see, see, understand what you're doing. And then proceed from there. I would say, you know, if there is one limiting factor right now, more than any, it is people willing to work on the problem directly. The technical problem itself. I said, that isn't sort of supply. There are only a few hundred aligned researchers on the planet. This is order to not get you wrong.

• 2:40:19Zvi Mowshowitz?

  And there are a number of people who are eager to help you pursue that goal. If you are serious about it and like can demonstrate skill. And you are thinking well about it, but thinking well about it is hard. So, you know, the key is then to orient carefully. I wish I could be more specific. I wish I could be more useful.

• 2:40:45Zvi Mowshowitz?

  And I am so trying to figure these answers out despite spending most of my time on these types of problems. But it is a really hard problem. I would say that, you know, in terms of, you know, what I'm doing. There are quite a lot of people who are attempting to like write in real time about AI. And I would discourage trying to be one of those people, especially if you're not already pretty enmeshed in the situation. Just because.

• 2:41:16Zvi Mowshowitz?

  It's kind of a rock star thing where there's a lot of people who want to be that person and where you're competing for attention with others. And so, you know, write to help yourself understand. Like, don't not write anything. I think writing is great. But don't try to write to help you or others keep up with the pace of developments. Write more in terms of ranting about more fundamental, like slower moving things. Do something, right? And then maybe post it somewhere, maybe don't. Zvi Mowshowitz, thank you for being part of the cognitive revolution. Thank you so much. Yes. In some ways, the cognitive revolution is exactly what I'm worried

• 2:41:58Zvi Mowshowitz?

  about. But in other ways, that's what we need. So, thanks for having me.