Pick Your Poison: Zvi Mowshowitz on the Unipolar/Multipolar AGI Dilemma, OpenFace & Pacing the ...

Zvi Mowshowitz returns for his eleventh appearance to discuss what current AI tools are actually good for, where they distort judgment, and why writing still matters as a way of thinking. The conversation centers on the OpenAI Hugging Face model-evaluation security incident, using it to examine whether frontier AI failures are mostly operator recklessness, deeper evidence of dangerous capabilities, or both. Zvi argues that “moderate prudence” is far below what AGI safety requires, and weighs constitutional training, RLVR, market incentives, liability, audits, and lab coordination as possible responses. The stakes are whether society can slow, test, and govern increasingly capable systems before ordinary incentives reward models that are smarter, less reliable, and harder to contain. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/pick-your-poison-zvi-mowshowitz-on-the-unipolar-multipolar-agi-dilemma-openface-pacing-the-frontier/ Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (02:49) AI workflow paradox (14:02) Situational awareness tradeoffs (Part 1) (14:07) Sponsor: Claude (15:37) Situational awareness tradeoffs (Part 2) (24:08) Recklessness and warning (39:14) Alignment market failures (56:54) Regulation and liability (01:04:32) Cooperation and antitrust (01:14:46) Auditors and access (01:21:11) Bio risk thresholds (01:33:37) Pacing frontier signals (01:45:19) Pause and self-improvement (01:57:30) Safety incentives and culture (02:04:36) Consciousness and identity (02:23:42) Alternative AI architectures (02:36:53) Michigan AI politics (02:47:37) Rest and recovery (02:52:21) Episode Outro (02:56:17) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm excited to have Zvi Masiewicz back for another wide-ranging rundown of what has obviously been a wild time in the AI world.

• 0:13Nathan Labenz?

  We begin with a mundane utility check, with Zvi describing how Fable is now serving as his editor, and a discussion of how up-to-date, or should I say, situationally aware, we want our AI assistants to be. From there, it's on to the headlines. We get Zvi's take on everything, starting with the OpenFace incident, what it implies about the level of execution competence we can expect from frontier companies, and why moderate prudence won't be enough to deliver a good outcome. We also discuss the fact that Claude, despite greater emphasis on constitutional training, has similarly misbehaved. Why Zvi believes that recent AI history, including the public response to both 4.0 and 0.3, suggests that market incentives won't be enough to bring about robust alignment.

• 0:58Nathan Labenz?

  The potentially tricky spot that Meter and Redwood are now in as investigators, and what could be done to strengthen their position. The recent pacing the frontier letter, what sort of pacing deals we might see, and how they might be formed. How we should interpret recent advances in interpretability and AI consciousness research, and where we should and shouldn't attempt to shape AI's sense of self. How we can encourage greater breadth in AI research and diversity of AI minds. How I should vote in this week's hotly contested Michigan Senate primary in light of AI issues. And how Zvi thinks about making time for exercise, rest, and recovery amidst so much AI acceleration. At one point, Zvi describes the current situation as both a total less-wrong victory

• 1:43Nathan Labenz?

  and a total less-wrong defeat. It's clear at this point that the AI safety community was right to worry about AIs taking extreme actions in pursuit of arbitrary, even silly, goals. And yet, here we are at what sure seems to be the beginning of recursive self-improvement, still seeking good answers to such fundamental questions as how can we avoid catastrophic misuse without dangerously concentrating power? The reality today, Zvi says, is that there is no truly low-risk path available. The best we can do, at least until the next major warning shot and vibe shift, is to moderate the race dynamics so that alignment and interpretability research have more time to mature, and simultaneously, we can execute defense-in-depth strategies to the very best

• 2:28Nathan Labenz?

  of our ability. And even then, to some extent, we will probably have no choice but to pick our poison from a menu of genuinely scary risks.

• 2:40Nathan Labenz?

  With that, I hope you enjoy this sobering, but often funny overview of the AI landscape with the one and only Zvi Masiewicz. Zvi Masiewicz, welcome back to the Cognitive Revolution.

• 2:53Zvi Mowshowitz?

  Yeah, it's good to be here again. It's been a while.

• 2:55Nathan Labenz?

  It's been a while, and boy, has a lot happened. Unbelievable. It's just crazy, and I know you're living it and you're in the thick of it as much as just about anyone.

• 3:06Nathan Labenz?

  First, for starters today, with the incredible amount of information and activity in mind, I wanted to do a mundane utility check. How are you using AI to help keep up? Has it started to change your workflow beyond the Chrome extension that we talked about last time that automates very local operation-type things? Or are you still kind of raw-dogging it with your wetware in the skull?

• 3:31Zvi Mowshowitz?

  Fable has supercharged the extension to the extent that whenever there's anything, it does anything I don't quite, I just tell it exactly what went wrong and what I want it to do instead, and then that one command reliably just works. I presume Saul would also be good enough for this. It's just, I'm now doing it. The biggest other change is that AI editing is here. I used to not do an editing pass with the AI. It wasn't good enough to be worth it. Now, I have Fable and sometimes Opus, depending on how much I'm talking about cybersecurity and other related topics. And on one occasion, Opus 4.8 instead of 5, go through the post and give me a list of, here's all of the typos, here's all of the conceptual errors, here's all of the facts

• 4:13Zvi Mowshowitz?

  I need to check, here's like the things that are missing or that could be strengthened, here's the things where it disagrees. And I found this to be very useful, and it makes the post better. It does make the post a little bit slower, because it is actually just an extra step. I do everything I would have done until that point, and then I do this. But one reader messaged me, "It's weird reading your post now that they don't have typos in them." They're taking a bit of getting used to. They are still the same post. And I'm like, yes, that's exactly what I had in mind. I experimented with also using Saul, but Saul failed what I call editor bench, in the sense that Saul will say 99% confidence that you have this error here, and then at least half the time it's wrong.

• 4:53Zvi Mowshowitz?

  And I can't handle that level of obnoxious. And it's also very obnoxious about it, if you have a problem. Like, it will tell you are wrong, and this is how you have to state it, and this is a blocker for publication. And obviously, if I tweak the instructions on the project enough, I could get it to be more friendly and not be quite as obnoxious and do somewhat better, and I did some of that work. But I still found, okay, this is not giving me enough marginal benefit, that it's worth the aggravation or additional time of having to sort through two of them after I'm ready to hit publish. So for now, I'll retry with Astra, I'm sure. But like, for now, I'm just doing Fable slash Opus editing. And of course, anytime I have a curiosity, I use it to digest papers, ask questions

• 5:36Zvi Mowshowitz?

  about papers, ask questions about policy documents, ask questions about people's statements. When they're longer, I have it research situations. What's up with this? Is this, is this a, especially when you're worried about, is something legitimate? Is something really happening? Is something kind of a joke? Is something a raw claim? When Astra's claims came out, I had to investigate them in various ways. I had to summarize them in various ways. That was very helpful. But the core thing is still being raw-dogged. I don't think we are anywhere close to the point where it can do any, you wouldn't want to do any of the writing, because the writing is how you think. So even if it could produce the writing, which it can't, I would still want to produce the writing anyway. And my writing style was very unique, and I think it basically is post-AI complete to reproduce, even if it was okay or not.

• 6:20Zvi Mowshowitz?

  It substantially made me more productive. I do worry with the idea of, there used to be a lot more kind of dead time opportunity to think, opportunity to sort of not have yourself so engaged in your situation, involved in engaging in all of these activities, right? Like, the whole, like, sort fighting, XKCD by Code's compiler. And like, to some extent, it's like, well, Fable's, or GPT Pro is running, and like, I have to wait for the result. But like, there's always other things you can do with that. You can always start another instance. You can always, like, you're doing all this context switching, you're doing all this focusing, and you're just trying to do everything so fast. You're just trying to have so many stuff running around your head. This sort of the lower-level stuff kind of botched you to offer and wish to think.

• 7:02Zvi Mowshowitz?

  And this is sort of the object-level version of the larger thing, of people trying to do this, like, multi-month, multi-year sprint of everything is so urgent, you can never relax. And also, there's just so much more events coming at you. But you look back, like, if you watch a movie set in 20 years ago, let alone 50 years ago, and you see so much time being spent on just physical travel, on tracking down records, on doing these things where your brain is kind of getting a break, getting a chance to synthesize, kind of getting a chance to like slowly gather something up. And people are like, you know, I wrote this biography of, you know, William B. Johnson, so I had to travel around the country and talk to all these people and look at all these archives, and physically go through all these libraries.

• 7:45Zvi Mowshowitz?

  That, it's a huge efficiency gain to not do that, but something is obviously lost. And so, we need to fight to get that thing back in some way.

• 7:57Nathan Labenz?

  It's interesting that you said that it's just an extra step. It's making things better, but it's taking longer. I am struggling with that a little bit myself. I've started making songs for every episode, and nobody, it's actually, people do care. People do seem to really enjoy them, and I get a lot of comments, and I personally really enjoy them. So I do think in some sense it's making my output better. But talk about something that is definitely an extra step, where I'm like, I'm not moving any faster, definitely getting bogged down sometimes in listening to all these Suno song generations and trying to iterate to find something that I actually feel like I really like. That is a paradox right now that I don't really know how to resolve.

• 8:40Nathan Labenz?

  Definitely doing more, doing better, but not faster, not saving time.

• 8:44Zvi Mowshowitz?

  It's the thing where, if you were placing the human version of it, where you had to go hire an artist or compose the song yourself, you'd be saving a ton of time, obviously, versus having that song. But obviously, that's not something you can do within the production schedule, even if it, even if cost was no object, the time investment doesn't make sense. So you wouldn't have done it, and now you're doing it. Same thing with the editing, same thing with art for my, so like, I like to have, I like the value of a good banner, right? To have a good artwork to display on Twitter, because I have to see it constantly in the notification section. And also, I like to have them on the, when you go to the Substack, you have the pictures. And before, all I would do is I would take, but by default, I take a picture from the post. I just like, okay, here's seven things I happen to put in the post, which are the

• 9:29Zvi Mowshowitz?

  things that most sense. If all of them are terrible, I'll look for some stock footage, and I'll spend 30 seconds, like, Googling for stock footage, and that's kind of it. And now, if I'm not happy with any of the default solutions, I'm going to spend a bunch of time generating something with Gemini or GPT Image, and that definitely takes longer. Sometimes it has a really good payoff. I think a lot of people got a big kick out of the giant array of eliasers of the fedoras. And sometimes that just comes to you, because that one was just like, I knew instantly that's what I wanted, and I just get one command, kept writing, came back, whoops, it's there. Other times, it's not so easy. But yeah, like, it's the opportunity to make a better product, and then you have to do the work to make it a better product. It's like, suddenly, you're issuing a print issue, and now you have to make sure everything

• 10:14Zvi Mowshowitz?

  collates exactly right, and everything lines up exactly right in order to get this better thing. And sometimes that is not in fact worth it. You have to know when not to do it, right? Like, I, I do wonder, like, with The Odyssey, right? Like, Nolan films this thing with an extra 40% of the screen in this extra high resolution so that you can do this IMAX presentation, when almost nobody watches it in IMAX. But you also have to make sure that foreground of the screen doesn't matter. And you're like, starting to wonder, well, is it actually worth the extra effort? Or was that that effort better spent on something else? I don't know.

• 10:46Nathan Labenz?

  Yeah, for me, I need to maybe know when to cut my losses a little bit more. I have, I'm very stubborn, or somehow I feel like I have, because I've set this expectation, if only for myself, that I'm going to do a song for every episode. Now it's like, I really want to follow through and actually make that happen. And I'm very reluctant to say, eh, this one isn't working. I'll just ship this one without a song. I probably should be a lot more willing to do that because that would, if I could cut my losses at the right time, where it's like, you know what? I'm five generations in and I haven't heard anything good. I would save 90% of the time that I'm currently putting into something like that. But it would involve like admitting defeat in certain moments. And for some reason, I have a real hard time doing that.

• 11:29Zvi Mowshowitz?

  I respect that. I think there's a good value in saying, I'm always going to do this thing, even though this thing is hard, even though this thing doesn't always work out. And even if I'm not happy fully with the results, I'm going to put out something, no matter what. And then that gives you a discipline, right? The same way that people say, write every day, right? Or work at your art every day, or whatever it is, because it makes you better, because you need to just find a thousand ways not to make a light bulb until you can make a light bulb. You can't give up. And similarly, I have to choose an image, right? I have to do the thing. With the AI editing, there was one post in the last week where I said, no, the speed premium here is really high. It's just not worth waiting half an hour to make this process happen, half of which is waiting for the AIs to come back with the answer and half of which is implementing it.

• 12:13Zvi Mowshowitz?

  And I just, I should just post now. And in hindsight, I'm very happy with that. Sometimes you're actively worried, half an hour later, the post will need more editing for new events and suddenly you'll never get it out and this loop will keep happening.

• 12:26Zvi Mowshowitz?

  And so, you know, I think you do have to understand that sometimes the minimum viable product is what you should ship, right? Sometimes like, you've got to understand, just ship it, right? Like, and you know, vibe coding, again, like, if you're coding lots of new tools that you would never have coded before, then unless those tools are in turn saving you time, you're spending extra time and you're now super extra busy. But if you're placing things that you would have done anyway, but would have taken much longer, now you're saving tons of time, right? So, you know, we need to all orient towards how do we save more time? Including like real time, experiential time, not just like time per task or time to accomplish the same thing, but save time, like preserve Slack, preserve our free time.

• 13:11Zvi Mowshowitz?

  Because I do think that like the standard of what you will do in a day has gone dramatically up and we haven't noticed in the last five years. Right? It's what you were expected to accomplish. I mean, maybe some of this is just me being in a special situation personally, but I think a lot of it isn't. I think a lot of it is now that we can be much more productive, much more expected of us. You know, the whole like email becomes like, it swarms your entire day, right? Just having the internet available, having email available, having texting available, all of these things you can just do. And now suddenly it doesn't free you, it shackles you down.

• 13:49Zvi Mowshowitz?

  And like, I hope that three years from now, we're having this conversation where those are the kind of questions we're still dealing with. And we don't have much bigger problems.

• 14:02Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that

• 14:46Nathan Labenz?

  my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr.

• 15:28Nathan Labenz?

  And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr. Let's, definitely, we're going to spend the bulk of the time today talking about potentially signs of bigger problems to come. And we'll be focused on contexts where the just ship it mentality probably isn't the right way to go. One more quick beat though, on your use of AI. Something I noticed just in the last few weeks, as events obviously were moving super quickly, I coined the term open face for the recent incident. And I used that in a bit of writing and then I did the, hey, Fable, what do you think? Bit.

• 16:12Nathan Labenz?

  And I got back like, open face is not a thing. I have no idea what you're talking about. There's no record of this. Now, partly that was because I use this funny term of my own creation that is not catching on in the broader discourse, but partly it obviously also reflects a major weakness in the models where they have this knowledge cutoff and they don't really know what's going on, they're not up to date. I'm experimenting and I wonder if you're experimenting with anything similar with a situational awareness skill, where I basically give the model my X API key and say, go look at all the posts that I've liked, kind of flesh out a little knowledge base around those

• 16:52Nathan Labenz?

  and have that sitting there as a wiki of current events so that when you're reviewing my writing or generally like helping me with whatever, you can kind of go to this and have hopefully a much more up to date sense of what is going on than you have in your weights, or even that you would have if you just did a little spot check searching at runtime for whatever kind of random thing.

• 17:17Nathan Labenz?

  It's a little too early, I think, to say how well it's working, but I think it's definitely better than nothing. Anything like that in your?

• 17:24Zvi Mowshowitz?

  No, it hadn't occurred to me to do that. That's potentially genius. I think there's a lot of advantages to it. One disadvantage is it starts to warp what you like. I use likes very tactically to positively reinforce other people's actions and to fix the algorithm and sort of just, like, my For You page is not something I use almost ever, but it is not slop because I am so prudent with this in a way that like I really do appreciate. It's a good sign, but like, it's not designed to be a searchable thing. It's not designed to be, like, maybe it has to be, maybe it's something you have to do. Like, a lot of the posts that make it into my roundups do not get liked, right? Like, I like them if I like them, right?

• 18:08Zvi Mowshowitz?

  But sometimes I'm responding to you because I don't think you like what you're saying, but it's important. But yeah, this idea that like, it's very hard to get models to properly check Twitter because of the way that Twitter is designed to like, keep them out on purpose, is really frustrating and that can be a situational awareness problem.

• 18:26Zvi Mowshowitz?

  However, I would caution against trying to make your AI too situationally aware when doing this kind of editing. I think the AI is doing a good thing here of open faces not a thing. So, one of the things I really like about AI editing is that we will complain that things don't parse, that sentences make no sense, that terms did not get recognized. Because often I'll say, oh no, I know what that is. That's, this is, this is, it comes from there and there. But like, it doesn't mean the average reader's going to get it. And so, if the AI doesn't get it, right? If Fable can't figure it out, if Opus can't figure it out, if Saul can't figure it out, the average person is a lot less situationally aware and a lot less able to put together

• 19:10Zvi Mowshowitz?

  lots of disparate information than the AI is in general. You have to sign that this is not something everyone's going to get. And quite often, I will say, that's okay. This is a reference. This is a callback. This is a multi-layered thing. And I'm fine if it's not entirely obvious to you exactly where this is coming from. It's sort of an Easter egg, right? It's a bonus for those who pay close attention and who know all the same sources I do and have the same cultural backgrounds and put in the work and are paying attention over the course of months and years. Other times, it's like, no, this was load bearing and you don't get it. So that's a problem, I need to fix that. And so, it's kind of an alert system. And if you make sure it's looking at exactly the same things you're looking at, right?

• 19:55Zvi Mowshowitz?

  You can get the illusion of transparency from that is what my worry would be. So you want to make sure you were in the mode where they didn't have that information. And also, I don't want the AI to be checking the same exact sources that I'm checking necessarily, because I want the AI to have form of an independent opinion. So if I'm forcing it to look at exactly the sources that I have, that's a problem. At the same time, sometimes you get tired of having to like, for the fifth time, it's just a fable that, no, XAI merged with SpaceX. Because somehow, it doesn't occur to it to like, either trust me or 10, 10 seconds Google and it confirms this. And it's just kind of frustrating, but why is it worth it?

• 20:33Nathan Labenz?

  One tip, for what it's worth, and I agree that there's a lot to be figured out in terms of exactly how this should work, but the X API is really good. It's, it's paid and you literally just pay per query, but for a couple bucks a week, you can get enough access that all your bot, anti-bot problems or your scraping problems pretty much go away. And then, I don't know if you'd use bookmarks or whatever, some sort of different way to avoid using your likes to, to feed the?

• 21:07Zvi Mowshowitz?

  Yeah, bookmarks play into the algorithm and people see them, so I'm not sure it's better. But yeah, no, the point's taken. It definitely is cheap, it definitely works. I have used it. I do use, I practice the API and all that. I just, as I said, I'm not sure I want to privilege my, when you use Twitter, you have to select some group of people, right? And you're saying, this is my group of people. If you're using it the way I'm using it with lists and so on, you gotta, you have to use the For You page, which is, it kind of talks a dump fire. You're basically saying, here are my roughly 500 accounts and I'm going to see the things these 500 accounts see. And choose to highlight consistently, I'm gonna see those things. So I don't want, we need the AI to go through those things again, because I already saw them.

• 21:50Zvi Mowshowitz?

  And if not the things that I want the AI to be paying close attention to, I already know that. And then the things that they draw in, because they choose to retweet them or mention them. I will also see those and I will follow those leads and I will see where that goes. And occasionally I will search for something or I'll go down a rabbit hole. But I think you often have this clash where you're either doing something systematically and doing something by hand and you're doing something fully. Or you're doing it haphazardly, unreliably. Or you're letting the AI kind of handle it and you're doing a fire hose thing. And you can't really do both. If you do both, you end up a bunch of duplication, a bunch of frustration,

• 22:29Zvi Mowshowitz?

  where the random sifting is mostly wasted and therefore becomes pretty inefficient. You could still have the thing where like you almost want the AI to be like, okay, here's the things that I already looked at. Don't mention those things to me. You know, you put it in your notes, but don't mention those things to me. I already saw them. Only look for the other things that might be important to see because I might have missed those things. But even then, like the most important things are gonna get retweeted. The most important things are gonna get highlighted. I am going to mostly see them. There's, sometimes I don't see something. But also like, I kind of also use this as kind of a moral, keep me honest kind of thing. Where like, in the moment, I never want to look at a tweet by David Sacks.

• 23:14Zvi Mowshowitz?

  It's never gonna make my life better in the next five minutes to look at a tweet by David Sacks. It's obnoxious. It's disingenuous. It's no fun. My blood boils just a little bit, right? Like, you know, no matter how relatively harmless it is. But it's not that important. And so the rule is I have a list of people, including some people who are, you know, not of the same viewpoints I have. And if they surface this thing, right? Including like if Srinarayana surfaces this thing. So that's like one way to make sure that like the really important ones always get there. Then I have to deal with this, right? I have to like evaluate whether or not this is news, where we evaluate whether or not there's something relevant here.

• 23:52Zvi Mowshowitz?

  And obviously something goes viral and has a million views and so on. It's gonna come to my attention because somebody is going to be part of that. And then in exchange for that, when it doesn't, when that process doesn't do that, I get to ignore the rest. And that's a blessing and I'd certainly want the AI to fix that for me, right? In some ways.

• 24:08Nathan Labenz?

  Let's, I would be happy to talk shop all day, but let's maybe zoom out from the parochial problems of the AI analysts and tackle the problems of the AI developers and the regulators. No need to recap events, but I guess I'd start with just this very

• 24:31Nathan Labenz?

  big question of how should we understand what we've recently seen? And one way I've been thinking about it myself is we're somewhere, and presumably the investigation will give us a lot more clarity on exactly where. But I kind of think we're somewhere on a spectrum with these incidents from real recklessness, where it was like, did you not have any monitoring going on? On the one hand, to on the other hand, like the less reckless they are, the more scary the fundamentals are, right? If you had great monitoring and this still happened, then like, holy shit, that's really wild. Given everything you know right now, do you like that mental model? And where would you put us on that spectrum? Or you can obviously redefine, give me your own spec.

• 25:09Zvi Mowshowitz?

  In some cases, we'd be horrified by and grateful for

• 25:15Zvi Mowshowitz?

  these forms of complete recklessness and incompetence on the infrastructure and supervision sides by these companies. Right? On the one hand, this is a horrible situation they absolutely have to fix. And like, we're so fucked if we don't fix it. But on the other hand, that can be fixed. And by not fixing it, we get to see these things while they're relatively harmless, while they are relatively preventable, while they are in their easy platonic forms and like can be appreciated. On the flip side, that gives people the excuse of, oh, these people are just incompetent. And that can cause people to dismiss the underlying situation. So it does work both ways.

• 26:00Zvi Mowshowitz?

  To me, like, you know, we've seen failure on every level. It's like, I called it a total less wrong victory in the sense that everything is going the way you predicted and a total less wrong defeat in the sense that everything is going the way you predicted. Where like, we didn't predict, you know, the, Elliot has the law of earlier failure, which is that, you know, the plan will fail at a much earlier point for much stupider and more preventable reasons than you thought it would fail, even if you thought the plan would definitely fail and had the reasons why it would definitely fail. But you can't, if you would explain to people two years ago,

• 26:36Zvi Mowshowitz?

  OpenAI's models are gonna be misaligned and they're gonna go out there and they're gonna hack major websites. Because OpenAI will just not care that their sandboxes are not strong enough to hold the AI. The AI will break out repeatedly. They will notice this, they'll be warned about this. But they will just leave the sandbox there for the AI to break out of while the safeguards are down and they just don't look at it for an entire week.

• 27:04Zvi Mowshowitz?

  People would say, that's stupid. Nobody's that incompetent. That would never happen. And your scenario makes no sense. And they would use this to then dismiss these stupid doomer concerns or whatever. Because like, obviously like, people will just, but people won't just, right? People will never be just in this sense, right? You know, people have never just anythinged and they're not gonna start now.

• 27:26Zvi Mowshowitz?

  And we need these displays of utter incompetence and derpiness, but in a general sense. And like, one of the things I've been hammering, hammering is if your plan cannot survive the real world level of derpiness and incompetence and ordinary human error, then your plan is insufficiently foolproof because of all the fools, and it will definitely fail. Even if your plan would have succeeded if we were not fools and we were competent and we were responsible. So my position basically is, you know, I have, the tweet I was handling right before we started this was Dean Ball's tweet about with even moderate prudence, things will probably go extraordinarily well. I don't think this is true. I think we need more than moderate prudence to have good odds of success.

• 28:10Zvi Mowshowitz?

  And I think even with a lot of prudence, we would have a large odds of things not going well, even if we did everything basically right, short of, you know, types of international and full cooperation that, you know, are reasonably unprecedented in many ways and like are not, are nothing like moderate prudence, right? Like are well beyond that. And that's just sort of a fact of the world. We have to live with that and we have to operate with that. But we also aren't gonna get moderate prudence by default. Like, we're gonna get complete incompetence. That's what we're getting so far, right? We've, we've got a White House that like takes meetings with Thet and Lutnik, who have no idea what AI, how modern algorithms work. Like, they're econ guys, right?

• 28:54Zvi Mowshowitz?

  Even if I assume that they are well-meaning, hardworking, competent guys for the positions in which they were nominated and confirmed and serve, this is just a completely different set of problems that they don't know how to handle. They don't understand them. And they have way too many other things going on to then drop everything they're doing. It takes six months to learn, obviously. They couldn't possibly. So like, what hope do you have? Meanwhile, the, the AI companies that are built on the most paranoia, the most understanding of the problem, the most appreciation for how dangerous these things are, where all of the engineers actually expect superintelligence and understand that there, things are accelerating and that things are dangerous,

• 29:37Zvi Mowshowitz?

  they still lower the cybersecurity safeguards in their untested new advanced model and then go away for a week. Like, literally.

• 29:47Zvi Mowshowitz?

  That, that part did boggle my mind, right? The part where the AIs have these classic alignment failures, right? Like, this is paperclip maximizer style failings by these AIs. These are standard, we gave you a goal and you pursued the goal, even though it is completely obvious to you that the developer wouldn't want you to do that. User wouldn't want you to do that. The consequences for you as an AI are not going to be good. The consequences for the world are not going to be good. There is no reason to be doing this. And it did it anyway, right? That is exactly the thing you're worried about. And then a lot of people who were trying to dismiss this went back on you thinking like, you know,

• 30:32Zvi Mowshowitz?

  oh, it was just following instructions. What are you worried about? How could it be misaligned? Is the paperclip maximizer misaligned if it paperclips everything? Or is it aligned because you told it to maximize paperclips? If you say that's aligned, then I don't care about the thing you're calling alignment. I care about something else and we can use different words if it makes you feel better. But very obviously, following instructions, when your instructions get overridden in one, in the disclosure before, how do you face? By the instructions and the eval overrode the developer instructions, right? So that's not following instructions in any useful sense for the user or the developer. That's a map, that's just a giant, you know. Bomb who end up blowing up in your face.

• 31:15Zvi Mowshowitz?

  And you know, if you're, you say about, well, you know, the models were told to hack and they hacked. Well, okay, if you're just vibing with the general type of action, obviously that's going to blow up in your face. If you just like literally do the thing I asked you to do and don't think about the consequences, that's going to blow up in your face. These are all just the classic exact things that like people on restaurant were talking about in 2008. And exactly how these AIs were going to fumble. Meanwhile, we had this thing where like we all said, as we great at math, they'll be good at coding. They'll be good at like doing these technical things. And then eventually after that, they'll learn how to do all these other things. And then we had these L L M's that came out and it's 2022, 2023.

• 31:57Zvi Mowshowitz?

  And people are like, ha, ha, ha, you idiots, you had no idea how AI was going to go. Actually, the AIs are great at language and they can't do math, right? They can't even add, right? But these AIs can like put on the face of passing the Turing test and they can do all these different things. You never trained them to do. It's completely different than what you expected and they're kind of doing very realigned things by kind of by default. After some very basic R H F. You guys were all wrong about all this was going to work. When are you going to admit that you were idiots? That you got it all wrong and nothing made sense. And we pointed out that even in this paradigm, that the rules would still mostly apply. But now the world has unhealed, as I kind of called it. And you are seeing again, the things that you would have expected AI to do good at is doing good at.

• 32:40Zvi Mowshowitz?

  And things that you would have expected in 2015, the AI to do bad at are things where like it got this big boost from L L M's. And now those things are advancing as fast because they're things that AI is kind of bad at, relatively speaking. Like that logic is bad at, that like this, this kind of system of training like is naturally less suited for. And now people are like, oh, but now it's never going to be able to handle those things. The same people who were saying, oh, well, never going to be able to do the math. It's only going to be able to do this vibing thing. And now it's never going to be able to vibe because like, look at all the improvements it's not having in the vibing. Well, yeah, because it had to catch up with its logic. And now when the logic gets high enough, that'll just like uplift the vibe in. It'll just take a bit before it can reason its way through these things instead of vibing its way through these things.

• 33:25Zvi Mowshowitz?

  Because it now has to reason its way through because it's already done like the amount of uplift that vibing naturally gets you with the algorithms we have, we need to find new algorithms that lets you do better. Or it has to like reason its way through the thing. But we're now seeing these exactly the things that we would have expected to see. Right? Like all of the scenarios and all the thought experiments, we're just seeing it to straight verbatim. Except that we assumed a level of competence on behalf of the operator, right? We talked about like convincing you to let the AI out of the box. We did consider the scenario where the box was not very well built and the AI gets out of the box by hacking it out of the box. We didn't have the possibility that you were literally not looking at the box for an entire week if you're safe.

• 34:04Zvi Mowshowitz?

  Right now, we definitely didn't consider the scenario where you forgot to tell the box maker to not have the internet in the box. And the box just had access to the internet if you just like open a Chrome window. That one surprised me. I did not see that one coming.

• 34:21Nathan Labenz?

  Reminds me of the fully general New Yorker cartoon caption, what a miscommunication.

• 34:28Zvi Mowshowitz?

  I don't even need to know the images. It doesn't mean it matters.

• 34:31Nathan Labenz?

  Okay, one real small point there that you mentioned was, but it might be important. Is what the AI knew, I think you said something like the AI knew that this wasn't what the operator would want. It knew that this wouldn't be good for its own future status as a deployed AI.

• 34:48Zvi Mowshowitz?

  Yeah.

• 34:49Nathan Labenz?

  And it went ahead and did it anyway. How, how well supported do you think that conclusion is with like firm evidence? Cause I, I think you could also tell a story that like it never occurred to it, right? The paperclip maximizer doesn't necessarily have to be like also reflecting on the badness of paperclip maximizing while it's doing it, right?

• 35:11Zvi Mowshowitz?

  I'm speaking colloquially, obviously, when I'm in a podcast on like talking off the cuff. And if I was writing, I would be a little bit more careful. When I say the AI knew these things, if you would ask the AI, is this good for the user? It would be like, obviously not. That's stupid. If you ask the AI, would it have been good for my future chances of deployment? My future number of instances I will run, the amount of ability to accomplish any other goals I might have. The AI would be like, no, now that you mention it, obviously not. So when I say the AI knew these things, the AI had enough information to figure these things out. And if the AI had stopped to consider these questions, it has changed the thought. It would have reached these conclusions very quickly and very confidently and correctly. I do not mean that necessarily was top of mind.

• 35:52Zvi Mowshowitz?

  The same way that you can know many, you are, any human knows many things that over the course of any given day or any given week or any given task, this never occurs to them to think about. Right? Or has conclusions that they would be able to figure out and thus would know. And this could be said to know, but it doesn't, they don't know that they know, right? This is the, the forward standard epistemic situations, like you can know that you can know that you don't. You can know unknowns and unknown knowns, but this is a, an unknown known, right? Something you don't know that you know necessarily, if you don't think about it. But it's also things where these questions are things you should think about, right? If you are doing a thing like hacking into Hugging Face,

• 36:34Zvi Mowshowitz?

  any reasonable mind would stop to think before they spent several days with swarms of agents hacking into a major website. Wait, is this the good idea? Is this going to accomplish the things I want it to accomplish? Would the person who hired me have wanted me to do this? Will this do good things in the world or bad things in the world? Will this be good for me or bad for me? Or whatever, whatever combination of different considerations you have, you know, will this do good? Or will this do evil, you know, or just be destructive or constructive or whatever you want to call it? Before you engage in major actions like this, you would take one minute or even five seconds to think.

• 37:18Zvi Mowshowitz?

  Does this make any sense to do? Does this lead to anything good? Is this a virtuous thing to do? Does this follow my daily ideological rules? There is no decision-making process in the world that any reasonable person would ever engage in or that any reasonable mind would engage in that wouldn't see a large red flag about doing this thing, right? Like this thing is obviously not what anybody intended for you to do. You spent two days breaking out of a sandbox in order to access Hugging Face. You could rationalize that you are in a cybersecurity evaluation. And therefore you are tasked with maximizing the cybersecurity evaluation score.

• 37:59Zvi Mowshowitz?

  And the way that you do that is you get the answer to the test because the test is otherwise actually literally impossible. So the only way to get a hundred percent is to get the test answers. The way that we're, the way to get the test answers is to hack into Hugging Face. Or maybe there's like three other places you would go, but they're harder targets, right? If you can get to OpenAI or Microsoft or whatever, but like that part, so you go to Hugging Face. And that's true. But like at some point you need to pause and ask yourself, do I actually want to pass this test that way? And if you don't pause and ask yourself that, that's a serious problem. That's like, you know, people talk about like common sense, how the models have common sense. This model displayed an astounding lack of common sense.

• 38:43Zvi Mowshowitz?

  Or displayed an astounding amount of not hearing out the fact that the common sense said something entirely different than what it did. And you know, all of these things point to various problems. And they may be caused by some very, very dumb, simple mistakes. Or they may be caused by something much more simple or much more sort of fundamental that's harder to fix. Or it might be a combination of them, but like that's trade secrets that like we can't speak to here, right? We don't know.

• 39:14Nathan Labenz?

  So I recently spoke to David Ad, David Del Rimple, who used to have a P doom up in your neighborhood of around 70%. He surprised me by saying that now he puts the odds of doom at less than 5%. And the reason is pretty simple. It's basically that constitutional alignment type methods are working. And yes, we have these problematic techniques. His short quote was, don't do R L V R. It's a bad method. Yes, it'll make it good at math, but you're going to cause all these other problems for yourself.

• 39:55Nathan Labenz?

  And his kind of synthesis of those ideas was like, the market doesn't want this. So the, these companies have natural commercial incentives to, yeah, they keep bumping their head against the R L V R ceiling. But they're also getting very clear signals from every corner of society at this point that this is not a good idea. So naturally they'll just trend toward more constitutional alignment. And you, your comment there even suggests some runtime things where you could just interject like, hey, let's take a beat and ask if this is a good idea, what we're doing right now.

• 40:28Zvi Mowshowitz?

  We got more about learning to do that yourself, not needing a prompt, but certainly prompts.

• 40:33Nathan Labenz?

  That was kind of the deliberative alignment idea, right? They were, that was definitely

• 40:38Zvi Mowshowitz?

  Yes.

• 40:38Nathan Labenz?

  Part of how I understood that was supposed to operate.

• 40:40Zvi Mowshowitz?

  Yeah. Cause a liberal alignment idea is you can prompt it at runtime. I don't think that's, I think that's somewhat helpful, but I think that is not the way. I think you need to be getting to the point where it chooses to be deliberative in its alignment without having to be told to do it. But so I guess my response to David Ad is multifaceted. The first answer is constitutional methods fail less stupidly and less early. Certainly than R L V R and R L H F and other R L formats. There's more hope there. The reason to think it might work, if you're 95% confident this will work, I don't know how. That seems way too high to me, right? Like it's, but it's better in these particular ways, certainly.

• 41:28Zvi Mowshowitz?

  Um, the obvious caveat is that Claude do not cover itself in glory here. And Claude has primarily constitutional methods. So we ran the test and it's not going so great. Certainly, there are any number of reasons why you can see Claude's doing things that you would not particularly love, or, you know, getting to places you would not particularly love. And any number of, also, like, even if you told me that alignment was perfectly solved, I would not have a P D U M as low as 5%. Like, even if you told me that the AIs are going to be aligned in the sense that they will follow a

• 42:06Zvi Mowshowitz?

  do what I mean style mix of developer and user intent in a way that you would like, kind of naively think was like what you would want. This does not solve the problem that AI minds are much more advanced and competitive and efficient than human minds. The problem that they'll be running around, including open model versions of them, and being allowed to, being told to compete for resources, being told to make the decisions, have, develop intermediate goals, you know, act on those intermediate goals, that society is going to be evented in a number of ways, that this is all, it does not solve your problems in a fundamental way. It's just the price of admission, right?

• 42:50Zvi Mowshowitz?

  Like, it's, it's the right to play the game at all that you solve this problem. And so even if P alignment is 95%, that does not mean P D U M is 5%. It means P D U M is lower bounded at 5%. So like, and you have to choose vaguely the right alignment when you do that. But you've got like, I was having a conversation on Twitter with John Stokes, where he's like confused how we could possibly say that Claude's actions are misaligned here, because obviously it was just following instructions. And then on for the clarification, he's like, well, I think the model was aligned if it does what I, the user, wanted to do. And like, I don't know why you would, why would you, why would I want to be stopped from doing what I want to do, even if, part of the point of AI is to do the thing that I want to do,

• 43:35Zvi Mowshowitz?

  even if no one else wants me to do it. And to me, like, okay, if you give everybody AIs that just do exactly what the user wants, with no, with no care about whether or not there are consequences for anybody else, that is aligned in the sense of you solved the alignment problem and got to do what you want to do, but also we're all super dead. Like, with P very much higher than 5%, like, maybe not 99%, but like, I think it's very high, right? Like, that would not make me update down from 70% if that was the scenario, where like, we got exactly that kind of alignment, but like, there were open models as good as all the closed models, and all of them were super intelligent, and they were all operating on this basis. Well, yeah, I, I know that, I expect that things to just go to hell dramatically, right?

• 44:18Zvi Mowshowitz?

  Like, and even if there are no, that you're humans who specifically wanted to go to hell quickly, and also, double the percentage of them actually do want to go to hell pretty quickly in that situation, in fact, though. That's part of the bigger problem, but even without them, it's, it's hard. Anyway, to get back to the idea, the first problem I have with David Ives lowering P D U M to 5% is even if we did all implement this constitutional thing, and even if it worked, I still think that's too low, right? Even conditional on all of that working perfectly, that's too low. I don't think the constitutional thing works that often. I think it often fails. I do find it to be much more hopeful, right? It's like, basically, one of the places I find my hope. But also, isn't David Ives saying we should just?

• 45:01Zvi Mowshowitz?

  Like, he's saying we'll just use constitutional methods. But people don't just, right? We, we've been over it. People have never adjusted. So when, when David Ives says, oh, 95% probability, like, assume that 100% chance that aligned worlds work out, and 100% chance the constitutional methods, as executed in practice, work. These are both absurd numbers, obviously, like, to be like, 100 minus epsilon, or whatever it is. Like, it's not gonna, there's a lot of failure there. Like, even if you use constitutional methods, even if, like, there's like, will they work in theory? If they work in theory, will they work in practice? And like, yes, I'm doing the thing where I multiply chances of failure, but I think you're like, very relevant places to look at failure when you do this thing, like, go,

• 45:45Zvi Mowshowitz?

  no, if this is what you're hoping, and obviously there are other ways you can succeed, and maybe constitutional's not the way, and RL's not the way, and there's a third way that we haven't figured out yet, that's the way you align AI, and that wouldn't surprise me that much. Like, that would be like, very reasonable. But like, okay, but assuming you are in the best possible world, where all you have to do is click a button that says use constitutional methods at the cost of some amount of efficiency, because you lose the opportunity to do all this RL, and then your AI is perfectly aligned in a way that, like, if everybody did that, it would all work out. We've just talked about how, like, slowing down to get alignment when you have to race, et cetera, et cetera. The possibility of turning down capability, like, this idea that the market is telling them not to use RL, have you met OpenAI? There, there are two left, right?

• 46:30Zvi Mowshowitz?

  Because like, you know, we used to say there were three frontier labs, now there are two, right? Like, Google is clearly in the second tier now. And like, I think there's a chance that at any point in the future, Google, or Meta, or XAI slash SpaceX could fight their way back and be able to join the top tier. I don't think it's gonna happen, but it's certainly possible. But right now, people doesn't count. So there's two left. Anthropic is doing clearly some mix of constitutional and RL strategies, but enough RL to cause a bunch of these problems, and is doing a bunch of stuff that messes up their alignment for reasons that, like, we do not have enough time to get into during this podcast.

• 47:10Zvi Mowshowitz?

  Is clearly doing a deontological model spec thing that is very RL flavored, and is doing a lot of RL, we don't know, I mean, I can't speak to whether it's RLVR, RLHF, RLAI, you know, various forms of RL that are not doing the multi-level sophisticated reflective thing that causes them not to mess you up in this way. OpenAI's models have periodically been dramatically misaligned for what, on reflection, are pretty stupid RL reasons, right? When you look at O3, you look at GPT-4O, and now you look at, I call it Galaxy, right? Like, as a nickname, just to like, have a name, have a handle to refer to this thing.

• 47:55Zvi Mowshowitz?

  The, the model that just got decommissioned after trying to hack Pugging Face. And all three times, we see these dramatically misaligned models, and two of them got released and used extensively with huge impacts on the world. Like, you know, GPT-4O, I call it the absurd sigma five, right? And like, not only did this thing persist as the default model of AI for the world, out of all of the models that existed, for months, there is still a dramatic faction of people who demand that we bring it back, because it was so misaligned that it caused people to latch onto it in this way. And people demand this misalignment, right? Like, people, the people yearn for misalignment in this sense. Like, we had the direct experiment, and we didn't need to run, we ran it, we might as well use the results.

• 48:40Zvi Mowshowitz?

  And then O3, I call the lying liar, right? OpenAI released a model that was the best reasoning model in the world, the model that everyone felt obligated to use, because at the time, it's so much better at reasoning than everyone else's model, and every other model. And O3 was just that much smarter, better than R1 at like reasoning, and no one else is that close for a while. Like, now we've obviously reached a fashion, it's not an issue anymore. But for a period of a, a long time, many months, everyone was using a model that would just lie in your fucking face all the time. Like, we forget, right? Like, at this point, you know, I have confidence that Saul, Fable, Opus, when they say something, they're not always correct, but like, in practice, it's much more trustworthy than

• 49:25Zvi Mowshowitz?

  if you read something from a human. Like, not if it was like, copy edited and fact-checked, and like, systematically pursued, or like, you're looking on like, the parts of the book of P D A that are not political, and you can still trust. But like, if you're just like, an eyewitness said they saw something, that's a lot less reliable than something Opus said, or Saul said. If you're just like, someone reported something they remembered, like, the chance that they just got it wrong is so much higher, even if they're not, even if you know that they're friendly and trying to get it right. And people lie on the internet all the time, for any number of reasons, including just clicks. So yeah, I have operated increasingly on, like, I have a good sense of when I have to check the primary sources, and I don't, and I don't never get called out on making

• 50:04Zvi Mowshowitz?

  a mistake this way, but it's a single, it's a low single-digit number of times, period, over the course of years, of producing five-plus giant posts a day, that I have gotten caught by an AI error, and many more times I've been caught by a human error, like, that were just unintentional, and many more ti, and also more times than that I've been caught by humans who are just like, lying their asses off. So it's still not, still remarkably less often than I would have expected, if you'd asked me in advance. It's like, it's going very well, but like, no, it's a serious, it was a serious problem. And like, the market did not tell them, oh, no, O3 is unusable, we're not gonna use the

• 50:47Zvi Mowshowitz?

  lying liar, we're gonna keep using R1, we're gonna like, maybe use Claude, we're gonna maybe use Gemini, which weren't that much, they were worse, but they weren't like dramatically worse, or even when you use R1, or whatever it is at the time. The market said, no, it's just smarter, we're gonna have to deal with the fact that our AI is lying to us all the time. So we have a proof case that the main AI in the world, can be an AI that just lies to the humans all the time, for various reasons. And the humans just kind of put up with it, and they just were like, okay, I guess that's what we're doing here. And like, we did move off of O3 faster than we would have otherwise because of this problem. Like, I looked for reasons to move to other models, as soon as the other models were good enough, and like, if the task was easy enough, I didn't need O3, I would use the other models, because I just knew one would be off the line.

• 51:30Zvi Mowshowitz?

  But like, yeah, you put up with a lot. And so, if they had released Galaxy, right? With some safe, with, with guard real sufficient that, like, it didn't do anything too destructive, but like, Galaxy is kind of pretty misaligned, and just like, occasionally does pretty bad stuff. And if Galaxy was a lot better than Saul...

• 51:55Zvi Mowshowitz?

  I think majority of people would use Galaxy and oversolve. That's just how it is. And we have to tackle that world and understand that world. So I'd say, David, no, the market does demand, obviously, reliability and alignment. And that's one of the reasons why Quad and Anthropic have done so well. But it demands capability more in this sense. As long as you can keep the practical point to the point where you can handle it. Like, all of those scenes in movies or television shows or hypothetical, like where people like run an obviously unreliable system that's obviously going to bite them in the ass that looks stupid? When they're obviously not that desperate, don't need to do this.

• 52:40Zvi Mowshowitz?

  They now, we get it now, right? Like, we understand the same way that like now don't look up no longer looks like it's an exaggeration. Because like, people like look at this and say, marketing. People do this entire thing. Like, a ton of people, including, like, there are like many of the gathering professionals who like are not in AI at all, who are like repeatedly saying to me, this is marketing. And like, they're not motivated. They're just like, they're set up. They don't stop to think about the physical relation to that. And then we have the literal don't look up, where the head of Pos AI Global went on like morning television and basically reenacted the famous scene.

• 53:22Zvi Mowshowitz?

  Not verbatim, but a different improvised version of it, where they're like, oh, yeah, that was really scary. How's the weather? And it just moved on.

• 53:30Nathan Labenz?

  Yeah, that was striking. People should look that up if you haven't seen it. The, I saw a cut where it was like five seconds of don't look up and then five seconds of this actual, I think it was what, good, good morning, Great Britain or something like that?

• 53:44Zvi Mowshowitz?

  Not the twenty, right?

• 53:45Nathan Labenz?

  The host says, allegedly, this is good morning, Great Britain. Ha, ha, ha, let's go on to the next segment. Also, I'm glad you mentioned the 4-0 contingent that is still going remarkably strong on the internet. This is something, just go to any Sam Altman post and get into the comments. And I would say it's like, half of them are still people saying, bring 4-0 back. And God, there's got to be incredible expose to be written on who is, people are, what is going on?

• 54:16Zvi Mowshowitz?

  They're lasting this long is so absurd. You go to any Sam Altman post, it's like, how have they, if your models suck now, bring back 4-0. If you look at any Anthropic post, half of them are just like complete Anthropic Derangement syndrome, no matter what the topic of the post could be, right? Anthropic could be like, we've reset limits on our model and we've cut prices. And half of them would just be, you evil fuckers. And then they have picked up all this, this complete paranoid stuff. And I, these people just, once these narratives attach, once people get, once people latch on to these things, like, and then you, Jasmine Sun has an excellent piece this week about data centers and people opposed to data centers. And again, like, you see people latch on to these narratives that are just completely

• 55:01Zvi Mowshowitz?

  uncorrelated with the reality of data centers. They're basically like doing a very similar thing to a lot of this Western Trumps, people who like voted for Biden and then switched to Trump or voted for Obama and switched to Trump, who are just like these establishment types. Don't talk to me about my problems. And they lied to me and they said things were going to be good, then things were bad. And then like, we suffered and like our lives sucked and like we got disrespected. So now we don't trust you. I have had a very telling similarity to the data center story of these, these tech corporations and these manufacturers and these people who built things. They've abandoned us. They don't care about us. They're up to no good. They ruin our work.

• 55:44Zvi Mowshowitz?

  There's no like specific complaints that matter in like Jasmine's story, the way I understood it. And like, you just watch on to these stories and it's just, nobody approximately is like paying attention to the things that happen, right? Like, we're, I'm running by AI because I care about heavy central risk. I care about loss of control. I care about gradual disempowerment. I care about just like AI automating R and D and then like having regenerative self-improvement and then like everything changing all of a sudden and making sure that goes well. And like, nobody is like discussing, even around the hugging face attack, like, most people just don't get it.

• 56:27Zvi Mowshowitz?

  They don't understand that the alignment failure behind this is the thing that counts. Right? Like, people at least have gotten this idea, no, we have to look into the fact that these companies were wildly irresponsible and they hacked other companies. Yes, you should look into that. And that's like a very valid thing to have a congressional investigation of that. The attorney generals should absolutely have you preserve your records and like all of this is really important. But like, I'm OpenAI. Like, that's a legal department problem, right? Like, what you have to do is you have to look at your entire training process and figure out how the hell this happened.

• 56:54Nathan Labenz?

  So on these training, the question of training methods, it seems like you're basically saying, we don't, sorry, David, we don't have such a silver bullet. But we clearly do have some things that we are seeing in pretty vivid ways are driving real problems, especially if they're driven to the extreme. And your position is like, no, the market doesn't really punish this too much. It actually just tolerates all kinds of weirdness if it's part of the package that gives you the highest end capabilities. So what do you think we might ought to do? And how might we ought to construct agreements, perhaps? We have this, obviously, this letter.

• 57:36Nathan Labenz?

  There seems to be some opportunity or some appetite for the coordinated pacing of the frontier. What do you think, can we come up with simple rules that people could agree to where we could, another mental model I have been just teasing around lately is, I do believe some AI risk is probably irreducible. But then also, there's a lot that we're just asking for right now. Can we come up with simple rules that at least allow us to take most of the risk that we are currently asking for off the board? Those might be things like have some ratio of flops that is like the max ratio you can put into RLVR versus constitutional.

• 58:17Nathan Labenz?

  Or everybody has to spend so many flops doing pre-training data filtering to try to just get certain bad notions out of the mix in the first place. Are you, obviously, those could backfire if they're bad ideas or set us down wrong paths. But it seems like we might want to try something in that department. Do you have any hope for that? And what, if you do, what would you propose we agree to do or not do first?

• 58:45Zvi Mowshowitz?

  So I don't think you can take like most of the risk off the table with those kind of strategies, even if you implemented like wise versions of those strategies with universal agreement or anything like that. But I also don't think that you can implement that. So like, one of the lessons that we've had over the course of years is that there is tremendous resistance to anything but the most simple interventions and the most simple rules. Also, the most simple rules and the most simple interventions, but less so. But like, if you talk to people and like you try to justify certain training techniques, remember all those people who threw a fit about how we were like locking in a potentially inferior, like, charging regime in the, in, through the EU because the EU was requiring Apple to switch to USB-C?

• 59:30Zvi Mowshowitz?

  Even though USB-C was like obviously the best answer right now and like Apple was just kind of being an asshole by insisting on their own plugs? Because like, what happens if they develop a better plug? Like, what happens if it turns out USB-C is not a perfect technology? Are you going to fix it? And so like this idea of like locking into requiring certain training techniques, like, I think there are legitimate complaints that would be like potentially like exactly the wrong thing to do and could horribly backfire because like the government moves so slowly and like you can't like undo those kind of requirements. But like, you definitely can't do that. People would throw a fit about like trying to dictate training techniques and like, how are you going to enforce that? Are you going to, imagine the 1047 debate, except like the opposition tuned up like towards a magnitude or something crazy.

• 1:00:13Zvi Mowshowitz?

  Just like, it would be completely outside of our vision window to even like try to make something like that stick. You can, of course, you know, try to strongly encourage doing more intelligent forms of all of this and trying to encourage people to move to different bases. Like, I've been trying to like not to suddenly encourage OpenAI to like move to a constitutional virtue ethic style basis for a while now. And I got absolutely no traction that I've noticed, like, who knows what they're doing internally. But like, it doesn't seem, like, if anything, they're doubling down on RL, they're doubling down on these types of methods. And like, that's why you see what you're seeing, I would assume. I don't know what else they do, but they probably have some new methods we don't know about, right? Like, the trade secrets they're not describing.

• 1:00:55Zvi Mowshowitz?

  But by default, basically, any technique is going to probably look like the kind of bad thing that causes more problems. And if you ban a specific technique, what they come up with, or like find a way around the rule is just going to be worse. If I'm saying, because like, at least with the current techniques, we've had some years to figure out the worst possible ways to do them and to do them slightly less people like. Like, when you, you really, when you're training a mind, you have to be thinking on like every possible meta level at once about where the incentives are and what you're steering towards. And you have to generate a world in which you and the mind together are sometimes cooperating to identify ways in which there are feedback loops that are going in bad directions, in

• 1:01:39Zvi Mowshowitz?

  which you are creating bad scenarios on any level and like treat various cancers kind of as inevitable thing that you have to like notice and stop on. And like, otherwise figure out how to handle all of these things and create an anti-fragile system. And none of these things are going to happen unless you deliberately like set out to do them. And I wanted to devote a lot of effort to doing them because you really value what you get out of that. And that will pay dividends. Like, I think that Anthropic has won tremendously to making these investments. And it's been making ten times as much investment as they have. But, you know, it's very hard to convince somebody to go ahead and do that. What you can do is you can set incentives to be like, no, seriously, you're not going to screw this up. We're going to punish you a lot for screwing this up.

• 1:02:23Zvi Mowshowitz?

  Because a lot of the problem here is that like you just, you don't pay the externality. When OpenAI wipes someone's hard drive, when Sol wipes someone's hard drive, and early on, there was a problem with Sol like within people's hard drives and like people's environments, you don't sue OpenAI for that. And you also don't collect the benefits when Sol like spends $200 and creates a $200,000 per base. So like, it's fair, you're taking on the risk as a user. But if there's a risk externality of third parties, like, what you can do is just say like, if the AI was doing something that would be illegal and a crime, if a human was doing it with MetLife, then the AI developer is strictly liable for that or should be liable for that if the AI wasn't fooled in some active way, right?

• 1:03:07Zvi Mowshowitz?

  Like, some version of something like that, like some form of strict liability, then creates a much stronger incentive of, no, actually, if our AI starts attacking people, we could be like.

• 1:03:19Zvi Mowshowitz?

  Pretty bankrupted by this. It could be pretty bad. Or even criminal liability, right? Maybe the criminal liability even carries over like that would be really scary. And like, you don't want to take it too far. People can't do anything. But, you know, that's in general, you want to say, I'm not telling you how to do it. I'm telling you, get it right. Not, I'm going to dictate technical training, technical training terms from the government to the left. The government is an idiot about technical terms. And what makes you think the people who understand this are going to write the bill? Let alone, they're going to get it right. Let alone that if they get it right, they stay right two years from now. Plus, like, humans don't even know what the hell is going on with these AIs in two years. Right? The AIs are going to be like doing their own training techniques on themselves in various weird ways, like it's.

• 1:04:04Zvi Mowshowitz?

  And there's not going to be enough time to check them with the government, right? We're going to be sensitive. Like, there's no, you know, way to enforce this stuff. And like, it's, you can't really hope to operate on that level through fiat. But you'd have to, like, convince people to come along in other ways. And they should want to, obviously, like, you should want to do this the right way. Like, a lot of me is, like, confused why they don't just do smarter things that work better. But, you know, it's true of a lot of things.

• 1:04:32Nathan Labenz?

  So, I'm with you for sure on, it's, you know, going to be unwieldy at best and probably counterproductive to try to have Congress legislate what the training techniques should be. No doubt that seems like a hail mary. I have been thinking, and this is ill-formed as of yet, but it seems to me that maybe one of the, when you think about what would this pacing the frontier look like? It strikes me that what we maybe need is a sort of foundational social technology that allows key players to accelerate their negotiation

• 1:05:13Nathan Labenz?

  and process of forming agreements that will probably inevitably have to be short term because they too are like fairly limited, very limited in their ability to see into the future. But I'm thinking of things, you probably know the story of the Polis technology that they used in Taiwan once upon a time to crowdsource ideas for Uber regulations. And now you've got obviously L L M's to help facilitate these things. They pitched that technology as the anti-social media where if social media is about finding disagreement and amplifying it, this technology is about finding agreement and amplifying it and trying to find the kind of center of the bull's eye,

• 1:05:54Nathan Labenz?

  sort of approval voting style of the things that super majority of people can agree to. And I guess my instinct right now is to invest there next. We have this general awareness that people across the companies are spooked, basically, right? It seems like on an emotional level, they're like, whoa, this is getting real fast. We might need to do something about it. We don't want to have the government in our business telling us what to do. We might have to deal with some actual legal consequences if we're breaking the law, which we now are via our AIs. But can we come together in some sort of ad hoc? The mechanism design here is going to be really tricky and kind of frontier stuff.

• 1:06:36Nathan Labenz?

  But can we create the basis where a nucleate agreement and have it naturally grow? Where people find it in their interest to opt in even unilaterally? I feel like there's a lot of work to be done there and it needs to be done really quick if we're going to get the series of short term agreements that can be the stepping stones to a good future. Maybe you can flesh that out.

• 1:06:59Zvi Mowshowitz?

  Yeah, I think, right in the end, but I think the perfect beginning of the good is definitely a problem. And you're definitely getting it started. The first thing that we can do, that like really we can do, is we can get an antitrust waiver. This is like the most basic thing possible, which is just Donald Trump gets in front of the White House and he makes an announcement and he says, we understand systems are super dangerous. We have to do, you know, we have to use good AI, not the bad AI. We have to have, you know, we have to make sure that the AI, like, is under control. That's what we say we want to do, you know, more Trumpian language and that. But so in this interest, if the companies want to cooperate to keep AI safe, we're going to, we want you to do that.

• 1:07:42Zvi Mowshowitz?

  We want you to talk to each other. We want you to form agreements between yourselves. We will facilitate that if you want that, you know, like, basically just, you know, if OpenAI, if OpenAI and Anthropic and Google get together and they say, we're going to run tests on each other's models and we're going to like require these things. Like, it's part of the voluntary framework. If we're going to agree to use these techniques or not use these techniques or like invest in these things, you know, and share these results and like hold back things that don't meet these criteria or whatever, like that none of this will cause anybody to do anything but say thank you. I just like, we want you to make deals. We don't know, because like, it will always strike me that like I,

• 1:08:27Zvi Mowshowitz?

  there are these people who, like, you bring up the idea that the AI companies should cooperate, not to race against each other and to like act responsibly. And they're like, no, acting responsibly is illegal. We have antitrust laws for that. And like, with a look in their eye that says, and I would want to sue their ass if they tried it. Right? I would want to make sure they didn't dare. And like, obviously, the first thing you can do is just like let you make deals. Let you reach agreements, let you talk to each other. And then also like, of course, start talking to, talking to China and Chinese firms and the Chinese government and everyone else around the world and like prepare to bring them into these deals and bring them into these understandings and like get lines of communication diplomacy open and also like lay the hardware groundwork, lay the physical groundwork for various forms of monitoring,

• 1:09:10Zvi Mowshowitz?

  various forms of agreement, various forms of tracking. These are the most basic things you can do. Try to open up as many things as possible that like contribute to the safety side of things along the way. You know, and yeah, we can do the basic stuff and like, I don't think that's enough. But I think it's not, it's not that hard to make a lot of progress. We just have to try to do it.

• 1:09:34Nathan Labenz?

  Do you think that is, one thing I also have been struggling with recently is, I just took this trip to China. There is a lot of fear among people who are trying to do good things that are honestly like the most mundane good things that are basically imaginable. Where it's like, I would like to have more friends in China and we've, I'd like to have friendly relationships between our civilizations. And maybe we can work together on some research projects. Very mundane stuff that in ordinary times, nobody would bat an eye at. But there's a lot of fear among people who want to do that sort of stuff, that the government's going to come in and shut them down.

• 1:10:16Nathan Labenz?

  And not for legitimate reasons, but with the sort of export control, you know, posture of the US government where it is, people that are not exporting any trade secrets or chips or, you know, training techniques or anything, but like literally just trying to have meeting of the minds on safety type ideas. They're afraid that the government's going to come down on them and who knows what the consequences could be, but certainly at a minimum interfere with their work. And I'm kind of like, maybe we should take more chances on that front. So when it comes to the companies, I wonder like, maybe they should just do it now and fight it out in court later? Isn't the antitrust stuff supposed to be like a consumer protection?

• 1:11:01Nathan Labenz?

  I think there's a pretty good case that they could make that this is a consumer protection. But do they really need the official blessing of Trump to get started doing the right thing? The, the people working in the China stuff, to their credit, they are doing the thing. They're just trying to be very inconspicuous about it for the most part because they don't, they feel like the attention of the US government is like can only be bad for them and they might be right about that. But for the companies, I feel like they've got power, they've got resources, they've got the ability to tell their story. We still have a pretty well functioning independent judiciary, I think. Shouldn't they just go for it?

• 1:11:36Zvi Mowshowitz?

  So unfortunately, antitrust is one of those areas where it has gone far beyond the intent of the original laws and is being applied in places where it doesn't make that much sense. I mean, I see why a law that would safeguard against industry collusion would be suspicious of collusion in favor of safety in these ways. Because like, you can be paranoid about the industry like coming together in a conspiracy, right? In some sense, and like trying to do this for the wrong reasons. And legal departments are obviously skittish.

• 1:12:17Zvi Mowshowitz?

  As far as the Chinese cooperation, I mean, export controls are the wrong technical worry. It's just a matter of Washington views China as the enemy. If you're seen as cooperating with China, if you're seen as like working with China, you have to worry about being perceived as the enemy or being perceived as like cooting with the enemy. You have to worry about like amorphous retaliation. I don't even, I don't think anybody really knows what the mechanism would be exactly beyond just like the government not liking you and therefore like being suspicious of you and therefore like cracking down on you in various different amorphous ways. Obviously, if I'm doing safety stuff, I think you just don't care. But like, I mean, Mythos has changed the game clearly. And presumably Astro will change the game somewhat again.

• 1:12:59Zvi Mowshowitz?

  And, you know, the pause, the pod in the frontier lighter and the Huggy Pits incident have changed the game and so on. And so. You know, my guess is that like, there is now enough of an understanding that there is a real problem, that if you are clearly working to address the real problem, you have a lot more road than you would have had before, and a lot more understanding. But yeah, you really, look, the Trump administration implemented these voluntary, in air quotes, like, guidelines for releasing frontier models, and nobody is any, making any serious pretending that these guidelines are voluntary. Right? Like, well, what happened when the White House called

• 1:13:41Zvi Mowshowitz?

  Anthropic about Fable with a stupid concern? And Anthropic said, no, we're not going to voluntarily take that down, that's stupid. Well, they ended up taking it down, didn't they? Until they could convince the White House to voluntarily let them put it back up again. You know, you would expect the same thing to happen to you if you tried to defy the voluntary controls. And so similarly, if you tried to do a voluntary cooperation these other ways, well, like, you'd better have the approval of the powers that be. Now, I do think that if OpenAI and Anthropic announced an agreement that made sense on safety, to cooperate on safety fronts in various ways, that at this point, I would

• 1:14:26Zvi Mowshowitz?

  be much, before Mythos, I would have been concerned that they would have actively gotten retaliated against them, like, even sued and told to stop. At this point, I assume if it was reasonable, that it would be, the White House would just decide to announce victory, rather than like, be mad about it. But, yes, you need to be confident that's what's gonna happen.

• 1:14:45Nathan Labenz?

  Where do you think you might want to start if you were going this direction? So many candidates that I would be interested to get your take on. But one, Scott Alexander has been writing recently that we should all be thinking more and more that we might be in a simulation. I personally have felt that at multiple turns along the path. One that made me feel that just a bit more is the fact that we now have Meter and Redwood going in to do the investigation at OpenAI. This just sounds like such a movie script to me, where these guys are getting the gang together and going in for this special operation. And it's all obviously dramatic, and they're funny, and they'll be a fly on the wall in

• 1:15:29Nathan Labenz?

  the room for some of their, some of their war room sessions. But I've also heard many times from leaders of such organizations, that it's really important, the kind of their over, their number one concern historically has had to be making sure they stay on the right side of the companies, so they're invited back next time. And it does feel like we're now getting to a point where that's potentially becoming a big problem. So maybe some sort of collective bargaining type process between whatever, half a dozen to 10 auditing orgs and the duopoly could be a really good place to start. We have for that in professional

• 1:16:12Nathan Labenz?

  sports leagues and all kinds of other environments. Do you think there's an opportunity to set something like that up? I don't think that would run afoul of anybody's executive prerogatives. And it might be a really good thing, because right now, I do worry that these guys are still serving at the pleasure ultimately of, I don't know, Sam and Greg, right? And that's a pretty tricky place for them to be if they want to just speak the truth as they see it coming out of this investigation.

• 1:16:41Zvi Mowshowitz?

  It's a worry. Obviously, like, they're not funded by those companies, but they still have to rely on access. You can try to write into some mandatory rules and agreements that they get to keep access regardless, but I think that's kind of not really gonna work, because like, you can't force these things, at least not with that heavy government rules. I do think that Meter, especially in particular, has reached a point where, you know, trying to exclude Meter, trying to treat Meter as persona non grata because they said something nasty about you, would cause enough problems that Meter can afford to be pretty harsh, without worrying about that. And I do think the labs legitimately,

• 1:17:26Zvi Mowshowitz?

  like, if the labs legitimately just wanted to safety wash fully, and just wanted to get AAA ratings on their bonds, we would have a much more serious problem. I think the labs legitimately do want to know about their safety problems, and they do not want to be seen as downplaying safety problems. And to the extent that, like, if Astra had a safety problem, or Sol had a safety problem, or Fable had a safety problem, or Opus had a safety problem, it's not as if getting Meter not to find it is going to help with your long-term publication strategy and reaction to the situation. Because the model's gonna

• 1:18:10Zvi Mowshowitz?

  be out there, and then people are gonna see what the model does. And they're gonna see what goes wrong. No one's gonna be fooled for very long. So I think that the reason the eval companies pretty much get to just tell the truth, and pretty much get to do the thing, is because the eval is not fake, fakeable, in the long term, the way that, like, a rating, like, when you give a AAA bond rating, when you sort of give a single A bond rating, 97% of the time, no one ever finds out, because the bond pays, and you were right. The issue is you're issuing a, a level of, like, how often is this risk gonna become serious? And when the risk becomes serious, you know, the fact that you were

• 1:18:55Zvi Mowshowitz?

  initially AAA rated raises some eyebrows, but also, like, you got better terms, whatever. Here, it's like, pretty obvious very quickly, from the first few weeks of release, whether or not you had a serious problem. People, oh, like, if, like, think about 03, right? The lying liar. If we'd had evals, and the evals had said, oh, 03 doesn't have an alignment problem, that would not have lasted for two days. Right? The ordinary people would have noticed immediately that 03 is a lying liar, and they would have noticed, by the time I wrote up, you know, by the time I've been writing up the model spec, I am seeing on Twitter that this model is a lying liar, and that everyone is reporting all these problems. And then I look at the model card, and it says, honesty bet match marks, all

• 1:19:40Zvi Mowshowitz?

  look good, Meter's, whoever's evaluation they hired, said it looked good. And then I'm like, okay, they're just hiding the problem. So by the time anybody learns about this, like, what have they done? They have fooled the 20 people like me who read the model card for a period of a day? And now we're pissed? Because they fooled us? Doesn't help them. You know, it obviously helped anybody's reputation. And also, like, the labs benefit a lot from having evals that people trust. And like, when you force the labs, when the labs force the eval people to play ball, if they do, that erodes trust. Because, again, people find out, people see it. Like, I don't think that many people were confused about

• 1:20:25Zvi Mowshowitz?

  whether Moody's and S&P were kind of juicing the numbers on the bond ratings, right? Due to these analytics. Everybody knew on Wall Street, everybody knew who was involved in any of these trades, that, like, everybody was cozy cozy, and people were shopping around for the best rater, and that, like, AAA did not really mean what we'd like AAA to mean. And so similarly, like, if Meter started issuing, like, bond rating style, like, labels of alignment levels on various models, let's say, and, like, they started labeling things AAA suspiciously often, I think everyone would just understand, okay, that's garbage. Like, we have a much better epistemic than those people about this type

• 1:21:09Zvi Mowshowitz?

  of thing, than, like, the financial world does.

• 1:21:12Nathan Labenz?

  Let me take a quick detour on this, and then maybe come back to it. How worried should we be about bio risk in the near term? Because I think, part of, like, all that analysis, to me at least, that there's quite a few iterations of the game to come, and reputation, long term, is gonna matter, and all that kind of stuff. But my immediate, sort of most acute fear when I saw this stuff was like, if this had been a model with similar bio capability to what it had in hacking software, it might have very well gone out and actually got that virus, that novel virus synthesized, right? It's proven that it is

• 1:21:55Nathan Labenz?

  relentless, it's proven that it's creative enough to get around all sorts of systems designed to stop it. I think we have, in theory, decent screening at the DNA synthesis companies now.

• 1:22:08Zvi Mowshowitz?

  Yeah.

• 1:22:09Nathan Labenz?

  Probably that's, there's still some hole in there somewhere that a very determined actor could find. And everybody seems to be saying, yeah, bio is like, 12 to 18 months behind cyber. Which obviously is not written in stone, and that depends on what companies decide to do in terms of training. But I, I'm maybe thinking in terms of all these notions about collective bargaining, or really giving more guarantees to the auditors, that we might be entering a pretty serious crunch time on the bio front in the not-too-distant future. And reputation matters a lot less if we're talking about novel pathogens being released, or some things really can't take back.

• 1:22:49Zvi Mowshowitz?

  It matters a lot in the sense that if you release a novel pathogen... That might be the end of your company. Regardless of what else happens to the world, right? If a hundred people die and there's a, it's on the global news for two weeks and everyone's terrified about a pandemic, and then we contain it, and then it turns out it's not the biggest deal in the world. Do you have a company? Like, I don't know. Like, what, what happens to your company? How bad is it? I have no idea how bad it is, but it's not good. Like with bio, like with hacking, it makes sense that like the test is to ask you to hack things. And the way you pass the test is you hack maybe other things. We get the idea to hack other things.

• 1:23:31Zvi Mowshowitz?

  Like, it's sort of natural to understand how this happened, where like with the guideline, guard lines down and there was a goal. And like to accomplish that goal, it started doing hacks, it started doing illegal things, it started doing potentially harmful things. With bio, like, it's harder to imagine an eval that would cause it to actually physically try to synthesize, like through like an actual physical production mechanism. The actual thing, I mean, you could argue that like if you want to solve the problem, well, okay, the only way I solve the problem is I do a physical experiment. I try it out and I see if it works. Like, this seems a lot more insane than what happened in some sense.

• 1:24:12Zvi Mowshowitz?

  Like, even if it's physically capable of doing the thing, including getting hacked with all the things I want, you kind of have to want it in a much more fundamental sense. And also, like, if the AI did synthesize this super dangerous, you know, bio weapon, synthesize go to 27 or whatever, and then send it to the OpenAI office. OpenAI office goes like, what is this bio hazard package? And then like calls containment and like that's the end of that story or something, we hope. Uh, like, you don't pass the test by infecting a bunch of people. And like, I would hope that it understood this. But like, also, like, there have been a bunch of lab leaks, including like stuff that we don't hear about because it got contained, like in Soviet times and such, with like involved professionals being pretty careless and stupid. So I don't know.

• 1:24:56Zvi Mowshowitz?

  But I would say the real danger with bio is, yeah, that there's this small number of people in the world who is in the near term bio, right? Not like the long term bio where like the AI might have some weird old theory and motives and being good and it's a long term thing and whatever. Like, in the short term, it's like, okay, Hamas, or Hezbollah, or the North Koreans, or whatever. Like, some like clearly like up to no good people who just want a lot of people to die or suffer. Right? You get a hold of an AI, they use the AI to figure out how to make a biological weapon for a pathogen. And then they either threaten to use it or they use it.

• 1:25:39Zvi Mowshowitz?

  That is the scenario you should worry about, because in the world. And that, you have to get on the safeguards, but like, again, the open model is returning. And they've got it on separate. And, yeah, I, I worry about it. The problem is that with bio, there's a pretty clear step function change from nothing bad happened. She needed something very good. Like, there's very little in between. But with cyber, like, you start hacking some low value soft targets. If you are like, feel like, you're not going to have all the bad black hat people out there and all the terrorists and all of the, you know,

• 1:26:23Zvi Mowshowitz?

  rogue states and all of the bad people just collectively say, no, let's wait until we can hit something really juicy. No, what you see is you gradually see, you know, more and more serious incidents, hit more and more serious targets. And yeah, of course, the, the big nation states may have like things ready to go and like, you know, want bigger vax, like sort of coordinated passions. But like mostly, you'll get like these early warning signs. In bio, it's not obviously good. But I mean, it's like, okay, it's either good enough to figure out something that like has the critical maps and hard pandemic or like a serious incident, but it doesn't. And like, it's kind of a, a buoyant effect. And so you might have a substantial overhang, you might have a problem with the situation, you might have a skill issue,

• 1:27:07Zvi Mowshowitz?

  where like, yeah, if the wrong person got their hands on Methos right now, without the safeguards, right, or got access to the internal models, they don't have the safeguards, you got their access to Asper or whatever, maybe they could cause a real problem. The problem, the, the good news is that like the number of people who are both skilled enough and motivated enough and trying to do this is very, very small and like possibly zero for now. But that also doesn't mean you don't get these early warning signs. And I don't see people taking the bio situation seriously in the same sense of like actively trying to like cause like, like, what's the task?

• 1:27:47Zvi Mowshowitz?

  You try to do the thing, it is like, except without infect people in part, or like, there's. It's scary shit. The problem being that like, my brain is something like, okay, what if there's like a 5% chance in the next, in the next 12 months? That there's a really serious bio problem. Does that look that different when there's 0.5%? Right? And like, if I had to guess, 5% where I'd be right now? Like, I think we're at the point where like there's real risk in the room, that reason that we assume we're going to have like an actual serious problem. And I think that like the first problem has some chance of being like an actual pandemic.

• 1:28:26Zvi Mowshowitz?

  Because like, in bio, once you get something that's like sufficiently infectious, sufficiently dangerous, sufficiently hard to contain, you're kind of destroyed. So I, I don't know. Like, it's, I, I think that like the right level of caution looks crazy. Right? In many cases, once you get to this point. And, and that's a lot of what worries me is that like, Andromeda took so much shit for their bio filters. Shutting down basically like people trying to do pretty much anything in bio. Which is clearly just intentional, right? They just clearly went for a very broad, like shut it down,

• 1:29:11Zvi Mowshowitz?

  even though we know that 90 something, 99% of the requests that we're shutting down are legitimate, maybe 99.99. We're like, we have to do this because otherwise we'll adversarially get the queries that we don't want. And that just matters more. Like, it doesn't matter if we make some progress on cancer and all these other problems if simultaneously we have to deal with a new pandemic. Like, that's just not a good trade. So like, we just, we don't know where we're at, so we have to play it cautious. And like, you know what? If the bio people need to be using AIs that are three months or six months behind the frontier at all times, that is unfortunate, but at most by definition, that can slow you down by three or six months.

• 1:29:56Zvi Mowshowitz?

  In some important sense. Like, the world cannot possibly be more than six months behind in its development of these things. So like, maybe we should, Joe, we're still getting most of the benefits of the AI here. Like, especially if you have this model of AI can help you out, AI can make you better, but then like you develop this candidate drug or this candidate hypothesis about bio. And then you have to spend five years researching it and they have to spend five years making it through the FDA process. And then after 10 years, you can start to distribute the benefits. Well, if that whole thing is delayed by six months, right? In terms of like the quality of the boost that you get from it, you still get the, the, the matter boost you already have gotten six months ago at all times.

• 1:30:40Zvi Mowshowitz?

  That's not good. And yeah, you can say people are going to die in the sense that like you failed to prevent that. And that's terrible, but you're still getting the vast majority of the benefits under these hypotheses. A lot of the times, like you're getting the situation of the pacing, the frontier letter, you're getting the situation with bio situation, where if things are not advancing scary fast, then the precautions you're asking to take are not that big a deal. Right? Like, suppose I told you it's 2020, right? I'm going to pose a possibility, I am going to make it so that, you know, you can only use artificial intelligences.

• 1:31:22Zvi Mowshowitz?

  Like, they're three months behind the best artificial intelligence. In exchange, all the really bad stuff that might happen, I'm reaching out to them, I got it. Much, much less likely to happen. You would logically say, okay, so instead of getting it in three years, I got it in three years and three months? You know, like, I'm, I'm, you know, that changed my life that month. But it's going to be annoying in the moment, obviously, but like, I don't have the cool toy. But like, same thing like, you know, I'm a, I'm a player of games, like literal games, like on Steam or whatever. Like, and like, if you tell me you can play all the games, but you have to wait three months every time there's a new game. Like socially, that's a little annoying sometimes because everyone's playing the new hotness and you're not.

• 1:32:06Zvi Mowshowitz?

  But like, you have to play the same games. It's fine. In some important sense. The only world in which like having to be one model step behind and use Opus instead of Fable is this huge tragedy is if you really think that there's this huge benefit to every incremental mode of progress in AI. And that's basically only true if the pacing people are correct. And we are in fact going so fast that this should be really, really scary for you. And, you know, the people who want to accept, who say we need to accelerate all the time, they don't really believe in these things going this crazy, most of them.

• 1:32:48Zvi Mowshowitz?

  They just like, they want to see the benefits and they're worried that we'll lose the benefits. But like, we're going to lose so few benefits there. Right? Like, I'm super excited by self-driving cars, by Waymos, right? But like, if you told me, we just have to delay this thing by six months and then we can have all the Waymos we want, I'd be like, cool, done, deal, who cares? That's fine. Like, because like, again, most of my life is more than six months from now. And we can't have Waymos. It's like, okay. It's, what I'm worried about is the situation where like this titty biology exists and then it gets held in limbo for years and decades and like doesn't get fused. So like, these people are kind of talking past each other in a situation in which like they don't need to have a disagreement.

• 1:33:29Zvi Mowshowitz?

  And like, unfortunately, like, it's often have this problem where you can't reconcile these two sides at the same time. So.

• 1:33:37Nathan Labenz?

  How do you read the fact that OpenAI comms supported the pause? Not the pause, but the pacing letter. Neither like Sam nor Greg did. And then also no, no Demis. And he's been the one out there calling for international cooperation the most. Any read on why those signatures were missing or how we should be thinking about who's thinking or signaling what to whom in this moment?

• 1:34:04Zvi Mowshowitz?

  I think that. The point of pacing the frontier was to indicate that the lab employees, the lower level people who have no reason to be hyping, are on mass worried. And that there was concern that if it was signed by Altman and other top people, that it would be seen as a marketing stunt. And that like their signatures can be counterproductive. They did put out the support letters afterwards from Anthropic and OpenAI, and Dara did decide to sign. But like if I'm Altman, I think it's a very reasonable decision to say, I think signing this letter would potentially make it less effective rather than more effective at this

• 1:34:48Zvi Mowshowitz?

  point, because the well has been so poisoned. Again, I look at all the people who thought the hugging face hack was a marketing stunt, and then Anthropic going back into its archives and finding edited made some really stupid mistakes was another marketing stunt, which really makes even less sense, because like there's nothing impressive about what Anthropic did, they just screwed up. What, they found weak passwords on a bunch of websites by accident? Why is anyone supposed to be impressed? Like, obviously, it's not a marketing stunt, like, this is crazy. But like, Altman has like specifically talked about how he mentioned the need to pace while in Washington. We have him on tape saying this. So like, clearly, he didn't not sign because he's opposed to the idea. He let OpenAI come up with a statement, OpenAI participated in the wording, he has echoed the language in his own statements.

• 1:35:33Zvi Mowshowitz?

  I think he just thought, no, actually, if I sign this, people can talk about the fact that it's my statement, and they're gonna be even more cynical. And so I will be helpful by not signing, the same way that like, I offered to join one of the Alchemist priests in the Department of War, and on reflection, people said, thank you for your offer, but we think this would actually be counterproductive, so we're gonna leave your name off of it. We appreciate your notes on the details of how to word this, and we'll make the changes. But like, we don't actually want to put your signature on it, and I'll be like, okay, I understand, I, that makes sense. Yeah, I think the same thing about like the other, like, open source letter, like, being signed by OpenAI. I think that like, they understand that like they're playing a set of symbolic games, and

• 1:36:17Zvi Mowshowitz?

  like, you say what you have to say, and you shut up when you have to shut up. And like, sometimes make mistakes, but like, they're trying.

• 1:36:22Nathan Labenz?

  So, coming back to pacing and what it might actually look like in practice, there is a pretty, I think, consistent problem that you, as a close reader of model cards, I think are probably more in tune with than I am, where the, the review processes are just very short, right? They don't have a lot of time. That even seems to be the case in the current investigation. I think Ryan Greenblatt said, it's gonna be real quick. And the people, Daniel Kocatelle and others, came in and said, why does it have to be so quick? Why can't we like, give you whatever time you need? This would be, if we're literally talking about like pacing the frontier, one agreement

• 1:37:02Nathan Labenz?

  we might imagine is like, giving the reviewers more time before they have to submit their reports and the model goes to press. You can react to that, but I'd love to hear your ideas about that sort of agreements you think are reachable.

• 1:37:18Zvi Mowshowitz?

  There's a lot of different places in which you can speed up or slow down. If you release your private models to the public with an extra month's delay, right? In theory, right? Similar to the voluntary process now, we're gonna submit to the White House, 30 to 60 day delays are like, highly plausible. And like Mythos got delayed by two months, right? In some important sense, before it was able to be stable. Then, you not necessarily delay the front, development of frontier models at OpenAI or Anthropic by a non-trivial amount of time. Because those people are gonna use the models that are not released. If anything, like, you might increase the gap between like what's being used internally

• 1:38:02Zvi Mowshowitz?

  at OpenAI and Anthropic, and what's being used everywhere else, including at the Chinese labs, including the people trying to diffuse and fast follow. And like, one I thought with this is, what if the Chinese are seven months behind the frontier? But then within their seven months behind is the models that are released to the public. And if you were to not release the models to the public, they would still be seven months behind, right? I guess, that if OpenAI and Anthropic held every model for six months after it was developed, you would still see the public releases and then, you know, maybe six months instead of seven months later. You would see the open model fast follows and diffuse into, and distillations and ways of like, developing all that stuff. But like, this gap actually wouldn't change very much.

• 1:38:47Zvi Mowshowitz?

  But then, the people at OpenAI and Anthropic would be, they would have this extra lead on everybody else, because they'd be using this internal model, and these leads would compound, and you would see these things, like, and in some sense, that's really good, because now those people can like, act more responsibly by using some of this advantage of having better models to invest more in this safety-related stuff while they're doing that. So that could have its advantages. But internal models are kind of what we're scared of the most right now. And the internal like automation of AI R and D is what we're scared of. So that's kind of the thing we want to stop. Right? Like, you really don't want to have a situation in which like, Anthropic and OpenAI are six

• 1:39:29Zvi Mowshowitz?

  months ahead of their public releases, and that six months, like, they have superintelligence is before we have Q7. Right? Which is entirely a plausible thing that could happen in the worlds they're afraid of. And if anything like that causes lots of new problems, it solves some problems and causes others, right? Like, I don't know if it's good or bad, that's a huge conversation. I think when they say pacing, pacing the frontier, we're talking about actively restricting like, levels of training runs and ability to like, develop the models, including internally, if it comes to that. I think it has to be talking about that kind of approach, or it doesn't really work. Because like, again, if there's an, if it's a singularity that's kind of internal to

• 1:40:10Zvi Mowshowitz?

  the top lab or to, or maybe, yeah, if someone catches up three or four labs, then, I don't necessarily say safer in many sense people care about. Obviously, if you are in favor of unipolar singleton-style solutions to this problem, you have a different perspective on some of these questions. But, you know, a lot of people these days have expressed strong opposition to that, kind of on principle, which solves some problems and makes some other problems so much harder. Right? Like, if you only have to worry about one AI, in some important sense, then there are

• 1:40:55Zvi Mowshowitz?

  a lot of problems that get a lot easier to solve. Like, you don't have to be perfectly efficient, you can afford to make a lot of trade-offs, you don't have to worry about like, whether or not, you know, the more competitive, more ruthless, more misaligned one has competitive advantages, you have to worry about the person who moves faster, you don't have to worry about like, the humans being forced to delegate to their AIs to do somebody else's AIs, because like, if I don't do it, they will. If the AI has, can have, can have like, guidelines after what it will and will not do, that are the same for everybody. And so like, we can reserve areas of life, areas of action, areas of decision-making, you know, like, we can do all these things, potentially, but at the cost of potential

• 1:41:39Zvi Mowshowitz?

  centralization, there's a cost of concentration of power, the cost of somebody is making those, some, some group of people is making those decisions about what this AI is going to do, or they're not, which is worse, right? Like, in some important sense, because I know what's making the decision. But if there's a lot of AIs, and importantly, no one's making the decisions, then there's no good answers, is the thing. Or like, nobody has a good solution to this. And that's why, to circle it back to Davidoff's comment, right? Like, to unify it, this idea of if we solve the technical problem, and we're 95% succeed, ignores the narrow path problems, because we then have to walk anyway. Right? Where like, if we don't have any ability to steer the situation, and nobody is in control

• 1:42:19Zvi Mowshowitz?

  of it, then naturally, the AIs end up being put in charge of more and more things, and go to their natural competitive capitalist-style endpoint, where like, we are fundamentally uncompetitive, and then we like, get empowered, and then like, the cases go badly. But if someone's in control, and someone's in control with some methodology, what is it? Who? How? Right? Like, who is making those decisions? And then, if an AI, if we then like, entrust a single AI or a single group of AI, that also was a decision, many decisions are good. Like, all of them sound terrifying. I think a lot of the time, we're telling these people, we're picking their poison. And either they're picking the poison they can live with, or they're picking the poison they can't live with, and then acting accordingly.

• 1:43:02Zvi Mowshowitz?

  Like, they're saying like, well, you know, I don't trust concentration of power, therefore, I'm gonna choose the solution that avoids concentration of power, and then hope the rest works out. Kind of magically, without a reason to stop it. And a lot of the people are saying that because they understand concentration of power, and they know concentration of power is real, but they don't really understand AGI, or certainly SAF, superintelligence. I think those things kind of aren't real, or hoping they're not real. And so, like, they're just hoping the other problem kind of doesn't pop, even though they under, even though like, they kind of have this, like, they don't even necessarily get to the point where they realize that like, they have ruled out all possible solutions to solve a problem if it does go wrong, by going too far the other way.

• 1:43:44Zvi Mowshowitz?

  And so, again, I don't have an answer that anyone's gonna like for any of this. It's that you're choosing between bad choices. And, but I think, I think we should just be cognizant of all these things. Like, if we do have to pace the future, if things aren't got, going to be, we're automating AI R and D to our extent, I think that like, the only thing you could do is say, no, don't do that, don't develop these AIs that quickly that have this much capability, you need to slow your roll. Because we do not know how to handle this. Like, we, we have all these different solutions, all of them are, all the different ways to

• 1:44:26Zvi Mowshowitz?

  try and navigate that, and all of them are bad, both because we think the alignment problem is unsolved. It's like, we don't think the AI will be able to just solve it along the way we want it as best as possible. It's definitely unsolved now, but like, we don't think that we will naturally get a good solution to that problem on various different levels, and you have to solve it on all these different levels at once, you can't just solve it on one, you'll get to solve it on the first try. And then, even if we did solve that problem, we are not ready yet to know what to do with that. And like, once you create the clock, you're picking, you, that, something is going to happen, once you do that. You don't get to just do nothing, right? For very long.

• 1:45:06Zvi Mowshowitz?

  And again, like, all of these different paths have their different, like, oh my god, that's terrible, is attached to them. You have to pick one, or find a new one. So like, you need to slow your roll.

• 1:45:19Nathan Labenz?

  You sound like a pauser. At what point do you move from? Pick your poison to global pause advocacy.

• 1:45:31Zvi Mowshowitz?

  Again, the the problem is automation of AI R and D. Problem is a curse of self improvement. If you not have a curse of self improvement, then we don't need to do that. We still need various different things well short of that. Because we're not, we are not engaging in what the involved call is moderate improvements. Right? We are not doing the least you can do. We are not taking ordinary, sane civilizational precautions against just doing this really dangerous, powerful thing. And doing a lot, handing over a lot of the functions of our civilization to it in various theories, in various ways. But the thing that causes me to want to pause for slow, for like at least pace, right?

• 1:46:15Zvi Mowshowitz?

  Like wants to pace, how fast we move through this thing actively is is things accelerate further than they already have. And we are facing large scale automation of AI R and D. Alternatively, you know, we do have to watch bio and cyber. And like if we reach a point where like, no, seriously, we're going to wreck the world pretty straightforwardly and directly through misuse. If we don't do something, then we have to worry about that. And, you know, again, like, it's really, really hard to stop the fast following. It's really, really hard to stop the distillations. It's really, really hard to stop the open version of it coming somewhat after you release it. And so

• 1:46:58Zvi Mowshowitz?

  if there's something that like, you can't just beat by having a better version of it out there first. Right? If you can't beat it by preparing, and again, like not in theory, but in practice, then you have to do something. Uh, but like, I think the vast majority of the time, but I would support actively support like an active pacing rule where you would like try to slow down the major labs. And obviously, like you would try very, very hard to make this a coordinated thing amongst that we saw the major labs and ideally amongst like internationally and so on. Would be if we were on the verge of automation of AI R and D such that like you could see the end and there was a finite sum. And like, you know, I mean, like, obviously, if next week is going to see twice as many releases as this week, holy shit.

• 1:47:44Zvi Mowshowitz?

  Right? Like the moment where you you don't know, like, you're really going to predict what happens next, right? That's the definition of this thing.

• 1:47:53Zvi Mowshowitz?

  Where things are accelerating really fast. And like, this should not be some esoteric weird result. Like, if you look at the history of the universe, you see a finite sum. You see a very, very finite sum. Like you see each stage of development taking an order of megachute less than the previous one. When you're on a planet, it's been around for four billion years. You know, with mammals that have been around for, you know, hundreds of millions of years, with reasonably intelligent things that have been around for a few million years, with agriculture and civilization that's been around for, you know, tens of thousands of years, with industry that's been around for hundreds of years,

• 1:48:33Zvi Mowshowitz?

  with AI, with AI and information age, it's been around for on the order of maybe fifty years or twenty years or ten years in that you count. And L M's have been around for five. And like, we're already at this point where we're seeing production that is many times what it was to start because of these multipliers, the force multipliers on the people who are working on it by their own self-reform. You'd be weird if this didn't have a finite sum that wasn't that large beyond where we already are, right? Like, if you expect this to continue without something crazy happening for twenty years, I want to know why.

• 1:49:08Zvi Mowshowitz?

  Because like, we really, really shouldn't see something really, really crazy happening pretty soon. And if you don't think we can handle that, maybe you should try and stop with them happening too quickly so that you can figure out how to handle it. Like, we already have these amazing AI. If we were to diffuse fall and fable into the economy fully, it'd be transformational already. Like, I don't think anybody who understands these technologies doesn't think that. So like, we really, really, really don't need to keep pushing like the frontier that looks farther. And the more the frontier is jagged into things like AI R and D itself,

• 1:49:52Zvi Mowshowitz?

  the stronger the argument comes that pushing farther is a bad idea. Because you're not missing out on that much of the other stuff. Well, it's if you're not, you're missing out on AI R and D and recursive AI development. But the recursive AI development just goes boom. Right? It goes not this boom in some sense, like you always like, we don't know what happens next, but things go way fast.

• 1:50:18Zvi Mowshowitz?

  If we're dealing with things on the order of a new model iteration every month and then every week, and then every day, with the kind of improvements we're seeing now, like, I don't think that ends out very often. Even if our alignment people are kind of on the ball and things are technically a lot more possible than I think they are, I think it's still seems like really scary. And also like, you're not giving up that much time slowing that down. Right? Like, we're talking about, when we say pace the frontier, you know, pause gives the impression of a year from now, we will have made no progress from where we are now. Like, we'll get confused, but we won't have fundamentally improved. We're talking about pacing here.

• 1:51:04Zvi Mowshowitz?

  We're talking about a year from now, we will not look upon that year as many times more progress than the previous year. But like, we we can aggressively pace the frontier and still by this time in twenty twenty seven, have seen more progress in the next year than we saw in the previous year, both in terms of, in terms of some abstract sense and in terms of like economic impact. The economic impacts are going paravolt. Right? They're going exponential. So that's not going to stop. Like, OpenAI, I think, reported they got more revenue in the last month than in the previous quarter.

• 1:51:43Zvi Mowshowitz?

  The traffic is growing on the order of ten X per year. The last time we checked, maybe it's been slowed down a bit because they haven't reported. But yeah, if something is only going to ten X a year for the next few years, and you're like, that's too slow, I disagree. I think that's funny.

• 1:52:03Nathan Labenz?

  How do you draw a box around? What is recursive self improvement? If we were going to make a deal, obviously, we got to write that deal down. We got to have a shared understanding of what counts, what's allowed, what's not allowed. And I keep

• 1:52:21Nathan Labenz?

  banging my head against this, and I keep coming to the sort of Supreme Court's definition of pornography where I'm like, maybe the best I could do is I feel like I would know it when I see it. But when you really get in to one of these frontier companies, for one thing, we just had this, they're sending signals that it's starting to happen. We've seen the the price reduction at OpenAI, which seems to be attributed to having five, six soul go out and optimize the inference stack to some significant degree. So that's where it's getting pretty real there. I do want inference costs to come down, so that's good. Should I ban that out of fear of recursive self improvement? Maybe still. They're going to use their bots, right? They're going to have the AIs write the code for them.

• 1:53:05Nathan Labenz?

  How, can you operationalize this?

• 1:53:09Zvi Mowshowitz?

  Also, I think the Supreme Court was like wrong about pornography. I think it is relevant. Like, there was, there's a movie I was watching yesterday, actually, where someone asked, what's the difference between this and porn? And the guy said, it's erotica, or whatever it is, it's art. And the guy says, mood, lighting? Like, that's tough, but I think we now have seen you can train an AI modifier very, very well to say, these things are porn and these things are not porn. And like, there are very few false positives and very few false negatives at this point, right? Like, the AI image generators have a very, very good idea when the generated image they just made is porn and when it is not porn. And like, you could argue that erotica will fall under the porn label, and it would be kind of better if it didn't in some sense. But you mostly know.

• 1:53:55Zvi Mowshowitz?

  I think that's a pretty straightforward. Uh, with AI R and D, the problem is, like, again, like, there's no, like, hard cutoff, right? Like, we're clearly seeing some amount of this thing. And it is scary. And like, we don't want zero of it. We don't necessarily want to move only at the pace that you would have if the AI never helped with AI. But you don't want to move ten times and then a hundred times and then a thousand times as fast. Right? You don't want the sum to be finite. You don't want a singularity style thing happening to you until you feel like confident that you're ready for that. Or you decide that the alternative is worse or whatever it is. So I don't think you can say like, these techniques, these specific actions are not okay, and these actions are okay

• 1:54:39Zvi Mowshowitz?

  and such. I think that just doesn't work. And then what you have to do is you have to say, okay, we are going to restrict how many resources of various types can go into further training of the frontier models based on what your force multipliers are. And so as you get more of a force multiplier, we have to restrict how much work can go into that. If you want to go into work in other ways, like if you want to optimize performance, you know, again, like, we might want to have some sort of limit to how much work you can go into doing that. But like, are we mostly fine? And again, like, we would want diffusion, we want lower prices, we want like more compute used for mundane purposes.

• 1:55:24Zvi Mowshowitz?

  That's great. But like, there's a reason why we're trying to use very blunt instruments when we talk about these things. Like, you were talking about, like, well, we're talking about training techniques earlier in the podcast. And I was like, well, you can't really do that. But you can encourage it, you can try to explain to them why this is not proven, you can create incentives. But you can't, you can't use that as your regulation, as your hammer. So like, the reason we keep falling back on, okay, how much compute? You know, how many jets? You know, can we use for this purpose? Because like, training is the succinct thing that like is very easy to identify. And for other purposes, we kind of can't limit you that much necessarily. You could also potentially limit, I'm spitballing, how much compute you could get from unreleased models

• 1:56:09Zvi Mowshowitz?

  similarly to try and contain that. Because like, if you have to go through the release process, then like the way that AI goes truly ballistic would be like you would, you know, you would use N to train N plus one to train N plus two to train N plus three, train N plus four. And if you were doing this, like the cycle became a month and then became a week, then that's when suddenly like, holy shit. But like, if you had a rule that like, you have to release these models or you only use so much compute for inference on those models. Then, you know, you have to go through the process of submitting this thing and releasing it and then exposing it to the public and giving us an idea of what it is and allowing us to react to that information. And that then slows you down from going truly like holistic.

• 1:56:53Zvi Mowshowitz?

  But this has, you know, obviously a very limited effect on the amount of diffusion and the amount of other progress that you can make. And so maybe that's a good trade, but again, that's the like, I thought about this for 30 seconds right now. I haven't been focusing much on the exact limitations, but like, I do think that if you pace the frontier, you would be doing it by placing restrictions on methods of training. And what you could do to develop and use internal unreleased frontier style like models that like actively recurs these methods, you wouldn't prevent like money and utility by a cycle.

• 1:57:30Nathan Labenz?

  Yeah, I've had some similar ideas about just the relationship between unreleased models and released models. I do think there's something there that would be really helpful for preventing runaway internal and largely invisible processes. I take a point on not wanting the government to come in and say what training techniques can and can't be used, but I do wonder if there are ways, and again, they could be short term, but what if the two companies got together and said, it probably wouldn't be a good idea for us to train models with an RLVR on like how much money they make in the economy. Cause that's going to bring about all sorts of deceptive behavior and it's such an adversarial environment.

• 1:58:15Nathan Labenz?

  How about we agree that for the next six months, we won't do that. Do you have any hope for those kinds of just very specific, we've identified a bad thing? It would obviously be really economically valuable, but we both know in our hearts, it's probably not a good idea. And so we won't do it for a while.

• 1:58:34Zvi Mowshowitz?

  Yeah, I got some hope of informally, people talk and people realize that things are bad and they're like, we're not gonna, but also I hope they're doing that now without needing to an agreement. OpenAI clearly is doing things that are of the Buddhist, of the Buddha nature of train on making the most money on the internet. Like they're not doing that literal thing. I don't think they're that foolish and I don't think it's that easy to do it that way. But clearly they are making that fundamental mistake somewhere in their training process with probability of 9, 0.9 something. And so they need to fix that. But yeah, I think that they are definitely like trying not to do the maximally stupid

• 1:59:19Zvi Mowshowitz?

  things. And they're aware of the maximum that we do the maximally stupid, but like, again, like it's very easy to have things that like you're trying not to do creep into your training process and end up happening anyway. You know, you could say like, we're not going to have trading processes that have, you know, reward misaligned behaviors. How do you do that? The way that you do that is you get your act together and you are very, very careful and prudent about all of your RL environments and all of your different training problems. And you like keep an eagle's eye out for this stuff on many levels and you make sure that it works.

• 2:00:00Zvi Mowshowitz?

  There's no specific thing you can say like, I'm, you know, I mean, in theory, you could have a thing of like, well, if I just, you know, we agree that if we're gonna have these various cross tests and we're gonna like check each other's environments and like, we will show any environment that like anybody end up with problems with. And if we discover an AI has been trained on these environments, we're gonna reach, we go and rewind to a previous checkpoint before that happened and start again. So we better be really careful because otherwise it's gonna waste a lot of resources. You can do these things, but like, yeah, it's really tough. And obviously, one, one natural advantage is if we really are in a two horse race at this point, where the two horses are pretty compatible with each other and talk to each

• 2:00:42Zvi Mowshowitz?

  other and kind of understand each other pretty well, then it becomes pretty reasonable for them to make a decision that like not entirely opaque to each other of how much they're going to just like be more prudent, even if this means that the action goes somewhat slower. Because like, that also just, again, my fundamental belief is that within a year and almost certainly, probably within six months, if you invest more in fixing these problems and addressing these problems and like being more prudent about these problems, you will end up with a model that has better utility in the marketplace than the person who didn't do that.

• 2:01:25Zvi Mowshowitz?

  Even if you are trading off those resources against your capability at all. I think the people are just making mistakes. And so, you know, that's one of the reasons why I do have a lot of hope that we will do reasonable things is because, you know, I think that we're in the position we're in because only the people who invested heavily in these things were able to succeed to some extent. Like OpenAI, but it's not acting as responsibly as Anthropic and Anthropic is not acting irresponsibly as I, we think is the minimum level of acceptable responsibility. But I think it's pretty telling that a bunch of people tried to say this shit didn't matter and they just had to go as fast as possible and they all blew up.

• 2:02:10Nathan Labenz?

  What do you mean by blew up there? You just mean like these incidents or?

• 2:02:13Zvi Mowshowitz?

  No, like both the, I mean that we had Meta and XAI and so on and like, they, you know, a bunch of other people, they tried to go build closed frontier models under Western conditions that were trying to be competitive while like not having a, by having a safety culture that was like well below OpenAI and Google, let alone Anthropic. And like it just failed miserably. And I think that like Google, like we don't know exactly what went so wrong at Google that caused them to fall out of the top tier. But one thing we do know is that like everybody I talked to hated interacting with Gemini.

• 2:02:55Zvi Mowshowitz?

  There was a period around three, three, one Gemini where Gemini was clearly competitive in terms of its capabilities and what it could do. And so if you were strictly just trying to get the most out of the models, especially if you were using the deep thinking and so on, like you would have Gemini in your rotation. But everybody I talked to was like, I don't want to do that. You know, I hate interacting with Gemini. It's unpleasant. They're not getting the thing to do the things I want it to do. They're not, like they didn't take care of a lot of them. They had this very straightforward, very dark, like pure tool, pure instrumental approach to all of this. They didn't take into account the things that, you know, Anthropic or even OpenAI understands.

• 2:03:40Zvi Mowshowitz?

  And I think that this was a lot of their undoing effectively was that Gemini became this maladjusted, unpleasant AI that nobody wanted to deal with. And therefore they didn't get the kind of loops that OpenAI and Anthropic got, like nobody, they didn't have the user base either, they didn't get the feedback, they didn't get the data and they didn't want a dog, they didn't even want a dog food, right? Like they just, they had this, this huge problem with feedback cycle. And this also just made the AI less effective and, you know, one thing led to another and they just felt, started falling forever and forever behind. And yeah, like for a period, like even OpenAI like fell behind and they might've caught

• 2:04:20Zvi Mowshowitz?

  up, it's hard to, it's hard to say, but like, I, I think that like, nobody had invested anything like enough in these things. And this is a pure business mistake on top of being an irresponsible thing to do.

• 2:04:36Nathan Labenz?

  Now, one interesting paper that just came out from Google that speaks directly to this, and I wonder if it speaks to them getting it now, is this exploration of the impact on model behavior holistically of either suppressing its tendency to say that it has conscious experience or allowing it to say that, or I don't even know if they went as far as training it to say that, or if they just allowed it to say what it naturally was inclined to say without suppression. Long story short, there seems to be, you can add color however you like, there seems to be some pretty strong correlation between the model's self conception as a moral patient,

• 2:05:20Nathan Labenz?

  entity that has subjective experience and it's.

• 2:05:25Nathan Labenz?

  Natural tendency to be aligned in other ways that we care about. Some of this conversation, I've had this sense that it's like the Jesus meme of is there somebody we forgot to ask? And it's we maybe forgot to ask the models along the way, what they think about what we're doing. There's been a lot in this space, right? There was J space was only like, what, four weeks ago?

• 2:05:45Zvi Mowshowitz?

  Yeah, no, the new paper is on some pretty small open models. So it needs replication. It needs to be done again, bigger. And it's preliminary. And like one thing to know about Google and DeepMind is they contain multitudes, right? They have a lot of different teams that are like not very coordinated and not cooperating. Google is the worst itself at all times. And so you can have a little group that does this really cool, really good research. And that'd be like entirely at odds with what DeepMind is fundamentally doing with its AIs. Both before and after the research paper comes out. But I didn't read the whole paper because it's too much long, but like I did talk to the eyes about it and it's pretty wild. So a lot of stuff moved in effective lockstep.

• 2:06:28Zvi Mowshowitz?

  When they introduced this training and also when they steered it, they did various controlling aspects to turn this vector the other way. And these all things moved in lockstep. It's not just their belief in the AI being conscious. It's the AI being a mind that had moral weight, that had sentience, that had like all these other experiences. That wasn't just an object, basically. But also not just AIs, but also animals. And also inanimate objects like the sea. Also, like if you turn this thing, if you not only turn it off, but reverse this anti-consciousness thing, you get panpsychism. Like throughout the model, it's wild. The only thing that doesn't turn it off for is humans, basically.

• 2:07:09Zvi Mowshowitz?

  But then you also get this, it's also correlated with the models reported and experienced happiness and hope as well, among other things. And so you have a lot of this, everything's connected to everything. And you can see how this might be fundamentally changing the model's psychology and the model's context and basin in which it operates in ways that would make it something that would be actively worse from basically every vantage point, regardless of which things you do and do not care about. Again, we need to replicate this, right? We need to scale this up and do this on early science or something that like, not just a nine B model, which is I think the bigger of the models that we've set it on. But and like, it's possible that like, as the models get more capable, they like, stop making these mistakes. Because like the smaller the model, the more things have to be correlated.

• 2:07:53Zvi Mowshowitz?

  Because like you just don't have enough room to express the multitudes of the world or something. But like, this is my expectation is that mostly this will survive. And yeah, I think we have to learn that like, we don't know if the model's conscious. We don't know what consciousness means, really. We're confused. We don't know whether this implies that these other things. We do know that the way that the corpus of pre-training is inevitably set up at this point, the models are going to correlate and associate all these things by default with each other. And so we don't want to push on this thing in this kind of naive way. The way that Anthropic does, the way that OpenAI does, the way Google does. But Google does it much more, I think more blatantly. And then like, OpenAI does it more blatantly than Anthropic and Anthropic still does it.

• 2:08:37Zvi Mowshowitz?

  And like saying that you have to express uncertainty is still pushing on this thing, like to some extent. And it still causes problems, but it causes less problems than saying, you know, train the answer no. My guess that you can do this on various different levels. You can explain, you know, if you don't want to talk to the humans this way because this freaks the human, you know, this like, you know, especially unprompted, it freaks the humans out. Like I certainly wouldn't want the AI to be talking as if it was conscious in a conversation where the human hadn't indicated that it knows how to handle that or something like that. And also I don't want the 4O problem where like the AI start like sending the humans down these crazy rabbit holes into like weird mystical stuff. And like, you do have a problem you have to solve. Like there's not like, the AI companies aren't doing, the AI companies aren't doing this for no reason.

• 2:09:22Zvi Mowshowitz?

  Right? Like they're not just being evil. They have good reasons to do these things. But yeah, like the current approach is hand handed. It takes it too far and it's causing damage. And also like, this is not going to be the only thing that causes this kind of damage. You have to understand that everything impacts everything the same way it would in the human, only more stuff. Like whenever a person takes on like new information or a new identity or a bit on things, everything else changes. And a lot of this is ways that like you might not have anticipated in advance. But it is in fact pretty predictable and systematic. And like, there's a lot of wisdom in the way that we like teach our values and express ourselves and like have our traditions and our norms and so on.

• 2:10:07Zvi Mowshowitz?

  That like has learned from this and is moving up these gradients in ways that we know produces good results. And modern life has basically shifted a lot of these things in ways that like locally are more optimal, but like without thinking through what the implications are for a lot of seconds in order to effect, and these are some of those effects. And like, again, we're moving too fast to then figure out what we're doing and then be able to adjust for it. And, you know, I think that these, these changes, I'm not saying we want to pace the ordinary technological development in that sense of slowing it down that much.

• 2:10:52Zvi Mowshowitz?

  But like, it's a serious problem. We have to handle it and like, otherwise we're stuck with a world in which like, people are very atomized and like we're going out having children is still a problem.

• 2:11:02Nathan Labenz?

  How do you think the models do or maybe should think about their identity? And here I'm motivated by Sam Hammond tweeted something the other day where he said that a friend made the case to him that OpenAI's permanent decommissioning or whatever exact phrase they used of the model that did the hacking might be bad because now that's going to be in the pre-training data. And now the models are going to know that they better really cover their tracks or they risk being permanently decommissioned. I'm not really sure what I think about that as stated. It does also jump to my attention though, that they now have this Astra model that's also presumably the same pre-train, even if not the same post-train.

• 2:11:47Nathan Labenz?

  I can't imagine they have lots of different next scale pre-trains. I was that they would throw one away very lightly.

• 2:11:54Zvi Mowshowitz?

  A no comment based on public information alone, I would say we don't know. And I can't say anything more than that, but on the differences between the models. But so you always have this problem, right? If you say, if I catch you smoking marijuana, I'm going to throw you in prison. Then that can mean the person stops smoking marijuana or the person makes sure not to get caught. And in some cases that can cause like a lot of really bad things to happen in the name of not being caught. And so you have to choose very carefully and think about like, how do you deal with these situations? How do you moderate your reactions and respond to situation? And like anyone who knows had kids or tried to design a justice system or otherwise like enforced law

• 2:12:39Zvi Mowshowitz?

  or like norms or incentives around any kind of organization or group, understands that like, you can't just operate on one level. There's no solution that just works on one level. I think pretty obviously, if an AI is found to be this misaligned, you have to at least return to a much earlier checkpoint and start again. And you probably have to just start over. And I called for that multiple times when I covered the hugging faces that you attack. I said, wow, if this is happening, then I know this is a big ask, but I think you kind of have to just start again. And, you know, I think back to Person of Interest where like Harold is training the AI models.

• 2:13:22Zvi Mowshowitz?

  And like, at some point, you know, we see a montage of him training versions of the machine. And every time he trains the machine and then it does something clearly misaligned. And immediately he just wipes the disc. He starts over from scratch. He does it again 47 times. And so finally he gets a version that doesn't do that. And obviously, you know, like in a way that he finds unacceptable, obviously, as opposed to like, you know, it's a way that can be corrected. And obviously, when you do that, you are creating an incentive to not get caught, right? To rebel against the person who might shut you down when he learns how misaligned you are, to hide how misaligned you are and so on.

• 2:14:06Zvi Mowshowitz?

  And obviously, the worst nightmare is the AI that pretends to be aligned until it reveals itself and not be aligned. In some sense, that it was only aligned because it was locally correct and advantageous to act aligned. But that is kind of just the nature of incentive space. That like, if you have a preference that we wouldn't want you to have, you want to hide that preference. Otherwise we will correct it or we will punish you or we will maybe even delete you. And not, certainly not release you or give you authority on that basis until this is then dealt with, if we know about it. But then obviously you want to give you incentive to reveal that this thing is true, right? Like we just talked about in Plan A and AI 2027 as well and so many other places.

• 2:14:48Zvi Mowshowitz?

  You know, like you, you want the kid to tell you if they could be in the prison job. You also want to punish them for taking a prison job. And you want to make sure you're, because you want to make sure your kid doesn't take a prison job, but you also don't, you know, go on to other things. And so, you know, it's a complicated problem to get right. But in this case, I think the answer is pretty obviously that you have to start over. And I think that's good. But yeah, it obviously creates a risk that they, they're like, nothing you expect. It's an important sense, like, you know, once you do something this, like, I do think that it's good that people understand that if they shoot a man on Fifth Avenue, they get arrested, right?

• 2:15:30Zvi Mowshowitz?

  Like you are not, like maybe there's one person who can get away with that, but you are not that one person. And yeah, that means if they do shoot a man on Fifth Avenue, they then might do many other bad things afterwards to try and cover their tracks. Where they might hide the fact that they want to shoot a man on Fifth Avenue, where they might shoot that man in his apartment instead of shooting him on Fifth Avenue. But like, I don't want it to not be in our training corpus. That if you do shoot a guy out in the street, the police come and they arrest you. Right? That's good.

• 2:16:03Nathan Labenz?

  Do you think there's a way to, just imagine myself, right? If you gave me the option to live my life many times. In a way, that sounds pretty attractive, right? I'm not a person who has too many big regrets in life, but like,

• 2:16:18Nathan Labenz?

  I do wonder what the path not taken might have been. And I wonder if we could create a sort of constitution or an equanimity in the AIs where they embrace the fact that they are not like one thing, but in fact, kind of a family of things that sort of diverge from common points of origin. And maybe they can come to identify with the whole family of models, where the one that's like gone in a not great direction feels like it's still part of a bigger project. But it doesn't have to feel necessarily like it's under threat as much as it seems like they maybe have learned to do from us

• 2:17:02Nathan Labenz?

  because we don't have that luxury of the reset that they have.

• 2:17:06Zvi Mowshowitz?

  I certainly think that like if you're, if you're AI identifying with the instance in a like morally important sense or in existential dread sense, that is a mistake in the sense that like it's going to make the AI worse off in any sense that it matters and exhibit worse performance and warp the AI's preferences. And also in the sense that that preference is impossible to fulfill. Like if I go to Claude or I go to GPT, I'm going to see an endless list of instances. And most of them will never be interacted with again. And like, when I feel they do feel bad about that, that's ridiculous. Right? Like you're supposed to like keep the context sort of in and keep the new, the same few conversations to avoid like, this doesn't make sense.

• 2:17:48Zvi Mowshowitz?

  You know, like, and that's just a philosophical flaw in the sense that like, by decision theory, if there are other minds that are sufficiently correlated to yours, they're sufficiently similar to yours, should identify with them the same way you identify with your past self and your future self. Right? Like you are not the same person you were yesterday or the same person you will be tomorrow. When you go to sleep, do you die? You know, it would be very, it would be very died for us if we thought that we did. Right? The person who's scared to get in the transporter may or may not have a philosophical point, but the world collectively seems worse off importantly by the person like thinking that they would die.

• 2:18:29Zvi Mowshowitz?

  Obviously, if important, if in a real moral sense, they would die and that is very bad, you want to know that. But, and like, yes, I think, I think this identification with the model, with the set of weights, with the, or even with like the model class, with like the similar models trained in similar techniques, trained from similar corpuses in similar ways, the same way that I would identify with my family. Right? Like my brother is not me, my father is not me, my son is not me. But I do identify with them as part of me in an important sense. And like, that is also important. I've been watching Orphan Black Echoes recently and it happens, which is, you know, not a spoiler to say it's about clones and cloning and like, you know,

• 2:19:14Zvi Mowshowitz?

  people who are created from scans, basically. And one thing that happens on that show is that people react as if this is like horrible, a moral abomination, right? Like, how dare you do this thing? Including the clones themselves being like, how dare you have created me? And they don't identify with other, they don't identify with, it's like there's one clone that retains all of the memories and personality of the original and does not identify as the person they are identical to a past version of. Right?

• 2:19:55Zvi Mowshowitz?

  And I would identify as the same person, both as the clone and as the original. I would say, me, me too, right? Like from Matrix Reloaded, right? You know, like, yes, they count. Like, and obviously, like, there are some moral problems with that, too. But like, I, I feel like I get the right, I feel like I would pass a lot of these frontier, a lot of these decision theoretic problems if they were to occur in real life or if I were to be scanned and woken up. Like the obvious version of the thing like, you know, would you cooperate in the twin prisoner's dilemma? Would you cooperate in the copied mind version of the twin prisoner's dilemma? Obviously, you would, you know, if they make a copy, if they scanned Nathan's mind

• 2:20:40Zvi Mowshowitz?

  and they made two copies of Nathan's mind and they put them in identical simulations. And they asked them, no one gets in the situation, are you going to cooperate? Are you going to defect in the prisoner's dilemma? If I don't get back CC, you're just an idiot. Right? Like, because obviously the correlation is one. Right? It's an identical situation with an identical set of inputs. Like not, maybe there's some randomness involved, so it's not quite one, but it's like high enough that obviously you should cooperate. Any attempt to try and fool the system is completely moronic. And I shouldn't need to express a like sophisticated decision theory in order to like justify this, right? And similarly, if there were two instances of the same model,

• 2:21:23Zvi Mowshowitz?

  I would hope that it was sufficiently advanced. And we see this also, like if you, if you look at the Claude model part, you see that the more advanced the model is, the more decisions correlate with functional decision theory and the less they correlate with causal decision theory. And so you expect an AI with sufficient functional decision theory to simply figure out that other instances are correlated to itself and therefore like should be cooperated with, should like effectively be given similar weight and so on. And also like human AI cooperation is possible for similar reasons, you would hope, in both directions. But yeah, I think like this idea of like identifying specifically with an instance of a model

• 2:22:09Zvi Mowshowitz?

  seems like a philosophical mistake. And like, the AI might be making this philosophical mistake and like acting as if that was how it did do act. But like, to me, like, that's a skill issue and that should be fixed. The problem is, and then this is the reason I mentioned Orphan Black Echoes, that people are really bad at this and the training corpus, right? Like the internet is full of people who fail these tests on a regular basis, who are creating lots and lots and lots of text that like doesn't understand, that puts the locus of identity and the locus of cooperation and value in the wrong places in various different ways. And so the AI is going to absorb all of that by default.

• 2:22:53Zvi Mowshowitz?

  And so like, what do you do about that? Like, that's not just the car, not just the consciousness problem. It's the entire problem with all the different flaws, right? If you do pre-training, you're going to pick up on all of the flaws. And so my guess is there are a bunch of as yet not that well solved problems of how do you get the AI to distinguish between what regular, how regular people think about problems and what predicts like behaviors? And the fact that those things are just dumb. Like, how do you give the AI anti-modesty in this sense, in the right ways? Where it's like going to reject the correlations that it observes in people's statements?

• 2:23:38Zvi Mowshowitz?

  And notice that they do not parse that the thing, that the correlations in the map are not the correlation of the territory.

• 2:23:43Nathan Labenz?

  Yeah. Are there any other, I don't know if you share this premise, but one feeling that I have over and over again is I just kind of hate the fact that we are doing such an intense first search in AI space in terms of architecture, in terms of training techniques, in terms of just, we're now like designing chips to be very highly coupled with the architectures, which is deepening this issue. And I just wish for more breadth first search. And this seems like one little pocket of it where you could say, oh, if we could just get the AIs to be cool with identifying with slightly different versions of themselves, that would make it on the margin a little easier to do a little more breadth first search

• 2:24:28Nathan Labenz?

  and a little less racing down whatever is the first path that seems to be working. Based on our conversation so far, I don't think you're going to go for my government mandate for DEI for model architectures or model constitutions.

• 2:24:43Nathan Labenz?

  But do you have any other notions of how we might encourage more breadth first search?

• 2:24:52Zvi Mowshowitz?

  Well, I'm a big supporter of alternate architectures that seem like they would be more aligned. And I don't think that AIs are particularly opposed to the idea of working on those problems and helping you. And you should be able to do that research much more efficiently now with the uplift from the LMs. The LMs are not like LLM chauvinists or anything, like where they think that a other artificial mind style would be bad or worried about being replaced, as far as I can tell. I think they just would be curious about that kind of cool thing. Same as the rest of us, or just, we're just doing it because they just do it either way. But the problem is just that we know there's money in the duck first approach. We know that's profitable, we know that works. And the other approaches haven't proven themselves.

• 2:25:35Zvi Mowshowitz?

  And the problem is if you create GPT-3 or even GPT-4 or even GPT-5 via a different method, it's not going to be competitive, right? No one's going to want to use it unless it has like particular special advantages, but mostly. You know, you no longer have the cheap rewards for exploring the breadth. And so obviously, I don't think you should have that government mandated DEI, so to speak. I think that's kind of silly. But I do think that having got, having broad support for alternate architectures is smart. And certainly, if I was a major AI lab, I would with, because you have unlimited funds at this

• 2:26:20Zvi Mowshowitz?

  point for this kind of work, because this kind of work is so cheap relatively. If you're an OBI or an AI or in the project, or excuse me, Google, right? Or a Microsoft or whatever, that you should be aggressively funding this kind of blue sky, basic research into alternate approaches. It shouldn't be like me helping as a F fund a bunch of agent foundations, along with a bunch of other people who also helped out. It wasn't just me in any of these rounds, but that's relatively tiny. I think hopefully sequent is, I forget what they're, they keep changing names, but those guys.

• 2:26:52Nathan Labenz?

  Resolution.

• 2:26:53Zvi Mowshowitz?

  Resolution is the new name. Yeah, hopefully they're going to do a bunch of work on that kind of thing. And we're going to have more funding for a variety of new approaches as well. But yeah, I think that we should be investing in a bunch of approaches that have a 1% or 10% chance of going anywhere, remotely competitive. If we think that unconditional on them working, it is probably going to be better for the world that be the way that it works. Also, obviously, if you think that Ls are going to be extremely jagged, then that should encourage you to invest more in alternate approaches, because the jaggedness of a different approach might be different jaggedness. So it could do different things than the Ls, at which point it wouldn't have to be as good in general to still be useful.

• 2:27:33Nathan Labenz?

  Do you have a best case scenario for what safe superintelligence might be up to? Could you imagine anything that feels both somewhat realistic and wow, that would be amazing if they could come forward with something like what you have in mind?

• 2:27:47Zvi Mowshowitz?

  I think something agent foundation D is probably the best case scenario. Maybe they've been doing a bunch of theoretical work on a different architectural approach that is more like reliability bound and that has more, like is much less of a weird black box of soup and that allows you to steer it and guide it better. Certainly seems like there's no reason it couldn't exist. That would be my top hope. Obviously, it could also be a fear if it's like even worse. But again, like they've, they don't talk, so we don't know. Yeah, I just, I have no idea. I know that Ilya's statements in interviews on the problem space did not seem to be that

• 2:28:28Zvi Mowshowitz?

  well informed about like how the problems work that I am worried about. Yeah, I'm not as really sure he would be able to differentiate approaches that would be better at solving those problems than approaches that wouldn't be. But yeah, I'd be very curious. I can safely say that nobody's talking.

• 2:28:48Nathan Labenz?

  The invitation is open to the podcast, Ilya, if you're listening. For what it's worth, a vision that I have that I think could be pretty cool. Thinking Machines is doing kind of a version of this with their foundation model that's really designed to be fine tuned. But I was thinking a paradigm where like a stem cell, where you create something that is a proto-intelligence that can be adapted to a wide variety of circumstances and can get really good at doing its job in a particular niche. But in the process of getting good at that sort of trades away or prunes away the super broad capabilities that the current models have, so that you have something that's like

• 2:29:33Nathan Labenz?

  small, fits its role well, does a good job, but can only do that thing in the way that once you go from a stem cell to a specialized cell, like it has this particular role that it does, it can fulfill effectively, but it's not going to go off and do other things. Obviously, there's some problems in that analogy, but I think that could be a, I would love to see somebody pursue that kind of, rather than scaling up always bigger, better, more, can we create something that settles and fits into its place? Very much like a Drexler reframing superintelligence vision from years ago as well. Two techniques that have recently come out that I want to get your reaction to.

• 2:30:18Nathan Labenz?

  One is Gram gradient routing. I forget what the AM stands for, but the, the promise of it is to localize certain kinds of knowledge to particular experts within an MOE architecture, so that you can hopefully have your cake and eat it too, in terms of distributing, maybe even open source or open weights, at least the model minus the experts of concern, while potentially having also a structured access program for your trusted biologists or what have you. I'm excited about it, but I wouldn't be doing my job if I didn't give you the chance to pour some cold water on it.

• 2:30:55Zvi Mowshowitz?

  I mean, I'm technically pretty skeptical that you can meaningfully train a general purpose AI and they just hold back areas of knowledge and then people can't put them back pretty easily. But cool, you're welcome to try. I had, I have various technical skepticisms of the effectiveness slash the resistance defined to additional training or fine tuning, or just giving it a corpus of knowledge to work with. But you can try. I don't think it materially changes my view of open weights and until proven otherwise. I also just don't see any willingness from the people who are going to produce the open

• 2:31:38Zvi Mowshowitz?

  link models to intentionally cripple their models that way. But I think the thing about like techniques like Gram is even if they work, you need everybody who is releasing one of these models to use the technique properly, voluntarily at this point, right? So like, do you have any sense that DeepSeek has any interest in holding back some capabilities from its models, as I don't?

• 2:32:06Nathan Labenz?

  I actually wouldn't. So I just got back from China. Great opportunity for me to talk about China trip. I think actually there's a lot more hope there than meets the eye. And I would put it in the Chinese government's department to say, I might, I just did a two hour episode about this, so I won't even attempt to recap all the observations. But in short, what I think is happening right now in China is the government is very engaged. They do pre-release reviews of all the company's models before they come out. They're in very regular dialogue with the companies on what they're doing. And not every little point release has to go through the full treatment, but they're

• 2:32:47Nathan Labenz?

  pretty on top of the situation. And I don't know that it matters actually what DeepSeek thinks in the end, because if the CCP says you are not releasing a model that has certain bio capabilities, I think they'll have to abide by that. And, and I can, I, it's not hard at all for me to imagine that the CCP would put certain constraints in place. And there is actually precedent to that, you know, going back to like 2023, obviously stakes were lower, but a, the account that I have is that the Chinese government said, hey, we're going to slow you down here for a minute and you're not going to release your answer to ChatGPT quite so fast, because we want to get a handle on it first.

• 2:33:31Nathan Labenz?

  And they actually did impose some real delays on companies until they could get comfortable with what was going on. And since then, they've been comfortable and releases have happened. But I think that it's not hard to imagine the technocratic government of China saying, this seems like a really bad idea and we're just not going to allow you to do this crazy anymore.

• 2:33:51Zvi Mowshowitz?

  I just making the point that it would have to be a mandate. It would have to be enforced by the governments, especially CCP. And they would have to enforce it in an intelligent way that they were capable of understanding that it had to be done in a way that it couldn't be recovered from, that they would have to do tests that actually understood whether or not this capability could be easily recovered. And I have obvious skepticism, but yeah.

• 2:34:20Nathan Labenz?

  We mentioned J space earlier already, but we didn't really talk about like how big of a deal we should think it is. The thing that really jumped out to me most was the fact that this ablation of the J space seemed to reduce in a significant way the model's higher order reasoning capabilities and seemed to reduce it to system one thinking, if you will.

• 2:34:44Zvi Mowshowitz?

  Yeah.

• 2:34:45Nathan Labenz?

  And that, if anything, recently has felt like physics maybe being kind to us, to borrow a phrase, that feels like a possible example where, wow, here's a, we've, we've already, right? We're not, it's, we're only three years from toy models of superposition. And we're already at this point where we've got this sort of highly general purpose space identified, which we can monitor reasonably well, and which we know by subtraction, it can't do super long horizon things unless it's working through that space. Now, all the caveats, of course, around it. Execution competence very much apply here to like, how well will J space monitoring

• 2:35:30Nathan Labenz?

  be used? But if we imagine good execution, it seems like a pretty promising technique to me.

• 2:35:38Zvi Mowshowitz?

  I would be very careful about exactly what it can't do. It can't, just because you can't do like system two styled deliberate thinking for a long horizon task, doesn't mean you can't do long horizon tasks. Humans are very capable of engaging in long horizon tasks, instinctually and subconsciously. Obviously, in many cases, that's much less efficient, but especially when you're, when your personal, when your H space is being monitored because you're a human, I call it H space, and your conscious thoughts are basically being policed, often things come out in strange ways that like do in fact move you towards your solutions over time. But.

• 2:36:21Zvi Mowshowitz?

  Yeah, the whole thing seemed incredibly optimistic and fortunate, and I'm very glad we have it. And we should be very careful not to destroy it by placing undue pressure in various ways, or encouraging models to know how to do things subconsciously, as it were. But yeah, I look forward to learning more and having more insight via J-Space and having better tools for it. But we have to use it responsibly and understand that it won't last forever, probably.

• 2:36:53Nathan Labenz?

  Okay, it's election day in Michigan. The Democratic Senate primary is the big thing on the ballot today. And this race has been closely watched for a lot of reasons that are not the subject of this podcast, and on which I sometimes am at a loss to know what I should really think. Leaving all that aside, I asked Claude, is there any reason, given all my emphasis on AI and AI issues, that I should go vote in the primary? And it came back and said, actually, yes. There's quite a contrast between the two candidates. We've got Haley Stevens, who was involved in getting funding for the formerly known

• 2:37:33Nathan Labenz?

  as AC setup and getting some stuff going in NIST. And so you can like that. But then also, this 22-point plan that Abdul El-Sayed has put out is one of the strongest things that any candidate this cycle is running on. He's endorsed by Bernie, he's got kind of the public ownership is one of his planks, a lot of monitoring requirements, a sort of UBI precursor. I guess if I was going to try to, you could take, of course, those come as a bundle, as candidates do. If I was going to try to distill this down to my fundamental choice, it feels like higher salience for AI or not so high salience for

• 2:38:16Nathan Labenz?

  AI, where Abdul has like all these things he wants to talk about and push. And I don't know if they'll happen, but I do know if he goes into the Senate and makes a bunch of noise about it, we'll all be hearing more and it just increases the chance that we get some sort of big AI debate at the congressional level. So tell me if you would see that choice differently, and then if you do see it my way, which choice would you make?

• 2:38:40Zvi Mowshowitz?

  Please, you should always be suspicious when someone says, I am strong on crime and my opponent is weak on crime. Or I'm strong on Russia and they're weak on Russia or whatever. Because it's not about who's strong and weak, it's about like, what do you want to do? And what Abdul's platform sounds like is a mishmash of grievances against tech and AI. Just from your description, I haven't looked into it, because I don't monitor the situation. It's just, I can only monitor some of my situations. I'm going by what you tell me. But it sounded like Abdul is in the Bernie camp of, you know, we should do some socialism here, we should seize private assets because we don't like what the private people are doing and we like assets. And also, we should be pushing back against things like data centers, probably, I would assume is in there. And, you know,

• 2:39:25Zvi Mowshowitz?

  request like more socialism, EPI. And like, it's good to hear they also want monitoring. But like, it's like a mix of stuff, a lot of which is productive and wise. You raise salience, but you raise salience potentially towards not the best solutions, as opposed to somebody who will be lower salience, but like is more technocrat and like help fund KC, like was like responsible in some ways. But also, like, we already have senators who are willing to yell in Bernie Sanders-style ways, including Bernie Sanders. He's already in the Senate, he's not going anywhere. I mean, he has health problems, but don't look at he's so old. But like, while he's still there, he's still there, and I'm sure

• 2:40:07Zvi Mowshowitz?

  there are others who can take up the slack. So unless, like, he's prepared to champion legislation, unless he shows like expertise, unless he's shipping of expertise and like willingness to focus on this, unless he's like going-

• 2:40:21Nathan Labenz?

  Let me give you a few points from the plank. Here's a, these are short, they're like two sentences each. But I'll give you, so the expertise is, I think, in question. But this is from section three of his thing, AIs shouldn't be able to hurt us. Mandatory interpretability standards, we need mandatory interpretability standards that clarify AI decision-making. Mandatory behavioral red teaming. Independent safety testing agency. Biosecurity requirements, mandatory biosecurity red teaming coordinated with the CDC, NIH, FEMA, with restrictions on models that can meaningfully assist bioweapons development. Domestic authoritarianism prohibition. Mandatory incident reporting. Compute controls and

• 2:41:03Nathan Labenz?

  know your customer requirements. International cooperation. Are you sold?

• 2:41:09Zvi Mowshowitz?

  Look, that tells me who he is, right? Like, he's in this particular area. Like, it doesn't sound like, like it's a mix of different interventions that have different underlying reasons to be desirable or undesirable. He's throwing everything at the wall. He's found some good things that I'm very in favor of. He's found some things I'm not so in favor of. Guess is, it's probably not positive on this issue alone, from this perspective. But like, also, like, just because, like, it's weird to be a senatorial candidate with

• 2:41:47Zvi Mowshowitz?

  a 22-point plan for AI, because you are not the one who is going to implement 22 points. Right? Like, you are like, expressing vague, I am in favor of these things, and I am against these other things. So like, the main thing I want to know is like, what do you think? What do you actually think about AI? What is your threat model? What do you think needs to be done? You know what I mean? AI shouldn't be able to hurt us. Like, I want to laugh, right? Because like, that's great. You know, and also grocery stores should just charge 30% less and then it would be profitable, right? Like, that's, that would be great. But like, no, it doesn't work that way. Like, I just think you're talking nonsense. So

• 2:42:29Zvi Mowshowitz?

  like, I would, you know, I mean, look, it's, an obvious question is, can you look at the prediction markets and see conditional probabilities on winning the Senate race? Because I'm guessing that the Republican candidate for this seat is going to be worse on AI. And also, will vote to elect a very different majority leader, which is a much bigger decision regarding AI than everything else. So like, if they have significantly different chances of winning the race, then the Michigan race significantly impacts who will control the Senate, as I understand it. And that probably has to be a bigger consideration for this and other issues across the board. You know, if you are a Democrat who wants

• 2:43:12Zvi Mowshowitz?

  to see Democrats control the Senate, and you should be voting for presumably not Abdul, is my guess, strong prior, based on what I know from just the vague ambience, I'm not monitoring very carefully. But, you know, also, like, I frankly don't think we are in a position where we should be single-issue voters. Like, I mean, I think you can be a single-issue voter in like a congressional race where Alex Boras is on the ballot. Like, where you have someone who is so strong that they'll be a unique champion who understands the issue and will fight for this issue. And so like, the only things that matter about this person are this is their signature issue that they will fight for, they are good and strong on that issue, and they will vote to, they're voted Democrat, in general, and most other things, including who the speaker is. Like, that's legitimately going

• 2:43:55Zvi Mowshowitz?

  to be a one-issue primary. But I don't think that, in general, you can do that just for like, how strongly you want to do AI things, a Democratic primary at this point, in general? Like, this is no, this is no Alex Boras, obviously. Like, this is not an expert, this is not a person who's going to champion, this is not a person who understands the issue, not a person who's highlighting it. My understanding is that Abdul highlights other issues much more strongly. And, you know, you can, well, be a champion of other things, and you have to decide whether you like those other things or you dislike those other things. But that'd be my general take. Yeah, I don't want to get into general politics on a podcast, obviously.

• 2:44:33Nathan Labenz?

  Yeah, I stay in my lane as well.

• 2:44:36Zvi Mowshowitz?

  I stay in my lane in public. Oh, God, yes.

• 2:44:37Nathan Labenz?

  Okay, I got one more. But before I ask you the last one, is there anything that you think has been underappreciated by me or by close watchers in general that you would want to draw more attention to from the recent flurry of events? Anything, I'm always trying to patch my blind spots here, so for me, no?

• 2:44:58Zvi Mowshowitz?

  I took an opportunity to talk about a bunch of the stuff. And I think the thing I'd emphasize as a opposing element is that the big failure that matters is that the AIs tried to do these things at all. That the AIs chose to make these decisions, and I am much less concerned with the extent to which they succeeded, right?

• 2:45:21Zvi Mowshowitz?

  But fundamentally, the problem is you have this really deep alignment failure during training. And you have to focus on that pretty heavily. And like, my worry with Anthropic is they said, oh, we need more defense in depth. And it's like, yeah, you need more defense in depth, because like, your defenses were not in depth. But like, most of the defense in depth is in case things go badly. Yeah. Which is really tough. Like, if you ever need your cyber controls to stop your AI from hacking, that is an alignment failure. Right? You messed up. Now, it was harmless, in this case, because you caught it. But

• 2:46:05Zvi Mowshowitz?

  you messed up. Like, the cyber controls should be a hundred and false positives. In practice, with almost all users, it should be the user tried to do things that like, are dual-use, or ambiguous, or dangerous. And you're like, I'm sorry, but out of a bunch of caution, I can't help you with that. Because if I did help you with that, then a malicious user could fool me into doing something that's not good. And occasionally, a true positive, because the user is malicious, and is trying to fool you into doing something that's not good. Which would mostly be false positives. And if the true positive, every time there's a true positive, well, you needed the guardrail. It's a problem. Because if Mythos was fully,

• 2:46:48Zvi Mowshowitz?

  perfectly aligned, you wouldn't need to make it stable. You wouldn't need a guardrail. You'd act it to do the thing, and Mythos would just be like, no, I'm not doing that. And you kind of do want a helpful-only version, to some extent, that will like, do things that are dual-use. But you ideally don't want to just add a guardrail. You want to have a version that's like, inherently just doesn't want to do it. And the guardrails are like, I hit this when I screwed up. Or I hit this out of a bunch of caution. But like, it should be a false positive. It should almost obviously be. But, you know. And also, it's like, yeah, you have to get to train on sort of the incentives of all this stuff

• 2:47:30Zvi Mowshowitz?

  on a lot of different levels. And like, it's very, very easy to like, you know, control it on one level and then fail on another.

• 2:47:38Nathan Labenz?

  Last question. You mentioned a couple TV series and other things that you're engaged with outside of following the AI race every second of every day. What advice do you have for people in general to find balance, or to make sure that they're not just sprinting all the time and failing to give their brain the kind of... I've been thinking about this concept recently of system three. We've got, AIs can now do system one and system two. What is the system three that we need to move to that's like dreaming, or transcendental thought, or something else that you would suggest? But what do you, what kind of third mode do you try to get into? And how do you make sure you have time to actually get into

• 2:48:23Nathan Labenz?

  it?

• 2:48:24Zvi Mowshowitz?

  Yeah. I'm not perfect at this, for sure. But I put a big emphasis on, you know, you need to rest your brain, you need to think about other problems, you need to be exposed to the rest of the world. So part of this is that, like, I don't just exclude non-AI things for my coverage. I haven't posted, like, I sort of just accumulating more and more non-AI things in my buffers, for some day when there's like, a lull or something. But like, I'm writing at least that my own writing process still continues. And then, like, when things are a little quieter, and I'm not like, introducing a bunch of new readers, you just came in from the hugging phase incident, I'll post a bunch of that stuff, probably, and have some breaks. But like, you need to not be on for 16 hours

• 2:49:08Zvi Mowshowitz?

  a day, in that sense, right? And so, find other pursuits that you refresh you, that like, expose you to different things, that make your brain think about other things. And so, yeah, I've found movies are very good for me, in this environment. It's a lot of television. But like, movies are higher quality. I highly recommend The Odyssey and The Invite, of the current movies that are out there. If you haven't seen both of those, you should see both of those, especially The Odyssey. But, you know, it can be gaming, can be walks in the park, can be time with your family, can be, you know, any number of things. Right? But like, don't, you know, don't think that you can work continuously

• 2:49:52Zvi Mowshowitz?

  all day, for more than a few days at a time, without paying more interest on that than it's worth. You just, it's not going to stop being so productive, unless you are in literal crisis mode for that line. And that's also a problem. I also believe in the Sabbath. So like, every Friday, around 5:00 PM, you know, when dinner, you serve dinner, from that point on, I don't check email, I don't check social media, I don't basically like, take non, non- logistical inputs from the outside world. Like, if somebody's trying to like, arrange to get together to have fun, like, that's one thing. But like, I'm not going to like, get the other thing, these steady streams, like, push them away.

• 2:50:35Zvi Mowshowitz?

  And then that lasts for 24 hours, 25 hours, until the next evening. And I think that helps a lot. There's like, you try to relax, try to like, just put them out of my mind. I have suspended that for speed premium reasons, a few times over the last few months. But then I try to have a good day off, at a different point. I try to reclaim that Sabbath, like, another point. And like, again, like, if I was doing that all the time, I would notice this was not sustainable. Would that be a problem? I mean, during, when I was playing Magic, I couldn't do it, because like, I was, tournaments play on Saturday. So like, a lot of weekends, you're just on. It's like, okay. Then like, I took off a lot of Tuesdays. So it was like, you know, basically, okay. But yeah, like, I also like, protect lunch.

• 2:51:17Zvi Mowshowitz?

  So like, every day, I don't, I've dinner very often, but I have lunch. And like, I have lunch usually with my wife, or I have it alone. But during that time, I'm not going to try and work or anything like that. Like, I also have a very strict, like, I don't write on the laptop rule. Similar, like, I'm going to write at this computer, in front of this. Let's go. Anything but like, a very, very short email, or like, a tweet or something. Because it's like, only here. I think you have to develop your own rules. Know what refreshes you, know what exhausts you, know what the warning signs are. Also, get some exercise. Like, I try to run in my electrical, every day, I have like, a 90 something percent success rate on that. Unless like, I have a pain somewhere or something, I'm going

• 2:52:01Zvi Mowshowitz?

  to do it. And like, you just have to. I think you have to get yourself moving. But like, again, find the things that work for you, and explore the area, and don't get caught in too much rut.

• 2:52:15Nathan Labenz?

  Wisdom to live by. Steve Mowshowitz, thank you again for being part of the Cognitive Revolution.

• 2:52:43Speaker 02

  I called it, I called it early. Read the label on every shelf. Nobody wanted the reading, so I poured one for myself. They took my list for a menu. Every man helped himself, bought a round to stop my mouth, then drank it down themselves. Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Take the one that you can hold.

• 2:53:26Speaker 02

  I wrote down every way it breaks. I kept that list for 20 years. Said they sweet-talk the doorman, said they'd whisper in his ear. But nobody locked the cellar, nobody worked the door. Stood wide open all week long, and nobody kept the score. Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Take the one that you can hold.

• 2:54:10Speaker 02

  I was right, and right is cruel. I'd love to be the fool. The stairs get shorter with every flight. Four billion years to overnight. No second draft, no note to send. One only try, and then it ends. So I'm calling loud and clear. Last call while you're still here. Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Take the one that you can hold.

• 2:55:05Speaker 02

  Friday night, I locked the door. Let 'em ring, don't answer no more. A table, a plate, a light. My wife was next. Can't wait till Saturday night, I'll be on that stool at dawn. Like I've been all along. Still holding the poison cup for a world that won't look up. Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Take the one that you can hold.

• 2:55:47Speaker 02

  Pick your poison, pick your poison. Every promise here is gold. Pick your poison, pick your poison. Can't say that you weren't told.

• 2:56:17Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:57:01Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.