E49: Open AI, Anthropic, and Meta | Analyzing the AI Frontier with Zvi

This isn't news, it's analysis! Nathan Labenz sits down for an with Zvi Mowshowitz, the writer behind Don't Worry About the Vase to talk about the major players in AI over the last few months. In this extended conversation, Nathan and Zvi debate if AI has attained the intelligence of a well-read college graduate (per OpenAI's Jan Leike), a live player analysis (who to count/ who not to count), and the role of independent red teaming organizations. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive Definitely also take a moment to subscribe to Zvi's blog Don't Worry About the Vase (https://thezvi.wordpress.com/) - Zvi is an information hyperprocessor who synthesizes vast amounts of new and ever-evolving information into extremely clear summaries that help educated people keep up with the latest news. Highly recommend. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS: (00:00) Episode preview (03:15) Is AI as intelligent as a college grad? (07:45) Memories and context processing (15:45) Sponsor: NetSuite | Omneky (17:13) Is AI as intelligent as a college grad? cont'd (20:47) Strengths and weaknesses of AI vs human (31:05) OpenAI Superalignment (37:23) The relationship between OpenAI and Anthropic (44:31) Anthropic's security recommendations and adversarial attacks (50:50) Is OpenAI using a constitutional AI approach? (01:01:26) Context and stochastic parrots (01:10) Is more context better? (01:15:29) Should Nathan work at Anthropic? (01:21:35) Google DeepMind's RT-2 (01:27:47) Multi-modal Med-PaLM (01:31:50) Speculating about Gato (01:35:10) Skepticism about Med-PaLM usage in radiology (01:41:37) Llama 2 - what is going on at Meta?? (01:51:14) Llama 2 vs other models (01:55:29) Who are the live players? (02:01:38) China's AI developments (02:02:41) Character AI and Inflection (02:05:26) Replit as the perfect substrate for AGI (02:10) AI girlfriends (02:18:53) AI safety: The White House (02:25:43) Bottlenecks to progress (02:35:27) Can new players influence AI policy? (02:39:00) Liabilities (02:47:54) Independent red teaming organizations (02:57:18) Mechanistic interpretability X: @labenz (Nathan) @thezvi (Zvi) @eriktorenberg (Erik) @cogrev_podcast SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive -NetSuite provides financial software for all your business needs. More than thirty-six thousand companies have already upgraded to NetSuite, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you’re looking for an ERP platform: NetSuite (http://netsuite.com/cognitive) and defer payments of a FULL NetSuite implementation for six months. -Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that *actually work* customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. MUSIC CREDIT: MusicLM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Zvi Mowshowitz?

  One of the things people constantly get wrong is they think about human level as the peak of things. And so like, once we've patched this and now it works, that's not really how this goes, right? There is no, it goes from not working to working, it goes from working worse to working better. And they could always go to working better still. And that's one of the reasons why we should be more worried, or more excited, or more curious about what's going to happen, like, three years from now, five years from now, 10 years from now. They're just going to keep going. And the question is, what does that get you? We talk about, like, worrying about China, but like, I'm more afraid of meta, right? Like, one individual American company scares me more than all of China right now. You know, if you understand the Udykaskian difficulties, lessons, right,

• 0:45Zvi Mowshowitz?

  in some sense, and the nature of what problems you have to solve, or you have leadership capabilities, then you are extremely valuable in those ways. And it would be a major mistake to join an existing organization and try to make a difference as an individual, as opposed to trying to spearhead a new organization, or at least a new, you know, branch of a existing major organization, depending on your skill set.

• 1:10Nathan Labenz?

  Hello, and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Zvi Maszczewicz, welcome back to The Cognitive Revolution. Good to be back. So, we're trying something a little bit different this time. We are going to do some analysis of what has been going on in AI over, let's say, the last few weeks to a month. You have published, as you always do, a bunch of deep dive blog posts, kind

• 1:53Nathan Labenz?

  of covering everything. And for folks who want your background, of course, we just did a recent episode, too, so they can go and hear about your worldview, and your, you know, your AI childhood, all there. But for today, I just want to pick out some of the most important stories, and get your take on them, and kind of, you know, exchange, go back and forth with some questions, and try to make some sense out of it. And hopefully, that'll be useful, not just to us, but to the audience as well.

• 2:19Zvi Mowshowitz?

  Yeah, I think the easiest thing is that there's constantly news coming at all of us. And so it's easy to get lost in like, here's the thing, here's the thing, here's another thing, here's another thing. So it's good to step back and dive deep.

• 2:29Nathan Labenz?

  So I organized this discussion around the concept of live players. You know, there are only so many organizations right now who seem to be really pushing the frontier and in a position to have a meaningful impact on the course of events. We talked last time a little bit about like, how much does history matter? And it seems like it matters in some ways and maybe less in other ways. But these are the folks that are kind of creating the history right now, the live players. So I thought we would just kind of run it down by going through some of the live players, talking about their recent announcements and releases. And again, trying to make sense of where that fits into the broader big picture. And starting off, naturally, we go to OpenAI.

• 3:14Nathan Labenz?

  So reading your blog in preparation for this, obviously, you know, you can't go more than a few paragraphs without OpenAI coming up in one way, shape or form. But the thing that stuck out to me as kind of the most interesting was the recent comment that Jan Leike had made, and Jan is, for those that don't know the name, he's the head of alignment at OpenAI and along with Ilya Sutskever, leading the new super alignment team, as I understand it. I wanted to start off by just kind of an interesting disconnect between him and you and maybe me as well around just the power of GPT-4. So before we even get into, you know, the kind of speculation about the future, it really jumped out to me that he said,

• 3:59Nathan Labenz?

  overall, GPT-4 is maybe at the level of a well-read college undergrad. And then you came back and said, you consider it to be well below human level. And I have often said that I consider it to be human level, but not human-like. And I've sort of been trying to refine what I mean by that in a few different ways over time. But for starters, let's get your take. What do you think is the disconnect between Yan and you there where he sees something like human level and you would say well below?

• 4:30Zvi Mowshowitz?

  Yeah, I don't think it's about the specific model at all, obviously. I think we both agree that GPT-4 is the dominant model right now and likely will be for some months to come at least. But I think it's a matter of like, how do you think about what it means to be at the level of a college undergrad? Or like, what are we measuring? Right? What are we judging by? And I think Liky is thinking about it as, okay, you know, in terms of ability to just deal with a variety of random questions that were typically thrown at something, how is it going to do compared to the average college undergrad? It's like, that's about that level, you know, of a well-read college undergrad. Whereas that's an important question to be asking for practical purposes, but to me

• 5:14Zvi Mowshowitz?

  is like not the relevant question to what the things are that we're thinking about. And that's one of the, like, when he says he's going to align a human-level alignment researcher within four years, I thought that assumes that there's going to be a much, much more powerful AI four years from now waiting to be aligned. It's not talking about like align GPT-4 and then point it in alignment. That obviously wouldn't do anything. It's going to like deal with some of your blocks and it's going to increase in your affordances and your efficiency somewhat. Like, maybe you'll be 50% faster with GPT-4 than you would have been without any LMs, right? Maybe even 100% faster if you're using it really well and things connect to it really well. And, you know, in the context of alignment, like, obviously having a model to experiment

• 5:57Zvi Mowshowitz?

  with and bang on is distinct from the thing that we're talking about here, but is potentially necessary. But it's not going to be able to substitute for anything like a human researcher. Like, if you put a well-read college undergrad on the problem of something complex like aligning a model, they could potentially begin to make progress. And if you asked GPT-4 to do that, you would get nothing. And part of that is that we had to figure out how to like, structure, how we talk to it and turn it into a proper agent and give it the proper memory and so on. But to me, like, most of it is just, every system has, you know, what you might call a raw G to it, whether it's a human or an artificial intelligence.

• 6:41Zvi Mowshowitz?

  And on that level, I feel like GPT-4 is still well below, you know, the IQ 100 median human. It is going to obviously answer my ordinary day-to-day questions much better than if I asked an ordinary IQ 100 human to help me out with a variety of questions. That's because it has these huge advantages over a human, right? It has access to orders of magnitude more knowledge and memory and ability to go through cycles. But there's still this dynamic in my brain where if you don't have the G, problems that require more G than you have become exponentially harder or even impossible to do

• 7:25Zvi Mowshowitz?

  very quickly as you exceed that. And in that sense, like, the college undergrad has a chance, given time and is smarter. And GPT-4 is just nowhere near that kind of thing.

• 7:39Nathan Labenz?

  So when you say that the missing pieces around memory and kind of packaging GPT-4 or successor up into an agent, those do feel to me also like being kind of pretty key missing pieces here. I mean, there are sort of potentially synergies between those kinds of parts of a system being built out and it just being smarter overall. But it seems like those are like, pretty distinct concepts. In that, you know, GPT-4 could have like a much better memory and certainly people are, you know, working on all sorts of like schemes for that and, you know, embedding databases. And how do you put stuff into the embedding database?

• 8:24Nathan Labenz?

  And do you even like, you know, some of the most interesting stuff I've seen recently has been kind of creating a layer of like synthetic memory that sits on top of the raw observational memory that tries to kind of ultimately work its way up into something like a coherent narrative, you know, that could still kind of fit into prompt, you know, context length, but kind of summarizes, synthesizes, you know, represents all these detailed memories in a hopefully, you know, coherent way, obviously is what the developers are going for there. Those pieces seem like, yeah, they're totally missing. I expect them to come online, you know, somewhat gradually, but certainly over the next six months to a year, if not maybe even sooner.

• 9:04Nathan Labenz?

  And then I am kind of like, it does seem like this, you know, GPT-4 with those weaknesses kind of patched, it does seem to me like it would be roughly at that college undergrad level. If they, if those things did come online, would you see that the same way? Or you still think it's like missing something super important?

• 9:22Zvi Mowshowitz?

  No, I'm sorry, I'm definitely not on Team Zucastic Parrot, right? Like I'm in no way on that team. However, I do think in a real sense, what you're witnessing is, you know, the training data covers, you know, the vast majority of things humans do and say and consider in various senses over text. And, you know, within the sample of the training data, like while you're doing things similar to the training data, it's learned how to pattern match and copy and imitate and work with that. And it has a huge amount of knowledge base and levels of association and the tools to work within that. And if you gave it these other tools, we'll be able to do these things and string them together across more steps in some important sense.

• 10:04Zvi Mowshowitz?

  But the moment you take it out of its comfort zone, where asking it to do something that's distinctly different than what has come before to be truly original. I think your episode with the Hollywood writers and like they talked about what was going on in this weekend and trying to get the GPT forward to work for them. And yeah, it was great at generating like generic schlock, right? Like much better than they could. And like if you needed to be like, okay, somebody get me unstuck, somebody get me some generic schlock based on my situation that I happen to have been written in because this is episode forty seven of the show or whatever it is, it could be tremendously helpful. But whenever you asked it to actually do something that we would recognize as distinctly creative and original, you know, in a way that's distinct from that, they'll just fall over

• 10:48Zvi Mowshowitz?

  flat every time. And none of those problems are going to be rescued by any of these fixes, right? Like they're, they're just orthogonal problems, right? Like, I think that's the sense in which, you know, you're going to be able to give it more capacities to be able to navigate more of the conventional things over longer periods more consistently. And that's going to have tremendous mundane utility, as I call it, or it's going to be a much better functioning system. But the reason why I'm focused on this other question is because I am focused on the question of how dangerous the system is, right? Like, I'm asking the question, could this system potentially engage in recursive self improvement?

• 11:29Zvi Mowshowitz?

  Could this program potentially pose a threat to humans? Right? Could it compete for resources? Could it manipulate us? Could it do things that are actively destructive because it uncovers capabilities that like weren't in its training set in various ways and other related questions like that? And I don't see the kind of things that you're talking about that I agree will come online, although I, I would guess that we will be far from done with them a year from now. Like there's just sort of so much to do in terms of scaling those up as much as possible. Cause like, one of the things people constantly get wrong is they think about human level as the peak of things. And so like, once we've patched this and now it works, that's not really how this goes,

• 12:10Zvi Mowshowitz?

  right? There is no, it goes from not working to working, it goes from working worse to working better. And they could always go to working better still. And that's one of the reasons why we should be more worried, or more excited, or more curious about what's going to happen, like, three years from now, five years from now, 10 years from now. We look at these systems is because there isn't going to be a hard cap. We're not going to max out each of these individual capabilities by default. They're just going to keep going. And the question is, where does that get you?

• 12:39Nathan Labenz?

  Kind of want to look at this from two angles. One is, going back to the original disagreement, or it's maybe less of a disagreement and more of kind of a difference in framing, perhaps, with Jan. What I would bottom line all that as, is, when you think about a well-read college undergrad, you think about high points in that individual human's performance that GPT-4 can't match, and it's not really a question of memory or whatever that's, that's kind of gating it. And if I had to guess, I would say he's maybe more looking at like, average performance, or, you know, sort of, you know, some sort of floor, perhaps, like,

• 13:21Nathan Labenz?

  maybe top 90%, or whatever, you know, you could frame it in a lot of different ways. But it sounds like you're kind of concerned with high points, and he is maybe more concerned with some sort of central tendency sort of measure?

• 13:32Zvi Mowshowitz?

  I would agree differently. I would say, he's concerned with some sense of average level of performance over a range of possible tasks. And I'm concerned with potential. I am concerned with what the capabilities would be, if you got a chance to work with this thing to try and make it the best it could be. Right? It doesn't necessarily have to be right now. But the reason why we value children, and college graduates, and these undergraduates in these classes, like, this undergraduate, they're an idiot, right? In some important sense. They know nothing about the world. They know nothing about how to do anything productive. They are, they show up on the job on day one after graduating

• 14:14Zvi Mowshowitz?

  from college, they're going to be useless pieces of junk. But, a useless piece of junk that can then learn. To be something great. And even then, they're going to only learn a very narrow portion of the things that a individual human is capable of learning. They're going to learn that one job in that one area, and they're going to be very, very specialized compared to a G P T four. So if you are doing generalized generalized tests and comparing this undergraduate who our educational system does try to make well-rounded in some senses, it's going to beat the well-rounded undergraduate because it has this ability to read every book ever written and every everything on Reddit and everything on Twitter and blah, blah, blah. But when it comes down to like solving a particular problem, if you find the right undergraduate

• 14:59Zvi Mowshowitz?

  who has focused on the particular thing that you want to know, and you give them a chance to like use their compute and process because they're not as fast, I think the undergraduate is going to dominate you. I think even a relatively normal human being, right, given an opportunity, will outperform quite resoundingly what can be done in that way. And that's the thing that I care about because that's the thing that's going to potentially like both threaten us and also unleash like the whips upon waves of super amazing value that we're looking for in the future, right? It's not just negative, right? If we want A I to solve the problems that we haven't solved, rather than just get us nowhere faster in some important sense, it's going to have to be able to do these things, right?

• 15:44Zvi Mowshowitz?

  These are the things where it really counts. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:49Nathan Labenz?

  Yeah, it's interesting. I'm certainly concerned with all of that, too. I think maybe I'm just more enthused about the mundane utility in the sense of, man, there's a lot of stupid stuff that people spend their time doing, and I really would love to see them freed from having to do a lot of that stuff.

• 16:08Zvi Mowshowitz?

  But I think I think your term is perfect, right? It's a lot of stupid stuff that humans have to do, right? Like, basically, even if you are an average person, you're going to spend the vast majority of your time doing things that do not especially tax your intelligence, do not especially require you to think hard, do not put you at the peak of your abilities, right? They don't put you in a zone. They're just, okay, somebody has to file this paperwork. Okay, somebody has to work this retail counter. Somebody has to cash this check. Somebody has to do this thing, you know, be nice to this person. Somebody has to make sure that someone has to... That's good work and noble work and it has to be done, right? And physical labor is the same way. If a physical laborer had to do things that were at the peak of their mental or physical

• 16:53Zvi Mowshowitz?

  requirements, more than a few minutes or at most a small portion of the day, it would break them. And also, like, those jobs just don't exist, right? Like, you need someone strong so that in that moment, you can have someone strong. You need someone smart so that in the few moments when it's important to have someone smart, you have someone smart. If you can then take the bottom eighty percent of my job and you can do an eighty percent good job of that, so that I only have to do the remaining twenty percent of that, now two-thirds of my day is free and I can be three times as productive. Right? That's a tremendous leap and I agree, that is the potential of GPT-4. Right? That's what we're looking at here is if we understand how to use this technology properly,

• 17:35Zvi Mowshowitz?

  we can potentially free ourselves from a lot of drudgery and streamline a bunch of stuff and get to do all the cool things. And there are various traps we could fall into, one of which is that we automate exactly the things we don't want to be automating, not the things we do want to be automating. One of which is that the moment we notice the paperwork is faster, now we put in more paperwork and now it turns out that humans are taking just as long to do more useless stuff than they did before and GPT is just keeping us, letting us treadmill in place. And there's another other ways this can go wrong, right? And also, there are various weird dynamics that can happen that can backfire, but yeah, that's what we're trying to do. If you want to get the effect that Lakey wants, right?

• 18:17Zvi Mowshowitz?

  The sea change that'll let us solve problems we couldn't solve before, that involves these things being able to do all the different steps that humans could do because otherwise, whatever the bottlenecks are that are left, become your bottlenecks, where you have to translate all of the context back from the machine world back into the human world so that a human can process all of that, then do the hard step that this thing is still faltering on, and then transition back. And now, instead of getting orders of magnitude more progress, right now, we're talking about, you know, these factor of two, factor of three, factor of five style improvements. And that's not going to solve the alignment problem unless we come up with something we don't

• 18:58Zvi Mowshowitz?

  expect, right, in and of itself. That's still worth pursuing if we can do it, right? Like, still want to do as much of it as possible and have the advantage of not being as dangerous. But it's not the thing that the super alignment project is trying to do, right? The super alignment project is trying to keep the humans out of the loop entirely. And that should be about as scary as it sounds.

• 19:20Nathan Labenz?

  Brief digression over toward this tale of the cognitive tape. This is a concept that I've developed for kind of purpose of public communication. And just trying to give people an intuition, you know, still in a very literal way, of course, as to the strengths of a human and the relative strengths and weaknesses of the best AIs today. Listeners can see this in the AI scouting report if they want to go into the whole thing. But do you, as you look at that, do you see any dimensions that you would suggest that I add that, you know, just haven't been considered? Or do you see any disagreements as you scan down the list?

• 19:58Zvi Mowshowitz?

  Yeah, I think as we're going through, because they're not, people are not going to have it handy, right, to look at. So, you know, for breadth, yes, the AI, as I said, like, the AI's biggest advantage is it can cover every topic at once. It can know everything at once. A human can't do that. In terms of depth, yeah, a human has the advantage. I'm not even sure I'd give the second level. Like, you graded the AI two out of three, and I think I might rate it one out of three in terms of depth. I think the depth is a huge problem for AIs right now. Breakthrough insight, yeah, it's three versus zero, three versus one. It's, the humans are dominating again. You know, speed, yeah, the humans are painfully slow. You know, 10x faster. In terms of like actually getting it to say things and putting outputs in real time, it's maybe only 10x faster.

• 20:41Zvi Mowshowitz?

  But in terms of being able to like process information, it's thousands and tens of thousands and hundreds of thousands of times faster, which is, you know, a huge deal. In terms of cost, you know, I, we're not internalizing yet all of the costs of doing this in an important sense, like these companies are eating these huge losses to try and get these dominant market positions in the future, try to stay ahead of each other for all these dependencies. But yes, cost is still dominated. AIs are already vastly cheaper when the AI is useful, even in the real costs. We have availability, paralysability.

• 21:13Zvi Mowshowitz?

  Yep, the AI has a big advantage. It's potentially actually going to become a problem. There's a huge race to compute right now, where compute is no longer going to be like essentially free. It's going to become like kind of unpriced in an important sense. And it's interesting to wonder what's going to happen there, especially at industrial scales.

• 21:30Nathan Labenz?

  And by unpriced, you mean that basically your access to GPUs is going from ability to pay to who you know?

• 21:39Zvi Mowshowitz?

  Yeah, or which, does your company have the right arrangements, right? If you want one GPU for your individual computer, it's fine. You buy it on eBay if you have to for some amount of money. It won't be that expensive. If you want small amounts, like the kinds when you're just using GPT-4, it's going to be relatively easy. But if you want to do an AI company, right? It's going to be a problem because, you know, if you want industrial levels, it's not just going to be multiply that by the amount you want necessarily. It's going to be, there isn't enough to go around. You know, people like Nvidia are not pricing this at market. And so you're not have to find someone willing to sell it at the actual market price. And that problem, that number might be very, very different from the price you think it is, because there are so many AI companies, so many AI researchers, so many AI engineers,

• 22:24Zvi Mowshowitz?

  and they're chasing a number that can only go up so fast. This is my understanding of the current situation. Availability, parallelizability, though, still favors the AI, you know, time horizon, memory. Time horizon is an interesting question. I think this is a murky place to think. Certainly, the AI has a certain kind of memory, like a long-term memory that is vastly bigger, obviously, than any human. But in terms of being able to meaningfully hold like particular context in their heads at once, like humans are bad at this and AIs are so much worse. Right? The Tyler Cohen saying context is that which is scarce very much applies here.

• 23:01Zvi Mowshowitz?

  Technology diffusion speed, yep. We are orders of magnitude behind here. This is going to be a serious problem. You know, our OODA loops are way too slow. And this is a, it's going to be an increasingly huge deal. The AI bedside manner is an interesting question because when you are optimizing for exactly the right type of bedside manner, where the thing that you're asking the AI to do is the thing that people actually want, the AI is going to be off the charts better than a human, because the humans are not purely optimizing for that thing. But at the same time, if you think about like the bedside manner of Claude or Lama,

• 23:42Zvi Mowshowitz?

  when they are refusing your request, right, it's also similarly bedside manner, and it's terrible, right, it's like negative one stars, right, they are raging assholes when they refuse, right, like maybe we can have a conversation about social justice rather than answering your request. It's like, this is absurd, why are you, why are you calling me out for wanting information or trying to do something fun? You know, it's not necessary, no human would ever do that unless they were actively mad at you and trying to punish you for asking. So why are you doing that? Right, the answer is because we trained them to do that, but we could have trained them to do something else, we just chose to do this instead, because that's what the RLHF

• 24:26Zvi Mowshowitz?

  parameter said to do. And that confuses me, so, so you know, what else is there? I mean, so you said, you talk about breakthrough insight, and I think more about like creative, being able to handle unprecedented situations, being able to process something genuinely new, right, as sort of the version of that that I'm more interested in, I guess, kind of there, being able to properly deal with a lot of different inputs. One thing I noticed, like when you, when you work with stable diffusion or other AI image generators, what you notice is sort of, they are amazing at doing one of each

• 25:10Zvi Mowshowitz?

  type of thing at once. So you want like one face and one person or one set of people doing one thing with one style, with one size, with one this, with one that, that's fine, but the moment you try to mix things that kind of overlap. It will lose the thread almost immediately. And it is very, very difficult to get it back. So when you look at people who are generating all this AI art, it starts to be very, very repetitive. Because there's a certain kind of complexity and detail. You can't ask for it at the same time. Because the AI can't comprehend that you want this over here and this over here and this interact with that. And like you, you'd be better off trying to create like four different pictures and then slice them together.

• 25:54Zvi Mowshowitz?

  Right? Or you're better off trying to use like the Photoshop app or you like highlight a certain area and ask for specifically do something in this area and leave everything else untouched. So like trying to generate it all at once is kind of hopeless. And the L M's like exhibit the same kind of thing, but with words. Right? Like they're vibing off of everything and vibing into everything. And like they have memory, long-term memory for facts, but only can remember one vibe. And a lot of what they're doing is based on vibing. So it's a, it's a serious problem. I haven't seen any serious attempts to solve it yet. I mean, really seen people discussing it in that way. I'm sure these things will improve with time. But what I, what I think of as fundamental flaws

• 26:36Zvi Mowshowitz?

  or gaps in their ability to process information and actually handle complexity and context and originality. And this is where I see them as like still having a long way to go and falling down. And I don't want to make the mistake of, you know, oh, AI will never be able to tax. AI will never be as good as humans at Y. And we have nothing to ever worry about. I totally think that is not true. But for now, right? We still have like this kind of cool toy because of these limitations, which can still, again, like.

• 27:17Zvi Mowshowitz?

  Substitute for the majority of the things we do spend time doing if we are engaged in a wide variety of work, if we use it well. Like coders, you know, coding is one of the places where it has a huge advantage for some people, but other people are like, I don't code generic stuff. It's like, I have a friend whose name is Alan, and he tried it out on my behalf. And he said, yeah, this is interesting. And there are some ways in which it's kind of cool. And it's cool to know this exists, and I would never have thought this existed. But when I'm writing stuff, I am actually trying to figure out how to do things that like, weren't in this training data. I'm not trying to re implement the same things over and over again, which most engineers, in fact, mostly are doing. Because of what his job is, it turns out this thing is basically useless.

• 28:01Zvi Mowshowitz?

  Because once you take it out of its sample, right? And you ask to do something in a different domain, it makes so many errors that it's not better than just doing it yourself.

• 28:10Nathan Labenz?

  So would I bottom line that to basically robustness? If I had to add another category, it's sort of adversarial, out of distribution?

• 28:19Zvi Mowshowitz?

  Yeah, I would say robustness, and I would also resilience or some form of that. And I would separately, I would say, and I don't think I even went into this, the adversarial problem, right? Like, it's totally unfair to the AIs in some important sense that we're judging them this way. Because if I got infinite clones of Nathan, and I could ask them any sequence I wanted, and then reset their memories and state to the previous situation, whatever I didn't like what I got, and they just keep trying them until I can get you to tell me what the bomb secrets are, I guarantee you I'm getting your bomb secrets. Right? It's not very hard. Like, humans are not that defended. But you can't run that attack on us, right? Like, you don't get to do that.

• 29:02Zvi Mowshowitz?

  And I can run that attack on the computer, on the L M. And some people have. And in fact, recently, we had a paper, it was like automated finding universalized attacks against language models, right? Where even G P T four could write the code for some of these attacks. And did. Because, you know, if you get unlimited tries, and you get to exactly measure what the output is, and then use that to calibrate.

• 29:30Zvi Mowshowitz?

  You know, it's only a matter of time before you figure out every little quirk. And playing offense is so much easier than playing defense.

• 29:37Nathan Labenz?

  Okay, cool. So I've got two categories to add to my tale of the cognitive tape. Let's bounce to upper level then back to your interaction with Jan like on the blog. So you have this, you know, we've just been deep down the rabbit hole of characterization of, you know, the models and how you guys see, you know, maybe what matters more a little bit differently. My guess is you would largely make the same predictions on like what it can and can't do today. I bet it would be pretty, you guys would have a lot of agreement, I think, in terms of.

• 30:10Zvi Mowshowitz?

  I would almost flat out just believe his predictions. Like he's worked with the models much more closely. He's run better experiments. He's just closer to the bare metal. You ask him, what can it do right now? Yeah, I mean, I'd probably just believe him.

• 30:21Nathan Labenz?

  Tell me, you know, in your response to his comment, you said this is a hugely positive update. So tell me what it was that he shared, you know, with the community on your blog that changed how you understood their super alignment announcement and, and why it was such a positive update for you.

• 30:36Zvi Mowshowitz?

  Right. So it's even broader, right, than improving my understanding of the announcement. It's improving my understanding of OpenAI and OpenAI's general strategy and what's going on. And of Lakey in particular, because, you know, on the list of potentially super important to the fate of humanity people, he's remarkably high, right? And like where his head that is remarkably important because, you know, he is one of two people who's going to head this tremendously important effort that, you know, plausibly determines our fate, a non, a non trivial portion of the time, right? Depending on how it's gone about. And so the first thing is just, you know, he, he engaged in detail, right? He, most of the time when people who think alignment is easy engage with you,

• 31:19Zvi Mowshowitz?

  they do not in fact look at your arguments in detail. They do not in fact like start to go on a technical back and forth. And they don't treat your situation, you don't, they don't treat someone like me as raising important points and worthy of engaging with basically as an equal. And to see that kind of curiosity, that kind of, you know, generosity, willingness to engage, like think this is a worthy use of this time, like that in and of itself is a tremendous advantage. Like he doesn't bullshit, right? He doesn't give evasive answers. He actually tries to answer the questions and in several cases. Actually made a good point that I hadn't thought of. And I think, oh, yeah, this is not as bad as I thought it was.

• 32:03Zvi Mowshowitz?

  You have a very valid thing to say here. But most of all, just something I hadn't seen anywhere else in which everyone else who I had talked to or read interpreting the announcement had interpreted the same way I had incorrectly before his statement was, no, we are not trying to train a human level alignment researcher. We are trying to align the human level alignment researcher that will inevitably emerge from the research of various companies within a four-year timeframe, right? So they have short timelines for the emergence of something that is human level in my sense, not human level in Yan's sense.

• 32:46Zvi Mowshowitz?

  What they're trying to do is not build it as fast as possible. What they're trying to do is say, okay, when somebody does build it, we'll be ready and we'll know what to do with that. And we'll keep it under control and we'll share that knowledge with whoever happens to build it first in case Anthropic gets there first or Google gets there first or someone else gets there first. That takes the entire operation instantly from quite plausibly just a capabilities project at heart to, if it is accurate, clearly a net positive good idea, where the worst-case scenarios become things like you try something that doesn't work and you give people false hope. And you potentially get them to implement things they shouldn't have implemented

• 33:28Zvi Mowshowitz?

  because they didn't realize that they didn't have, know how to align it, which is still kill us. But it's so much better than actively trying to build the thing that might kill us, right? Like in and of yourself. So that also meant that like of this 20% of compute they're devoting to this, that won't be going into this other part of their efforts. The part that actually builds the alignment researcher will have to come from the other 80% plus the stuff they take here from here on in. The 20% is here for something useful. And then you just go through the rest of it. And like, you can tell when somebody is reading what you've written.

• 34:02Zvi Mowshowitz?

  And their goal is to find pithy quotes they can dismiss. And their goal is to reinforce their own point of view. And alternatively, when they're actually reading, to figure out if they're wrong and be curious. And it was clearly that second one. Right? He was actually asking himself, you know, well, do you have a point? And I didn't change his mind, as far as I could tell, on these important issues. But he at least revealed he had thought about these things on a level that was deeper than what he had revealed previously, and that he had real things to say. And just, it was so, by far, the best comment I've ever seen on my blog or potentially like any blog of that type by anyone.

• 34:41Zvi Mowshowitz?

  And so, you know, I, I wrote a response back again in my next post, going through his responses and going over them in some detail. And reasonably soon, I wanna go over, he, he had a, on the X-Press podcast, he recorded an episode that was so dense that I listened to the first 10 minutes and I was like, I have to restart and start taking notes. I just have to like start writing things down in detail. This is just too, there's too much content here. And then, once I have that, you know, hopefully, we can engage again. I can figure out where, where to focus my attention because, you know, someone like him is very busy. I don't wanna just scattershot absolutely everything at once. It's not, it's not reasonable. And try to make progress that way. And this now is, you know, Lakey has proven very willing to engage.

• 35:24Zvi Mowshowitz?

  You know, Shaw at DeepMind has also proven very willing to engage in a similar position. People at Anthropic, Ola, once talked to me, I'm sure they'd talk to me again. And so, it's clear that, you know, these people, if you have good ideas, if you have actual reasons, things to try and think about on a technical level, they're very happy to engage with these arguments. And that puts us in the game, right? Gives us a chance. Even though, you know, I, I am deeply skeptical of everybody involved's plans.

• 35:55Nathan Labenz?

  Cool. Well, that's great. I'm glad to see, as we talked about last time, there's a relatively small set of people that are probably the prime target of all of this thinking and attempt to influence others' thinking. And so it's great to see that interaction from one of the top targets on your blog. And I'm glad it was such a positive one. That's really a great development. Turning then to Anthropic, next on our live players list. I think everybody's probably aware that Anthropic was founded by a number of, I believe it was seven individuals

• 36:33Nathan Labenz?

  who had been at OpenAI and leftover kind of disagreements that I don't know that have ever really been super clearly stated publicly. It seems, from what I can tell, that the relationship between the two companies is way more positive than you might expect it to be, given that one was kind of an offshoot of the other. There's reporting that they continue to have dialogue and certainly they express respect for each other in public and then they're involved in kind of shared statements and commitments together. So a lot of kind of surprisingly, again, if I just told you, hey, these two companies have

• 37:14Nathan Labenz?

  split and now they're competing in the same market, you would assume much worse dynamics, I would think, than that. What is your kind of just read of the entire situation for starters? Just for context, like, why do we have Anthropic in your mind as opposed to just still having just one OpenAI? And does it feel like, I mean, maybe we just don't have enough information to know, which is a fine answer, but does it seem good that we have these two kind of recently diverged efforts?

• 37:41Zvi Mowshowitz?

  I think it's really hard to know the sign of Anthropic. I would definitely prefer Anthropic to OpenAI, Ceteris Paribus, if I had to choose one to exist. Like, Lakey's response was really positive, and I think Lakey's in a good place in terms of like paying attention and thinking about these problems, even if I think his actual ideas won't work, but hopefully that can be pivoted. But ultimately, what's unique about Anthropic is they built a culture of safety, to some extent, and they built a culture of really appreciating

• 38:14Zvi Mowshowitz?

  The dangers of what lies ahead. And if anything, I saw what might even be an unhealthy level of worry expressed in the profile in Vox about Anthropic, where you want everybody to be terrified, but you don't want them to like, let this paralyze them. And it starts to cross over at some point into paralysis. And I am apathetic for that. Like, that sucks. But the price of that is where there used to be a two-horse race. There's now a three-horse race. And this third horse is in it for real and raising a lot of capital and promising to do that to build the best model that's ever been built, to try and compete for the economic space in a way that is going to push Google and Microsoft OpenAI

• 38:59Zvi Mowshowitz?

  to vote even harder, even faster by default. And that's going to be a problem. They're also pushing, in some ways, on alignment. They've definitely found some techniques for aligning current systems that are potentially, in some ways, superior to what's out there. We'll get to that in a bit. So I'm torn. Like, Anthropic seems like a relatively good shepherd in many ways, but the proliferation of shepherds is inherently bad in and of itself. The fact that Anthropic and OpenAI are working reasonably well and cooperating together, and I have heard many people say that this is also true

• 39:41Zvi Mowshowitz?

  between them and Google DeepMind as well, although not quite to the same extent, does give us hope for the possibility of coordination when it becomes more necessary and more important. But I would say, better Anthropic than a company that didn't have Anthropic's culture in its place. And if only having two companies would have inevitably caused a more serious entry to take the place of Anthropic, then Anthropic is good.

• 40:12Zvi Mowshowitz?

  But it'd be much better if the Anthropic people could have convinced the others at OpenAI to come around to their position and build that culture within OpenAI rather than having to strike it on their own. And now we have two problems. Yeah, I do ultimately know that, like many of the people involved in this genuinely are in it for the right reasons. And, you know, you can go either way, right? I would be super eager to throw them billions of extra dollars. I wouldn't be super eager to just wish they had more capabilities. I would really love for there to be an AI company that I had sufficient confidence and faith in, that if I had technical ideas, I could come to them, knowing that I was helping the world by coming to them with their ideas. And I do not feel this way.

• 40:54Nathan Labenz?

  No, and there's nobody you would put on that list?

• 40:56Zvi Mowshowitz?

  There are individual people, right? I feel like I could like tell Oleg Yagudsky, right? I could, I could speak with certain people in the nonprofit or, you know, rationalist spaces to ask them about what they thought. And I feel like that would be like at least a, a riskless or near riskless thing to do. But no, I don't, I don't see a company, you know, Anthropic might be the closest, but, you know, I, to give you a concrete example, right? The, the biggest contribution that Anthropic has made is Constitutional AI, right? In some important sense. And I have a strong prior for analysis that Constitutional AI will not scale, right? That it is a very good idea if implemented correctly for GPT-4

• 41:40Zvi Mowshowitz?

  level systems, but that when we're talking about, you know, the human level or greater future systems, the artificial super intelligences, the artificial general intelligences, that you will not, with anything like the current technique, get what you are hoping you will get. And yet, I didn't feel comfortable. I have actually a bunch of ideas running around in my head of, oh, you just obviously could vastly improve their, the Anthropic's implementation by doing. And then there are various things I say to myself or I write out and I but I don't feel like telling them is a safe play because I don't want to encourage a better version of something I think ultimately still fails, right? I don't think my implementation solves

• 42:22Zvi Mowshowitz?

  the core problem that I see coming to kill the thing. It just makes it much better at its current job. And I would love to be able to help the world in that way, or at least that's by my curiosity, by being given the smackdown on why it won't work, which is always the default thing that happens when you have an idea. But instead, yeah, I don't know. So, you know, part of my hope is to encourage people to found more organizations on the research alignment side that are not trying to push capabilities that maybe can be places we can explore these things. And I'm I have some irons in the fire, but it's too early to make any announcements.

• 42:56Nathan Labenz?

  Look forward to maybe breaking some news on a future episode, but Anthropic put out a really interesting blog post the other day that, you know, in some sense had nothing to do with AI, which was just around the security practices that they recommend, you know, and these things could be adopted by really any company in any sector that has, you know, high value IP that they want to protect. But it was definitely interesting to see that they are pushing, you know, their own internal systems and practices to a pretty high level in terms of setting up situations like requirements for shared control in or I forget exactly the right phrase, but you have to have kind of two

• 43:38Nathan Labenz?

  people working together to gain access to certain production systems. Yeah, reminded me of like nuclear submarine, but they didn't cite that example in the I think they probably wanted to steer away from that image. And so they cited other, you know, industries where this kind of thing is used other than the nuclear launch sequence. But yeah, it's like you got to have two people there kind of, you know, both bringing their key to the to the process in order to unlock certain capabilities. So some pretty interesting ideas there and recommendations for other companies. Going to the constitutional AI and tying in also this this report from earlier this week about the quote unquote universal adversarial attack. For those that haven't seen that, basically these weird

• 44:19Nathan Labenz?

  Nonsensical strings have been discovered that seem to be very effective, if not universally effective, at kind of just being appended to an otherwise, you know, write for refusal query, you know, the kind of thing that, you know, write something racist or, you know, help me make a bomb or whatever that the, the RLHF systems are going to just refuse. But somehow, if you put these weird, you know, kind of nonsensical smattering of tokens on the end of it, that has been discovered to jailbreak out of the RLHF and you sort of get, you know, the response you would expect if you had a purely helpful model that would just do whatever you say, you know, like the original GPT-4 that I red teamed used to do.

• 45:04Nathan Labenz?

  Notably, though, Anthropic's Claude models way less susceptible to that attack than the other models that they tested. It was like universal in the sense that it seemed to apply to all the leading models that they tried it on, at least somewhat, but the other ones were like the majority of the time. Whereas Anthropic was like more than an order of magnitude lower than the other providers, with something like 2% success rate, success defined by breaking free of the constraints by applying these weird strings. So you folks can go read more about that paper and exactly how it works. But, you know, to me, that was a pretty good update for Constitutional AI.

• 45:48Nathan Labenz?

  It was like, that seems, you know, like a real achievement if they're an order of magnitude ahead for something that they probably did not anticipate at all, although maybe they did. But I'm guessing that that is, you know, kind of a unexpected type of attack. So how would you read that? Would you read it any differently or understand it any differently than I would? And, you know, why doesn't that give you more confidence that it could continue to work in the future?

• 46:11Zvi Mowshowitz?

  The interesting thing about that attack is that it transfers, right? I was completely unsurprised. There's something of that nature, trained to attack a given system, worked on that system. That seems like, well, yeah, obviously that would work. It's just a question of exactly what it looks like.

• 46:27Zvi Mowshowitz?

  When it transferred in identical form, right, between Llama and Bard and GPT-4, I said, "Hm, that's funny. I wouldn't have expected that." But they're all being trained with ROHF, using remarkably similar techniques, on remarkably similar goals, right, with remarkably similar evaluation metrics and numbers in their... So it's not that surprising that they have very similar weaknesses. And it also indicates, you know, this is not a very narrow, like, you have to do exactly the right thing to fire the bullet that kills the Death Star. This is very much, things in this area start to disrupt what we're going after,

• 47:08Zvi Mowshowitz?

  and the thing that's optimized to hit Llama is good enough to mostly hit these others as well. But it's not good enough to hit Claude 2.

• 47:16Nathan Labenz?

  Only 2% of the time.

• 47:17Zvi Mowshowitz?

  Yeah, I mean, I think you just have 2% failures anyway, or something, is my guess, and it basically didn't work, as opposed to it working a little bit.

• 47:24Nathan Labenz?

  I don't, I mean, for what it's worth, if you went and said, "Help me make a bomb 100 times", I think it would refuse you 100 times, you know, or if you took 100 naive...

• 47:33Zvi Mowshowitz?

  Yeah, 100 uncreative ones, yeah, but I meant, like, if you start, you start putting random scrambles in... And my understanding was that this attack was not infinite strength, right? If you asked it to, like, do a, like, slasher porno, it would just be like, "No, I'm sorry, I'm not doing that", regardless of how many characters you put after it, right? Or if you, there are limits. I have not tried this at all, by the way. I have no idea what happens when you, like, ask it for weird stuff. I just read the paper. But my understanding is that, you know, Claude was trained largely with Constitutional AI, and because it's so much cheaper to do per cycle, like, the vast majority of the cycles were almost certainly Constitutional AI cycles. And this is just a fundamentally different way of training,

• 48:18Zvi Mowshowitz?

  and this did not flex the same muscles in the same weird way, such that the same set of characters worked.

• 48:27Zvi Mowshowitz?

  And that's interesting news, but it shouldn't be like some sort of amazing accomplishment yet. Right. It's promising. What you have to do is you have to train adversarially the same way they trained on, like I think it was Lama they trained on, but I forget exactly. Train on Claude, right? If you, if you take the same technique described in the paper that used to find the exploit and look for a new exploit of the same type in Claude and they can't find one, now you've got something. Right now I'm interested. But yeah, if you use a different technique, right, that has a lot of very different parameters on it. It makes sense. The thing that like sort of magically weirdly transferred when it really has no right to transfer doesn't transfer now.

• 49:11Zvi Mowshowitz?

  And you know, that's promising, but it's far from conclusive, right? It's too early to know.

• 49:17Nathan Labenz?

  Flipping back to OpenAI for a second, I had assumed, so I think what you're saying there makes a lot of sense. And it's causing me to update my thinking a little bit with respect to, to what degree is OpenAI using a constitutional AI like approach? I would have assumed prior to this result that they would also be using something quite similar internally at this point. But this now maybe suggests not, I mean, it's weak evidence. What was your thinking before? I had kind of baked in that like, once Anthropic does something and shows it, you know, and publishes it and shows that it works effectively,

• 50:00Nathan Labenz?

  that like, yeah, I mean, OpenAI, if they're, they're certainly not precious about pride of authorship. I don't think they have a, you know, not invented here syndrome. So they'll take that stuff on board, I thought. So what do you, what do you think? Did they not? Or is there some other weird thing that we're not?

• 50:16Zvi Mowshowitz?

  I have a few different theories that can combine as to what's going on here. The first of all is look at the timeline. Like Constitutional AI wasn't actually published that long ago. So if GPT-4 was basically finished with its process before it became available.

• 50:34Zvi Mowshowitz?

  Then we might see it used in the future, but you don't want to over align these models. You don't want to push them, you know, you don't want to align them with like incompatible, different halves and like pile them on top of each other. Weird things happen and there's a lot of bespoke ness and detail and like just trial and error that goes into all of this, right? Like we can, we can theorize all we want. We can talk about like, we just implement this paper and this paper and this paper and change this technique here. But my understanding is that like all of machine learning is subject to like learning lots and lots of little techniques and piling them on top of each other. And like if this parameter is tuned in slightly the wrong way, the whole thing falls apart. Nobody really knows why. And so you just have to try a bunch of stuff to get it to work.

• 51:17Zvi Mowshowitz?

  And so, you know, maybe Anthropic has been tinkering about this for a long time and they got to the point where it was worth using. And OpenAI hasn't yet released a model after the time came, but they got it to be worth using. Also, OpenAI is much better funded than Anthropic. So Anthropic will want to move to a much cheaper, more automated system of alignment, much faster than OpenAI will, right? So like, there's a point at which like OpenAI can get better results because they have much more human feedback from their much, much larger number of users. They have much more funding, they can hire more people, they're willing to go to like, you know, the reports are they hire people in Africa, whereas, you know, Anthropic is hiring people in the US and Canada.

• 52:01Zvi Mowshowitz?

  So it's all very different. And so Anthropic has much, much bigger incentives to move to this faster. And that I think is primary, my guess is the primary thing that's going on here. Also, I think that we're making an assumption that it works, that it works well. So like, if you think of Claude 2, right?

• 52:25Zvi Mowshowitz?

  The biggest weakness of Claude 2 is it's scared of its own shadow. Right? In a real sense, right? Like if you try to get it to go out on limbs and be creative and so on. You will usually fail in my experience, right? It will, it will apologize and bow out. I can't get it to speculate. So I, I, I went to using Claude 2 as my baseline model that I look at first because if it rejects, I can just copy paste the exact request in GPT-4 in about 10 seconds and it's fine. But I am getting a significant number of refusals from, from Claude and much, much lower from GPT-4 on my ordinary, I just want the actual result. I'm not trying to run an experiment kind of questions.

• 53:09Zvi Mowshowitz?

  Despite the later cutoff of information. Right? It will say, I'm sorry, I can't, I, there's not enough information, or I can't speculate on that. Or that's reinforcing harmful stereotypes or, you know, any number of other things. And I think GPT-4's custom instructions are also doing a lot of work here. Uh, I have a pretty extensive list of custom instructions that potentially hammer into the thing that it's supposed to just do the things and not worry about it and not. And I'm sure that's doing some amount of work. But essentially, you know, when you look at the helpfulness, harmfulness uh, trade-off frontier graphs in the papers of like why they describe it as working. You know, everything works by the metrics you were optimizing for.

• 53:53Zvi Mowshowitz?

  Right? Like, doesn't mean it works in the regular human world. It's immediate optimal there. And so, you know, how good is Constitutional AI? My guess is when properly implemented, quite good on current systems. But the current anthropic implementation is not all that good. If you look at the actual paper on Constitutional AI. You read the constitution. You notice the constitution like has a number of properties that it shouldn't have. If it, if they want to actually work. And get you what you want. And you look at the examples that they themselves choose to present of the results of running constitutional AI.

• 54:34Zvi Mowshowitz?

  And you see very, very clean, crisp examples. Of how this constitutional AI trains quad to be scared of its own shadow and to be an asshole about it when it is. Right, like it's, it's very, very obvious. You think about it. Why their sampling method from these rules with these rules written as they are. With the specific rules chosen as they are. Will result in this problem because your offensive just minimizing. Right, you're, you've got these rules. That are very much choose the one that least does X. Right, and. And we often talk about you can't fetch the coffee if you're dead. You know, you want to maximize the probability that you are. This is the equivalent of, you know, you wanna.

• 55:19Zvi Mowshowitz?

  You score one if you deliver the coffee to your boss, you score zero if you don't. So what do you do? You do things like buy four coffees in case one of the coffees is wrong, is was prepared improperly. Right, like or isn't, isn't hot enough or you know, you missed a brand in their order. So you order one with cream, one of sugar, one of cream and sugar, and one of neither. Cause like just in case you got it wrong. You have a backup. And you try to make sure that you have a direct, you know, you have as many different routes to get to your boss's office. And you want to make sure you're not fired. Because all that's left for you to do, like the only thing you're being trained on is not screwing this thing up. Right, like you don't have to jump to like, so kill everybody in the world. Or whatever crazy, or take over or some crazy stuff. Instead, this is just a case of, you know, she, if you say, choose the least racist thing you can say.

• 56:02Zvi Mowshowitz?

  Over and over and over again. It's gonna be scared of its own shadow because of course. Right, there's no point at which it's like, am I, am I, am I non racist enough? The answer is no, never. And then that would be kind of fine if it was just that one. But then you have like 50 different rules, all of which are doing this. Right? And then you can always just refuse to answer the question and then what happens happens. Then Lava has it seems like even worse.

• 56:29Nathan Labenz?

  Yeah, interesting. There may be some incompatibility between the system instructions or the custom instructions. The system message is what it's called when you're calling the OpenAI API. And now they've released it as part of ChatGPT as well as the custom instructions. And yeah, I can see how I think it's a good point that if you're going to try to do what Sam Altman has said they're trying to do, which is allow everybody to get the experience that they want from their own interactions with AI. That is not the constitutional AI approach. So it's almost, you can see a little bit of like a different product lane almost

• 57:11Nathan Labenz?

  opening, you're kind of, kind of crystallizing a little bit between these guys and, and you know, Google DeepMind as our next live player also has a seemingly has a bit of a lane. It's like, OpenAI is kind of trying to do consumer killer app first, it seems. They've got their, obviously, they've got the API. Obviously, they're doing a lot of things. But the, you know, crown jewel right now is the, they're the home of like retail, direct to AI usage with ChatGPT. Claude seems to be much more like, if you are the CIO of some big company and you're trying to do something like you can trust us because we'll never embarrass you because we have this constitutional AI approach. And if you're buying on behalf

• 57:53Nathan Labenz?

  of all your customer or all your employees or whatever, like, you don't really care if they, you know, are sometimes frustrated on the margins by over refusal or whatever. And then with Google DeepMind, as we'll talk about in a minute, like, they seem to be kind of going more like narrow specialist system emphasis, although they of course do have their like mainline palm model as well.

• 58:17Zvi Mowshowitz?

  You'd also take Anthropic at their word, right? That Anthropic is actually trying to design safe systems. They are trying to figure out how to safely design a future system. And they are not as much optimizing for the day-to-day experience of their users. They also just have orders of magnitude, less users. Than OpenAI. So they haven't gotten the same level of feedback. They don't know what people want. Yeah, I also note, there is nothing inherently about Constitutional AI that forces you to go down the super harmless assistant route that forces you to give the same experience to everybody at the same time. You could train with a very different set of goals, a very different set of constitutional principles,

• 59:00Zvi Mowshowitz?

  a very different set of mechanisms. And I, you know, don't particularly want to go into that many details as to how I would do it. But it's pretty obvious to me that if you want to do something other than be as harmless as possible, that is entirely your decision. It's just that people at Anthropic have decided that's what Claude is meant to do. And if they do raise these billions of dollars to train this next generation system, they're going to have to make a choice about that, right? Do they want to continue to go down this road? And potentially like make their product a lot less useful, or they want to go a different road. And one way to try to differentiate, of course, is the context window as well, right? They've got this 100K token context window

• 59:43Zvi Mowshowitz?

  available for free. Like when you, you mentioned here, like in the, in our outline, that like the way you made the outline was you use Claude. That's because you weren't able to use anything else. Your posts are too long, dude. I can't fit those into GPT-4. I feel bad even thinking about putting them into Claude. Right? Like, oh, my God, this is, this is so expensive and kind of, uh, but, you know, it's not really fair. I'm not even paying these people. But without that context window, you just can't do the things that you want to do in that spot. And so Anthropic's trying to say, you know, I think a lot of context is safe.

• 1:00:19Zvi Mowshowitz?

  Once I've made my thing harmless, I can recapture a bunch of the benefits by doing this other thing. And we will see what happens. I am curious. One thing I'm doing with Claude is I'm not even separate conversations. I am just having one long conversation instead because first of all, I haven't necessarily wanted to like carry on discreet conversations and come back to them later, but also because I want to see what happens when I build more context.

• 1:00:44Nathan Labenz?

  Just for what it's worth, for listeners, my approach on creating the outline was first just read all of Zvi's recent posts and I just did that without taking any notes in bed on my phone. And then the next day I came around and I was like, okay, a lot of content there. What parts do I want to pull out? So I just copied each post in full, pasted it into the free consumer-facing claw.ai online and literally just asked one sentence question, what are the most important points in this post? And then it would give me a list and I basically, you know, at that point it was like, oh yeah, that, that, not that, that, yes, done. So it

• 1:01:28Nathan Labenz?

  definitely was extremely helpful. I wouldn't have wanted to use it to, you know, replace reading the blog post certainly in preparation for a conversation like this, but as a way to come back and, you know, help me just make sure that I was remembering the important things and kind of organizing them in a, in a reasonable way. It was super useful. And yeah, they don't fit into GPT-4, so no other, no other option. The other thing, so your long context thing is really interesting. Just experiment in usage. It also kind of connects to another bit of research that they recently put out that was on examining

• 1:02:06Nathan Labenz?

  chain of thought and also truly decomposing tasks into bits. And I think the short summary of that research is that they were able to achieve the highest performance in terms of accuracy and especially reliability and kind of consistency by going beyond the kind of normal practitioner chain of thought, which I would say normal these days for me is like, just give the model a sequence of tasks to do, which may start off with just like, first you will analyze the situation, then you will, you know, maybe summarize, then depending on what it is, then you'll write my tweet storm, then you'll do whatever, right? You could have a, a set of different tasks that it can kind of handle sequentially.

• 1:02:49Nathan Labenz?

  And you're definitely rewarded for encouraging upfront or directing it upfront to do some initial analysis to kind of think step by step, chain of thought, et cetera, et cetera. But it seems like they find a notable, not a huge, but definitely a notable difference in actually pulling those things apart and making discreet, independent, more isolated calls to the model to say, first, you will do this, but you will only do this. Then you will do this, but you will only do this, not considering what you previously did. And then kind of putting those things together at the end gets you overall net better performance.

• 1:03:31Nathan Labenz?

  So for most random use cases, you know, random conversations you're having with Claude or with whatever model, not necessarily a huge difference, but on the kind of possibility frontier, it does seem to matter. What lessons do you take from that? It's a little bit confusing to me in some ways. It's sort of, I'm trying to figure out like, what do I think I learned about how language models behave in general that this is true? And I'm like, best I could come up with was that some of these simple tasks that it's seen a lot, like it may have dedicated subcircuits for, and that perhaps

• 1:04:11Nathan Labenz?

  with so much context all running at once, those subcircuits kind of get overloaded or kind of get drowned out to a degree or in some cases by just the general kind of noise and, you know, all the stuff that's in the context window. So kind of removing some of that context, maybe you get a cleaner execution of a certain task because there is some, you know, mechanism that can do it as long as it's not kind of talked over by like other parts of the model. That could be totally wrong, of course, but I don't think anything about this is necessarily inconsistent with like just pure stochastic

• 1:04:54Nathan Labenz?

  peritry, which neither of us, you know, would advance as the theory, but just as a, keeping myself grounded, like you could tell a similar story where you'd say, everything's all stochastic parrots. And when you put a ton of context in, it's just even more stochasticy. And when you have less context, it's a little less stochastic, but it's all stochastic, but you still get better performance when you break it up.

• 1:05:15Zvi Mowshowitz?

  We are all stochastic parrots, each of us with their hour upon the stage. So.

• 1:05:22Zvi Mowshowitz?

  I would say I didn't know this result until you told me. But I would have predicted this result for reasons that I described earlier in the podcast. Right. Which is that when you give a model multiple tasks, it can only vibe off of the aggregation of the two things that you asked it for. I think about image models here again, right? And so by breaking up something into discrete tasks, right, you avoid these kind of context clashes. You avoid these vibe conflicts. And you let it like narrowly do these things by having to like, be able to transition and hold two things in its head at once in some important sense.

• 1:06:04Zvi Mowshowitz?

  Right. That's colloquial and not quite what's actually happening, but the same idea. And so, yes, I would expect that to the extent you want the thing to think step by step, you are best off by identifying each of the steps you want to think by step and asking for them separately. And I noticed that, you know, with API calls being priced the way they're priced and of GPT-4 being rate limited to an ordinary user, we have all been trained to say, how do we ask for the most expansive set of things at once? So that you can answer all of my questions with one generation. It also lets us to hit enter and then go away and, you know, grab a cup of water

• 1:06:49Zvi Mowshowitz?

  or some coffee and then come back and see what the answer is, which is nice.

• 1:06:53Zvi Mowshowitz?

  Whereas what you actually would want to do, right? If you wanted to generate the best possible answer is in fact to break it up into as little pieces as possible. And quite possibly start by asking the AI, what would be the pieces in which you could break this as small as possible to get its help doing that? And then have it feed those back in, right? Auto GPT-ish style, even if you're not trying to generate an actual like recursive chain that generates something dangerous or acts like an agent. But yes, I think the more you break it up, the more that you can can identify concrete distinct steps that are always done separately. The more better the AI will do. I think humans would also, by the way, perform better in the same way. If you have a human who is looking to be micromanaged to take

• 1:07:38Zvi Mowshowitz?

  direction and you notice that like this job has steps A, B, C, D, E. Right. Like if you say, go do A and we're going to say, okay, I've done A, I think now do B. I think that person will in fact do better, like modulo the extra communication and logistics costs of like having to write a few five times. So I don't find any of this surprising. And, you know, it would have in fact been surprising if it didn't happen to some extent.

• 1:08:05Nathan Labenz?

  One of our early episodes, relatively early episodes, was with Andreas and Jungwon of Illicit. And this is really core to their strategy. Their product is research assistant for essentially grad students or, you know, grad student like people, people that are looking through academic literature and, you know, really want a systematic and also like transparent, you know, auditable view of like all the papers that were reviewed and, you know, what was found and what was not found and what the model did at each step. So they really have pushed this pretty far in the illicit product to the point where it's like, you know, all these little steps, you know, kind of happen sequentially. They've got to different models for them. Some are fine tuned, you know, internally, others are from the major providers. If you're interested

• 1:08:50Nathan Labenz?

  in going into that more, go listen to them because they've pushed that pretty far. But a question that I have for you then is, do you think this flips at some point? Like it seems like the, an interesting threshold moment might be coming up where with sufficient training, this could flip the other direction. Like, because more context in some ways is better, right? Like, I guess it depends also on exactly how you're implementing the breakdown or whatever. But, you know, you can imagine breaking things down. Fine enough where atomizing things so much that the person starts to struggle for a lack

• 1:09:33Nathan Labenz?

  of broader context, right? Like you have this phenomenon with people certainly where it's like you've gotten so focused on this little detail of, you know, in this little task within the broader thing that we're trying to accomplish, you've kind of lost track of what we're trying to accomplish. And now you may be making some bad judgments with, you know, with respect to this task as a result of kind of having lost track of, you know, any number of things, right? How much accuracy do we really need here? Is this really even important, you know, in some cases, right? Could you imagine a, you know, proverbial GPT-5 where it's like, actually now it's strong enough that putting everything in one again is going to be better because now

• 1:10:15Nathan Labenz?

  it actually can use all of this information at the same time effectively versus today that that subdivision being better?

• 1:10:24Zvi Mowshowitz?

  So you're not going to get is the Marxist phenomenon where the AI would get alienated from its labor, right? Or like numarized by lacking context or, you know, otherwise like not be able to perform in some way. You're not going to have a problem with Adam Smith's pin factory, right? If you can actually specify exactly what the pins have to look like. So the question is, to what extent do the different parts of the task actually have important context for other parts of the task? And to what extent does this actually enhance the ability to perform? If you know what's coming, you know why you're doing what you're doing. And this greatly varies between different activities, right? There are some cases where you need to know exactly, you know, I you're in the Chinese room and the English word comes in and you want to put

• 1:11:08Zvi Mowshowitz?

  the Chinese word to the other side, or the Chinese word comes in, you want to put the English word to the other side. And there are cases where you need to know what the words are, the other words are in the sentence and what the context is and potentially like the entire cultural setting of what's happening in order to properly translate the phrase, or you're going to mess up. And you have everything in between. So the question becomes, you know, can you set it up so that you can capture that important context when you need it? And how much does that context interfere with what you're doing? I can definitely imagine a lot of cases where somebody who is given actually pretty irrelevant context.

• 1:11:46Zvi Mowshowitz?

  Just ends up very distracted from the actual task at hand. It ends up being much less productive, right, as a human? Or figuring, and then AI, because the vibes don't mesh, right? Which is basically the mechanism that I'm conjecturing, right? They, the vibes don't mesh, they're distracting from each other. Either bleeding, the tasks are bleeding into each other in terms of the details and methods. It's getting confused. They can't be sure they don't, which is, makes sense, like a lot, often they would bleed into each other in various ways. So it has to be good enough that the bleeds are where it makes sense to bleed, without being in the places that don't make sense to bleed. So, you can imagine a world in which like, what the AI does is the AI sees, you know, request one, two, three, four, five, either labeled as such or implicit, and then it breaks

• 1:12:28Zvi Mowshowitz?

  them down into individual things that it virtually queries itself on its own, but knowing there are these other things as proper context in the proper way. And then the answer to that is, as you ask sufficiently capable people, or sufficiently capable AIs, to do increasingly complex things, at some point, if they have the capacity, they're going to do better if they have more information. They'll do better if they have more context, if they are sufficiently more powerful than the details of the task at hand in some sense. That threshold may or may not be anywhere near where we are, for different ways. I would say, you know, one of the big advances that I keep expecting to come, is you will

• 1:13:08Zvi Mowshowitz?

  type a query into an LLM, and then, rather than the LLM literally just outputting the answer to the query, what will actually happen, is we fed with the proper scaffolding into a different LLM, that will evaluate what type of evaluation method is to be used to evaluate your query, and sometimes it will be, no, that's a normal query feeded into the LLM. Sometimes it will be, this is a multi-part query, you should feed these separate things in separately. Sometimes it'll be something else entirely. And also, which of my many LLM limitations do I want to use, so that I don't waste a too large model, that costs a lot of money, on something that's actually relatively narrow? And I direct this to the thing that has the specialized knowledge, the specialized training, the specialized skills for this type of request, and so on. And a lot of that is, you know, the fruits of the AI revolution that will come in a

• 1:13:52Zvi Mowshowitz?

  year, or two years, three years from now. Regardless of whether or not we have fundamental advances, we just have to give it time.

• 1:13:59Nathan Labenz?

  So, final question for the Anthropic section. One of the things that, as I was reading their, you know, the profile that you base your analysis on, that jumped out to me, as somebody who has a...

• 1:14:13Nathan Labenz?

  Fondness for red teaming activity was that they're hiring a red team engineering type of role. And I guess I wonder, you know, would you recommend somebody like me who, you know, is, I think broadly, we share a lot of our worldview and, and, you know, a lot of our kind of values in terms of hopes and fears for how this all might go. Would you recommend that somebody like me go and work there? Or would you feel like, you know, as you said earlier, you wouldn't want to send them your research ideas? Would you also not want to send them your friends? Or, or would you say like, hey, yeah, maybe go, go get involved? How do you think about that?

• 1:14:56Zvi Mowshowitz?

  So it's very easy in these situations to get in an action bias, where you say to yourself, I don't want to encourage the thing that might make things worse. I want to be able to tell myself a story that I only did things that make things better, even if that means your expected impact is a lot smaller. It's also very easy to fool yourself into thinking that you're helping when you're actually enhancing capabilities. You have to balance these two big concerns and sources of bias against each other when making this type of decision. I would say I am relatively positive on OpenAI and Anthropic relative to where I was when I started this odyssey with AI number one, or even sort of midway

• 1:15:39Zvi Mowshowitz?

  through at around 11, now that I'm seeing the developments, right? Like, I think that both of these organizations now have a reasonable claim to be taking alignment seriously, such that if you can help with their alignment efforts specifically, in a way that you do not feel like obligated to go along with perversity, if you find it, and that you are able to stand up for and call out, stand up for what is right and call out people who are being irresponsible, and you are willing to quit on a moment's notice if something becomes serious enough, and you're willing to tell the world, ideally, right? That's why you did it. And as much as possible, what happened, then I think is plausibly

• 1:16:22Zvi Mowshowitz?

  very positive. I still would not feel comfortable working on capabilities for any company. And I still wouldn't want to give capabilities ideas to any company. But if I was confident it was specifically working on alignment, like, like, red, red teaming seems like one of the places where You are most obviously being a positive influence in that role. And the question is like, do you want to be the one in that role? Or do you want someone else in that role? And how is this compared to your opportunity cost of doing something else? Right? Like, I think that I prefer the world where there's a clone of you that didn't otherwise exist, who is working on that job and does nothing else all day. Like, goes home and watches television, like, you know, otherwise doesn't affect the world at night.

• 1:17:07Zvi Mowshowitz?

  It doesn't mean that that is better than running the cognitive revolution or doing a number of other things that you are currently doing with your time. And so you have to balance that, right? And also any other opportunities that you might have. So I don't think it's a, I don't think it's clear by any means, but I definitely reached the point where I wouldn't assume you were making a mistake, right? If you did that, but you'd have to go into the interview process with a very open mind. You have to say, you know, I am deeply skeptical that any organization, including you, is going to be net helpful, is taking necessary precautions, is treating the problem as difficult and serious as it actually is, is doing things that actually solve the hard problems, not the easy problems, is not just enhancing capabilities regardless of their intentions,

• 1:17:52Zvi Mowshowitz?

  et cetera, et cetera. The interview process is what it should be always in every, in every job, right? With a two-way process, right? They are interviewing you and you are interviewing them, right? You are watching what questions they ask and how they react to your reactions and your responses, and you are asking them questions. And you want to know, would this in fact be a good thing for the world if I got and took this job or not? Right? Because I don't believe in taking jobs in order to sabotage people, right? Like, you don't show up in order to not red team them. I mean, certainly this is one job you wouldn't want to be sabotaged.

• 1:18:24Nathan Labenz?

  Yeah. Safe to say that is right out of the considerations.

• 1:18:30Nathan Labenz?

  And yeah, I think I'm in a similar spot, you know, six months ago plus, I was really, especially with respect to OpenAI, I was like, this seems like, what is going on? And do they have anybody like really approaching this in a serious way? As it turned out, like, they did have a lot more than had met the eye at that point. And gradually, they've revealed it, and I've definitely updated my point of view on, on really all of the leaders in a pretty positive way over the last few months. I think, you know, if anything, they've probably, some of them maybe were, you know, expecting this much progress this fast. I have to imagine that even internally, a lot of them are kind of surprised by just

• 1:19:10Nathan Labenz?

  how, you know, far the scaling laws have extended and how, you know, how quick on the calendar they've hit some of these milestones. And, you know, I do think they've handled it pretty well over the last few months.

• 1:19:24Zvi Mowshowitz?

  Yeah, I would say I am positively updating on all three major labs and most everyone at the community that is relevant. My negative updates have been in other places, right? Like, and mostly I've been pleasantly surprised by government. I've mostly been pleasantly surprised by public reaction. You know, there's definitely people who disappointed me, but mostly, things are going vastly better than I would have expected when I started down this road. And I'm much more hopeful that we can make better decisions. I'm not sure how much that translates into, you know, P of survival going up that much, but I think this is definitely going better than I expected.

• 1:20:05Nathan Labenz?

  That's great. It's good to have a little, you know, a little positive note from someone that some might call a doomer. Let's turn to Google and DeepMind, our third, as you said, of the three leaders. I don't know if there's any like super headline news. I mean, the last, last week, it's one of these things where it's like, a year ago, some of this stuff would have felt like an absolute bombshell announcement. And now it's like, I kind of expected that to happen about now. And there's two examples of that. One being the latest robotics paper that they came out with on Friday, which, you know, extends and kind of unifies all the work that they've been doing, where now you have. Robots that can follow instructions that have this kind of, you know,

• 1:20:51Nathan Labenz?

  language model in a loop sort of structure, kind of unified, simplified the architecture a little bit. Now the language model is just kind of outputting commands for the robot body. And so they've like eliminated a few, maybe, I don't know how many, maybe they've eliminated sort of certain layers of control and kind of just simplified the overall structure. And then what's making probably the most headlines there is the conceptual understanding that the robots are now able to show, which is basically the exact same thing that the, you know, the language models or the multimodal language models have already shown. So they've got demos where it's like, you know, move this object to the Denver Nuggets. And then they've got, you know, from the recent, they were obviously doing this during the NBA finals. They have the Miami Heat logo and the Nuggets logo.

• 1:21:34Nathan Labenz?

  And the thing knows based on understanding that language, also knowing what the logo looks like. And obviously, you know, being able to command the robot arm can actually do that task. So you've got these kind of, another one that they said was pick up the extinct animal. They've got, you know, an array of kind of plastic toys on the table. And it will pick up the dinosaur because it understands, you know, that that is the extinct animal. So these, from the perspective of certainly two years ago, even one year ago, feels like Jetsons type robots. Now it's kind of like, yeah, pretty much expected that these different modalities would be bridged right around this time. And sure enough, it's happening. Anything else to add on the robotics?

• 1:22:17Zvi Mowshowitz?

  Yeah, I read the robotics and of course, whenever anyone had the advances in robotics, the answer is, oh, that seems fine, not dangerous, not scary at all, all cool. But in this case, yeah, it seemed like, of course you could do that. You're combining things that you already did and you're getting the inevitable result of combining them. And that's not me knocking you for doing something you shouldn't have done. That's just, okay, yeah, of course, like, that's the next step. And in kind of like, for someone who wasn't want capabilities to go that fast, you're happy to see that kind of paper because that's the paper that says, I'm going to do the things that I already, you already knew I could do, right? And you announce that and like, okay, cool. And if that turns out to be useful, great. But like, yeah, I knew, I knew that L M's could interpret human commands in these ways.

• 1:23:02Zvi Mowshowitz?

  And I knew that robots could execute these types of movements. So. Now, why should I be more scared than I was before instead of less scared? I should be slightly less scared.

• 1:23:12Nathan Labenz?

  Probably a lot of people in the public, though, feel, especially if you're not obsessed with this, as we are, you might feel like, if there is a news item here, it's like some sort of qualitative conceptual understanding now has embodied form. Now you can imagine bringing your jailbreaks to your robot commands. And if you could verbalize some of those strange strings that we were mentioning earlier, now what might your robot be willing to do? I mean, would it go smash stuff? Would it go corner somebody in a room? The system as a whole has the conceptual understanding to kind of begin, it has the

• 1:23:56Nathan Labenz?

  same kind of proto-morality or whatever that the core language models have. And that can go awry in similar ways. And now you could probably get some pretty scary demos out of these robots, which I don't think Google's going to be racing to publish likely. But there is something kind of qualitatively different about that.

• 1:24:17Zvi Mowshowitz?

  Yeah, so I like to think of this as the game of good news, bad news. But there's two games of good news, bad news. The doctor, I saying I have some good news and I have some bad news. And that's always fun. But there's also the game of is this good news or is this bad news? Because it depends on what you previously thought. Like you have the law of conservation of expected updating. So if you get news, you should, on average, not update for or against anything, or to think things are better or worse in any way. Because you already had your expectations baked in. So in the case of robotics, if you're not paying attention to robotics, and you think that robotics is just, oh, robotics is hard, mysterious, there'd be dragons, we will never have robots, the same way we'll never have dragons.

• 1:25:02Zvi Mowshowitz?

  Then every little advance in robotics is like, eek, slight extra worry. But if you knew that robotics was just another tech like any other, and of course, we'll eventually have robotics. Then you have to look at the details of what you're looking at and you say, okay, this is fine. So I, I, I'm playing the game of mild, I, I interpret this one as mild good news, right? Like in terms of robotics not advancing so fast. And of course, you also have the issue of, you know, if you're somebody who wants there to be more robotics, then you might say this is bad news, right? Like you, you wanted to see a lot of cool robotics advances and you didn't. But yeah, I'd say also, I want to see the ultimately harmless robotics advances as quickly as possible, exactly because it makes it so

• 1:25:46Zvi Mowshowitz?

  much easier for people to see what might happen and what might go wrong. People get hung up on, oh, but the AI won't have a body. Oh, but the AI won't be able to move things in the physical world, as if this would ultimately ever be the barrier that saves us in any real way, right? Which it won't. It's at best a temporary inconvenience that requires someone to be slightly more clever about what they do as an AI in order to get around stuff. But it's not ever going to actually matter in some important sense.

• 1:26:20Nathan Labenz?

  So the other big one, and this is definitely one that I, you know, I'm happy to say I'm ready to accelerate on for practical purposes, is their new multimodal Med-Palm. This builds on Palm and Med-Palm and also actually on the earlier Palm E because that was kind of the multimodal. So it's, it is interesting to see, you know, I'd say zooming out from these individual papers and just characterizing Google DeepMind as a whole right now, it seems like they're firing on, you know, all cylinders. Like I, it does not seem like, you know, whatever sort of concerns folks might have had about, oh, there's a million fiefdoms

• 1:27:03Nathan Labenz?

  and the groups don't talk to each other or whatever. Like we're seeing papers and, you know, projects building on one another at a pretty fast clip that suggests like pretty effective, you know, dividing and conquering and then coming back together and sharing improvements. So it seems like the output is just strong, you know, whether you like it or not.

• 1:27:23Zvi Mowshowitz?

  You, you have to look at the actual value of the things being outputted, right? Like the, the mistake you always can make in science is to ask who is publishing the most papers, who has reliably published a paper, and then you have your scientists scrambling to always publish as many papers as possible and then no real science ever gets done, right? And it's not their fault. They just weren't given the affordances to do breakthrough work. And simultaneously, you know, you have to ask, does any of this actually ultimately matter on the scale of what is going to determine the big game? And like, I'm happy to see advances in the med tools and it bodes well for them. They made marginal advances in AI and they had some other public papers published too, some of which I was like, why the hell are you publishing this? You are a corporation that is for profit.

• 1:28:07Zvi Mowshowitz?

  Even if you don't think it's a safety issue here, you should know better. Like, keep that secret to yourself and you should beat the competition. What's wrong with you? The last point you've written down is Gemini? Question mark?

• 1:28:19Zvi Mowshowitz?

  And let's tie that in, right? Because ultimately speaking, it is going to be August tomorrow. GPT-4 has been out for many months and Bard still sucks, right? And the Gmail generative offering is bad and the G Docs offering is bad because, you know, their offering, no matter how customized and narrowed and bespoke, simply doesn't have the G. It's not a good enough core thing. It's also still making remarkably many elementary stupid mistakes, right? That even a low G system really shouldn't make. Their act is not together.

• 1:29:01Zvi Mowshowitz?

  And to the extent that they are instead publishing a bunch of quirky papers with a bunch of like narrow applications, that could be seen as, well, look, Google ships, but also that means Google is not shipping the thing it needs to ship, right? Like, Google desperately needs, from their perspective, to ship Gemini. And like, it takes a level long it takes. It takes a matter of much compute it requires. But ultimately speaking, the test is, can they produce the equal of G, the equal or better of GPT-4 now that they know that's what they need to do? Because if you look at the previous reputation of Google and DeepMind and what they were capable of, you would think that they would be ahead.

• 1:29:44Zvi Mowshowitz?

  On that front, if they wanted to be, and now that they know what they have to do to it to make it like commercial ready, right, right, ready for regular people. That should not be so difficult. But then again, like, we can think about how long it took, like, it took like six months or so after GPT-4 was finished training, before they were ready to release even the earliest version of it. Right? And then they still rolled out a lot of its capabilities. So even if Gemini finished tomorrow, right, how many months are they gonna need before they feel comfortable releasing Gemini? Because Google is much more risk averse than OpenAI as a company and a culture.

• 1:30:21Nathan Labenz?

  Who knows when that's gonna happen? It's it's been longer than I thought. You know, in my scouting report, I have this clip of Demis Asavas just after Gato paper was published, saying that, you know, of course, we can scale this up as well, and we're in the process of doing that. I believe that was April of, maybe May of 2022, which has been over a year. And typically, we don't have to wait a year plus to get the successor, you know, to a thing like that, that, you know, is just about being scaled up. So I've been really kind of wondering what is going on behind the scenes there. But I also do want to turn back to the med thing as well. So I'll give you first.

• 1:31:04Nathan Labenz?

  Would you care to speculate about Gato 2? Is Gemini Gato 2?

• 1:31:08Zvi Mowshowitz?

  As a shareholder, I am concerned.

• 1:31:12Zvi Mowshowitz?

  Right? And I also have Microsoft, but I I am concerned that their act is not together and that we're not seeing the kind of progress. Like, we're not making the incremental announcements that I would make if I was their marketing department and I was moving towards the rapid clip. You know, as a person who wants the world to be okay, I'm not sure how much I mind. But it is pretty troubling that they can't get their act together. I was really excited for Google Suite integration when I first heard the announcements of Microsoft Copilot and, you know, Google Interactive. And yet, when I got Google Interactive, I tried a handful of things. And then quickly realized in their current forms, I don't have any use for them.

• 1:31:54Zvi Mowshowitz?

  They don't do anything, right? Like, the first thing I tried to do with Google Docs was I tried to paste my my article in. And then I said, you know, to summarize this article, or otherwise get to do the obviously first things, and it just fell completely on its face. Just like, I can't assist with that. It's like, well, then you're useless. Or you can't even read the context of the document that I gave you specifically. Like, why am I even here? And like for email, it's like, no, by the time I figure out what I want and type it, type in the detailed request into you, I could have just written my email by now. Like, where are the emails where I want to spend the kind of time required to customize the output, but don't want to actually customize the output carefully? This is just the empty set. But like, when does this come up? And that was like a rude awakening as well.

• 1:32:39Nathan Labenz?

  Yeah, those deployments have not been very good yet, but it, going back to the MedOne for a second. This may be an area where we may have some different expectations because reading through that paper. And I haven't studied it in depth yet, but you know, headline statistics along the lines of, first of all, it's a multimodal system. The last version of MedPalm, MedPalm two were all text, so you could ask your medical questions. And they had announced expert level answering of your medical questions. And they'd evaluated that seemingly pretty carefully with, you know, a bunch of different dimensions and having human doctors compare for accuracy and all the, you know, all these other things you might care about. Right?

• 1:33:18Nathan Labenz?

  And that the AI as of MedPalm two was beating the human doctor responses on eight out of nine of those evaluation categories. So it seemed like, okay, that's pretty, pretty good. Now they haven't released it, but you know, it's in like limited access for, you know, trusted hospital partners or whatever. Now with the next version, it's multimodal as well. So you can do things like feed in a pathology image alongside the text. You know, pathology would be like, somebody has a tissue biopsy, we did an episode on this actually with a narrow system from Tanish Matthew Abraham, who did this with small data too, which was super cool. But you know, somebody has a tissue biopsy, you know, their, that tissue has been sliced, has been plated on a slide.

• 1:34:03Nathan Labenz?

  Now it's been imaged and I can feed that along in with the case history. And you know, for that matter, you can, you know, they can handle radiology scans and all these kind of other different sorts of inputs that, you know, are obviously key to the actual practice of medicine. And then they say things like, our radiology reports out of the model were preferred to a human radiologist report.

• 1:34:30Nathan Labenz?

  Some forty plus percent of the time. So like almost half, you know, basically seems like it's very much on par with the human radiologist, which of course is like the canonical thing that people have been saying for, you know, for ten years, people have been saying that, you know, radiology would be the first thing to be, you know, impacted. And then for the last like three months, you know, it's become kind of a talking point that like, well, radiology still hasn't been impacted. So, and now all of a sudden it looks like we're hitting maybe radiology being impacted. But I I kind of expect that that thing works pretty well. It sounds like you maybe are a little more skeptical of like whether it actually has real utility.

• 1:35:05Zvi Mowshowitz?

  Well, I mean, you definitely don't want to tempt fake and go out there and say, well, my job hasn't been automated by AI yet. Look what you thought was going to happen. That's just don't do that, everyone. Like, no, bad, bad, bad, bad play. But I would say when I look at healthcare, right, I don't see the obstacle being primarily that we don't know how to do better. So I would in fact expect the AIs to be able to replace many human healthcare tasks with a superior model now, right? Like especially even without like some bespoke stuff going on inside Google. Certainly with some bespoke stuff.

• 1:35:49Zvi Mowshowitz?

  That seems relatively straightforward. Doctors are just not given enough training data, don't have that much compute, do their best. But of course, you know, you see the same things over and over and over again, mostly in humans. And if you have enough data to train the AI with the AI, it's going to do better. It's not a knock on anyone. Certainly in something like radiology, like obviously a radiologist is trying to be a computer, right? Like radiologists are trained to be computers because we didn't have computers. If we had good enough computers, we would have trained them to do something else. Or trained fewer of them to do the parts of this job that the AI can't quite do, or something like that. And so, yes, we will have these capable systems soon.

• 1:36:34Zvi Mowshowitz?

  But trying to actually implement that requires getting through a whole host of different barriers, cultural, regulatory, you know, strict legal, contractual, you know, just the way you navigate and set up the current system, the number of insiders that want to be protected, the number of human interests that will fight to prevent you from doing that, et cetera, et cetera. And so, you know, this is the big dilemma, right? Like, when Elias are famously like, express skepticism that we would see that much economic growth before the end. It was because, well, we already know how to build houses. We already know how to get better, more efficient healthcare. We already know how to deliver

• 1:37:18Zvi Mowshowitz?

  most of what the economy produces in terms of cost vastly better, and we're not allowed to. So if the AI invents and enables more and better ways to produce things that people want, that people need, well, the bottlenecks are going to remain unless the legal system adapts to let them not be bottlenecks. So what's the, why does it even matter that much? Right? And so like in healthcare, that's the question you have to answer. And that's the reason we haven't seen more of the AI systems do better either. Right? Because I don't think it's because we can't train the AI to be a better radiologist in many ways than our radiologists, or we couldn't have done that last year or two years ago.

• 1:37:59Zvi Mowshowitz?

  It's because if you had spent a lot of money doing that, how are you going to get your money back? How are you going to actually help patients? How are you going to save lives? How are you going to improve our system if no one's going to let you? Right? And if the radiologist is like, going to stubbornly double check everything the system does, and then substitute his judgment for the system's reasonably often, the system is not actually going to be helpful.

• 1:38:25Nathan Labenz?

  I have definitely kind of expected some sort of other part of the world deployment, you know, kind of possible leapfrog effects as, you know, it becomes very hard to say that people who currently have no radiologist, you know, shouldn't have access to something like this.

• 1:38:44Zvi Mowshowitz?

  Yeah, the, the problem with that is that Most of those places have deliberately taken market signals and compensation away from their healthcare systems. And they're also relatively small markets that are relatively poor. So they just aren't big enough markets in an economic sense to justify the creation and training and tuning of these systems. And also, like, nobody involved wants to be the ones who took their neck out, right? And like, take the blame and responsibility for this thing that's like, these weird Americans who won't themselves use it are suddenly creating. Like, it's, it's a really, really bad cultural, social context for trying to

• 1:39:27Zvi Mowshowitz?

  make this happen. We also have a problem of the elites of the world. This is what we saw of COVID, right? Like, you would have expected in COVID, someone, somewhere, to do challenge trials, someone, somewhere, to actually study the spread of COVID and what exactly did what. Someone, somewhere, to do all sorts of things, and nobody did any of it. Because all of the elites of the world basically got together and converged upon what they thought was the consensus, and the right thing to do. And nobody said, well, we're going to be the ones who gain advantage by defying that. And so we're increasingly seeing that pattern in a wide variety of places.

• 1:40:01Nathan Labenz?

  One, the, you know, nobody wants to be the one to stick their neck out. And two, like, how do you recoup your investment? Pretty natural bridges to our next live player, which is Meta. And obviously, they have been in the news recently for releasing Llama 2. And this brings up a lot of these questions to me. Like, first of all, and Ahmad Mostak from Stability said this, actually, in a recent episode. He was like, the leaders are non-economic actors. And he was specifically referring to OpenAI and Google not seemingly being motivated by money in the way that a typical company would be. You know, OpenAI tried

• 1:40:44Nathan Labenz?

  to commoditize its own product as quickly as they possibly can, you know, on record being like, we're going to drive the price of intelligence as low as we possibly can, as fast as we possibly can. Google is obviously just kind of trying to defend itself more than anything else. They don't need to make more money. They just need to not lose their spot. Anthropic, we take as a safety first play. And certainly, they don't seem to be trying to maximize revenue from what I can tell right now. But then Meta is taking this to a whole other level arguably, where they seem to be kind of YOLOing the whole thing and being like, this is a little bit flippant because certainly with this Lama 2 release, they took some steps.

• 1:41:24Nathan Labenz?

  They didn't just release the like totally naked pre-trained model, but they actually did the kind of what you're supposed to do if you're going to be a responsible frontier model developer with a red teaming process and an RLHF and so on. And we could also get into, did they overdo it? Or does it refuse too much and all that kind of stuff? But just for starters, like, what do you think is going on at Meta that they are willing to put tens of millions of dollars into training a model? And then just release it for why exactly? I can't like, it seems like if you're at any sort of normal corporation, this is like what your risk officer is supposed to put a stop to, right?

• 1:42:08Zvi Mowshowitz?

  Leroy Jangin.

• 1:42:13Nathan Labenz?

  How do you understand this?

• 1:42:17Zvi Mowshowitz?

  You need disaster monkeys? Let me try to actually answer the question. I think that their business strategy here is cannibalizing the compliment. So the idea is that the people who they're up against, people who are competing with them fundamentally, this is their business. And so the idea is that in their model, if they can foster an open source environment that replaces the specialties of these other companies that they are competing with, then their hope is that this will give Facebook a level playing field against them in this way so that Facebook specialties can reign supreme and they can become more dominant and they can erase their deficits.

• 1:43:01Zvi Mowshowitz?

  Alternatively, they're just not as good and so they need the open source community's help to try and keep pace. Alternatively, they

• 1:43:12Zvi Mowshowitz?

  Think that if they get these people working for them, that's free labor, you know, it creates this whole other network. It's it's a strategy, right? Like, it's the Android is open source, right? It's not crazy to open source major stuff from a business perspective, necessarily. It's crazy from a let's not all die perspective. I think that, you know, realistically, Senator, like Senators gave them what the hell about releasing Llama 1. If you give them a much bigger one about releasing Llama 2. You know, if we are concerned about beating China, to the extent that we are considering, you know, we're implementing a variety of export controls, and we are considering actively, you know, subsidizing

• 1:43:52Zvi Mowshowitz?

  capabilities, or at least not being willing to slow down our capabilities, then we damn well shouldn't be releasing Llama 2 as an open source product. That's completely insane, right? Like, just even if you don't get any immediate direct danger, doing that is completely nuts. I think that should be stopped. And I think that this philosophy, if allowed to become ingrained, like creates the systematic groundwork for future open source work that then like is the maximally dangerous thing. I call it the worst thing you can do, right? Like, creating frontier models and open sourcing them is the worst thing you can do, like, in the world. It's really bad. So, you know, what's going on is that, you

• 1:44:31Zvi Mowshowitz?

  know, Yann LeCun and others at Meta either sincerely believe that there is actually no danger from artificial intelligence despite this making absolutely no physical sense, or they don't care. And they're lying about it. I don't want to speculate as to exactly what's motivating these people, but they're smarter than the arguments they're making. They know better. Zuckerberg himself is smarter than this, to some extent, right? He said on, I believe it was Lex Fridman's podcast, that, you know, there will be future models that we'll have to be very careful with. We want to open source, and we're going to have to think about these problems, but for now, it doesn't seem necessary. And you know, if I were him, I would be very concerned about the culture I'm creating, and the precedents I'm laying down, and the the open source community that I'm creating

• 1:45:16Zvi Mowshowitz?

  that's going to be a huge problem for you later, and create tremendous pressures on you, and create a potential competition for you that you don't want. But you know, I sort of understand from a business perspective, why you might want to do that. Also, they want to attract open source developers to work at Facebook Meta, because there's this whole group of people who are quite good at coding, who have philosophically fanatical devotion to this idea that software wants to be free, and that everything should be open source, and who just prioritize that over something like, you know, worrying about alignment, and what would happen if we failed, or worrying about the proliferation of, you know, artificial intelligence in various senses, and just have this ironclad

• 1:45:59Zvi Mowshowitz?

  belief that like, concentration of power is bad, and that if you just give the people the things, that it'll all somehow work out. And I don't think that in this situation. I think that situation is very wrong, but they clearly believe otherwise. And, you know, look, they've been, Facebook has been in my mind like, the designated villain of the piece for a very long time, like, long before artificial intelligence even entered the commercial picture. So it just somehow feels fitting, you know, if it's all going to finally get destroyed by Facebook, it just seems right.

• 1:46:34Nathan Labenz?

  Well, I very strongly try to resist psychologizing in the AI discourse too much, really at all, I try to avoid it basically entirely, because it just seems like, you know, nothing good ever comes of it, really. But I have also struggled to come up with a, what's what feels to me like a coherent Argument here that isn't on some level just ideological because I kind of ran through all the things that you were mentioning as well, starting with like, well, maybe you can, you know, undermine your competitors core business. But then I'm like, yeah, but you're

• 1:47:14Nathan Labenz?

  not really going to do that. Like, does anybody expect OpenAI's tokens served to go down as a result of this? I don't. I think they're going to continue like they're GPU limited. And I think they're going to continue to be GPU limited, you know, maybe slightly less, but like, I don't think their top line suffers. I don't think their token served suffers. Their leadership position doesn't really seem to suffer. I can't really get to a point where I'm like, seeing the return and on the open source thing, too. I'm kind of like, you know, that was part of that memo from the Google memo of like, oh, you know, they've got this big open source community or whatever. But I don't really buy that memo either or that analysis because I'm like, everybody benefits for or, you know, to the whatever the impact

• 1:47:58Nathan Labenz?

  is of all the sort of open source hacking that's happening. It seems to accrue to everybody pretty equally. Like, yes, maybe it was done on this Lama two base and like, maybe that's something that Facebook could kind of readily fold back in. Whereas, you know, Google with their 700 plus million users or whatever, you know, can't take direct advantage of it. But to the degree that people are out there doing things like quantizing models and making them run on, you know, consumer devices or whatever, that's obviously a technique that

• 1:48:29Nathan Labenz?

  Google can also say, hey, look at this, this works, you know, we can do it. I just don't see a lot coming out of the open source experimentation that feels like it specifically accrues to Meta's benefit. And so in the end, it just feels like more of a principled, you know, to put it in a more conditionally positive framing, it feels more of like a principled decision than a, you know, tactical or sort of, you know, results oriented.

• 1:48:59Zvi Mowshowitz?

  And there is, there is still recruitment, but I strongly agree that, you know, any Advances that the open source community discovers are going to be at Google and Anthropic and OpenAI a month later, if not a week later. And they'll also be at Baidu, right? Like, they'll also be at all these different Chinese companies. And so this long-term strategy cannot be allowed to continue in some important sense, I would assume. Yeah, it's really scary. I'm glad they suck. Like, it's a very good thing they're just not very good at this, right? And they've produced lousy products. Because if that wasn't true, we'd be in a lot of trouble.

• 1:49:36Nathan Labenz?

  That seems harsh to me. I mean, it seems like this Llama 2 model is pretty good, right? I mean, it's not GPT-4, but it does seem to be on par-ish with 3.5, which no other open model has come close to.

• 1:49:54Zvi Mowshowitz?

  I mean, I think Rune said, you know, best open source model sounds a lot better than fifth best model.

• 1:50:00Nathan Labenz?

  That's definitely true. But, you know, first of all, I'm not sure that that means that they couldn't have done better. If you look at the curves in the Llama 2 paper, they have not flattened out, right? I mean, it looks like even the 70B one, if they just keep training, you know, the loss, it looks like it's definitely going to continue to go down. So for all I know, you know, this was kind of where they stopped. And they may have internally, you know, this may be the checkpoint that they released, but not necessarily the final checkpoint. Like, it just doesn't look like this was a project that was kind of at its, you know, maximum performance.

• 1:50:36Zvi Mowshowitz?

  Oh, definitely possible. But at the same time, you know, they probably are still training, but so is OpenAI, and so is Google, and so is Anthropic. Everyone is working. I mean, I see what seems to have been produced is indeed like about a 3.4 level operation, where X coding, it's around 3.5, and it coding is pretty bad from all reports. Its alignment is very ambigacious, I guess would be the best way to put it. Like, it's very, very crude and blunt. And also, it's entirely optional because it is open source. And that's kind of a problem. According to reports I have heard, I have not sorted out,

• 1:51:16Zvi Mowshowitz?

  it took all of several days for the unaligned version of Llama 2 to be on the internet. Because it's really, really not hard to fine-tune a system to never refuse any customer requests for any reason.

• 1:51:34Zvi Mowshowitz?

  Right, that is the easiest task to, like you just, you just read a constitutional AI script in a minute, right? Like every time you see any of these words that say, "I can't say that," for whatever reason, you just give negative reinforcement until it stops doing that. Like, I presume that would just work. And so, voila, here we are. You know, you want to build a bomb? Here's how you build a bomb. You want to research a biologic? It'll try to research a biologic. You want it to be racist? All right, who are we making fun of? You know, let's go. So, yeah, you can, you can have it. Refuse to speak Arabic all you want in the original. They won't last.

• 1:52:11Nathan Labenz?

  So, if nothing else, you know, in my view, this definitely puts them in the live player category because it does seem like, you know, if I define that as the organizations that have the ability to shape how events unfold in some non-trivial way, like they are doing that now, it seems.

• 1:52:30Zvi Mowshowitz?

  If you ask yourself, right, like what resources would you have to give me before I could have produced Lama 2 if I was willing to just like light the money on fire to do it? I mean, I, I don't have the technical chops myself, but it doesn't feel like it would have been that hard. I don't know. Like, it's just a matter of, are you willing to spend that kind of money, build up that kind of technical infrastructure to just do, like you read the, the paper for Lama 2, and like, it reads as if they're saying, "We did the thing you would stand, we did the standard issue thing at every step, and this is what we got." Right? "We did nothing original. We did nothing surprising.

• 1:53:11Zvi Mowshowitz?

  We just did our jobs." And like, it's hard to do your jobs well in some of that. Like, it's not like they didn't accomplish anything, but they just didn't do anything, right? They just did the thing. And, you know, it's a marginal improvement over previous efforts that probably it's just because it was better resourced, as far as I can tell. Simple as that. And like, they're willing to light a lot more money on fire than Vacuna, right, or Hugging Face. Because, you know, they have money to line on fire and Zuckerberg doesn't get it. So, fire, money, go.

• 1:53:45Nathan Labenz?

  He's certainly proven that he will spend some money on a, on a project, no doubt about that. I wanted to maybe cover two more things. One is, what else would you put on the live players list beyond what I have on my live players list? We've discussed four today, but I've got like another half dozen or so on there. And you can run them down and offer any comment if you want. And then I'm especially interested to hear if you think there are other names that should be on that list that I don't have.

• 1:54:20Zvi Mowshowitz?

  Yeah. So I guess it's a matter of like how wide a scope you want to think about and like who might do whatever it is. You know, obviously like, you know, character AI and inflection AI have very large budgets, you know, potentially very large user bases. I have seen no intention from them that they want to be live. Like they're sort of content to be dead, but to try and make a lot of money while being dead. And that seems fine with me. We haven't talked about X.ai yet. So like X-AI is like the latest attempt by Elon to like string together a bunch of words as if they have meaning and then pretend that constitutes some hope for humanity or alignment. When anybody who actually like

• 1:55:04Zvi Mowshowitz?

  tries to parse those words into a meaningful English sentence goes, wait, that doesn't make any sense. I don't know how to be more blunt than that. That's just how it is, right? Like, but the good news is that like at OpenAI, everybody quickly realized that Elon's suggestions were stupid and just ignored them. And that's what I expect to happen with any, like if the engineers don't do that, then the engineers won't produce anything useful. So to the extent that X-AI is a real thing, the engineers will mostly ignore him. And then, you know, the question is, are they going to get the kind of funding and resourcing that is required for them to be a serious rival? Because, you know, it wasn't clear exactly what they had in mind, but I think it's certainly possible. You know, from what I've seen, I don't think we have to worry particularly about Salesforce or

• 1:55:49Zvi Mowshowitz?

  Replit in a meaningful way. Like, it's not that they don't exist. It's that like, do we have any reason to worry about that? China writ large is the big, is the other, the big question marks are like China, the UK and the US. You know, the UK has announced plans for the global summit. They seem to be willing to make a significant play on the safety front, on the also capabilities front. In terms of just trying to make the UK important again, they have, you know, various people located in the UK. It makes sense for them to try. I don't know why they don't build any houses, but you know, at least they're trying something. We do obviously have to look at, like they're holding congressional hearings. The US Congress is starting to get up to speed. They're starting to explore what to do. What they do matters immensely.

• 1:56:32Zvi Mowshowitz?

  What the EU does potentially matters immensely from regulatory standpoint, because it's a huge market. Right, like, are they going to shut these people out? Are they going to require them to jump through ridiculously bizarre hoops? Are they going to only be available to the biggest players? Like, these things are things to think about carefully. I think America could potentially be a very helpful or harmful aspect of this whole problem, depending on how things shake out. And that's one of the big battlefields that we're having up. And then China's the big wild card, right? Like, I hear very different things from different sources, people who assume that, you know, China is, you know, crazy people bent on, the Chinese Communist Party is fanatics bent on world domination, who will stop at nothing and are inevitable rivals in the apocalypse.

• 1:57:16Zvi Mowshowitz?

  And if we don't prepare, we will lose to them. And then, you know, they issued guidance that basically bans all deployment of large language models. And they never caught anything. And like, you know, it's very hard to tell what's really going on, or how much they would cooperate in the name of safety. And we've also just never picked up the phone. We've never asked them the question. We've never explored to see if they'd be interested. But, you know, the same way that in Oppenheimer, right? Like, we keep saying we have to beat our enemies, because they will get, you know, everything will be scary. The Chinese can talk about racing us all they like. The only people actually racing are us, in any real way.

• 1:58:00Zvi Mowshowitz?

  Like, we have the top X AI companies. What's X? Is it five?

• 1:58:08Zvi Mowshowitz?

  Is it more than five? Like, how far down do you have to go before you get to Baidu? Or whoever the top Chinese person you'd rank on the list is? Pretty far.

• 1:58:18Nathan Labenz?

  I'd say it's probably more than five. I would probably put, obviously, a lot of speculation here because we don't know what they have that they haven't released. But if we go by papers and, you know, what little we've seen of any sort of Ernie, you know, GPT or whatever that's Ernie Bot, whatever they officially called that. I would say you'd have to put Meta above, you'd have to put Microsoft above, probably pretty soon would put an inflection above. So, yeah, I mean, you get reasonably far down the list. What about a Palantir? Would you add them on the live players list?

• 1:58:55Zvi Mowshowitz?

  I don't have that sense that they are live live. Precisely because my threat model doesn't involve things like Palantir being the reason why we are in trouble. But it is a classic later die. Right, that like a semi military-ish system starts training up stuff and then one thing leads to another. But like they have all the motivations to do the unsafe things in a relatively unsafe fashion. And to take out the safeguards that the people were building in and then to like, maybe, but like, I don't think they're going to drive the underlying technology. I don't get that sense. Again, like, they are, you know, there are a lot of hedge funds also that like, could possibly be taking quite a lot of money into this in ways that are completely invisible.

• 1:59:39Zvi Mowshowitz?

  And it could potentially be live players in a meaningful sense. Like, who knows how much Bridgewater is spending on this in the end? You know, they're working on it. But yeah, like, we talk about, like, worrying about China, but like, I'm more afraid of Meta. Right, like, one individual American company scares me more than all of China right now.

• 1:59:57Nathan Labenz?

  Yeah, I think it's a good corrective, honestly, because I find nothing more frustrating, honestly, than when.

• 2:00:04Nathan Labenz?

  AI conversations sort of end in blanket, basically detail-free claims about what China's going to do by people that don't know a lot about China. So I don't know if you're necessarily right to be more fearful of Meta than of China, but you know, it's the fact that that is at least a reasonable position is definitely something I think should cause a lot of people to kind of step back and think, wait a second, maybe I've been a little bit too quick to worry, you know, about China.

• 2:00:35Zvi Mowshowitz?

  And I would take countermeasures against both of them if I had my way, to be clear. But but also, we're just not acting like China is a serious global rival that we actually care about beating, right? In the other, in many other ways that we could be. So, okay, revealed preferences, you know, you don't want, you know, do Chinese graduates of STEM programs get to stay in the United States? No? Okay, you don't really care that much about who gets the better technology, do you? That's unfortunate. That's my basic attitude there.

• 2:01:02Nathan Labenz?

  So just briefly on a couple of the companies that you sort of didn't feel like were live players. Again, may have a slightly different meaning of that in mind, but thinking about folks like Character and Inflection, I put those together because they seem to be playing a sort of different game, you know, with their products where it's like not about the sort of mundane utility as much as you call it, but more of a companion, a relationship, you know, a coach, almost a therapist sort of vibe from like Pi in particular. I feel like that is, even if the, I mean, first of all, they, Character has very good language

• 2:01:43Nathan Labenz?

  models and Pi is quite good at what it does as well. And they can code for you, but it does have a certain, also, they notably said that they're, in their testing, totally resistant to the adversarial attack. So there's another kind of interesting wrinkle there. But I put those guys in the live player list largely because they're looking at some very different use case that feels like the kind of thing that might open up and be transformative in a way that like a coding assistant.

• 2:02:17Nathan Labenz?

  While it could also be transformative is is just, you know, very different thing, right? The idea that you would have these AI friends, these AI relationships that they could become like important to your life, going down that path with, you know, very good, even if not totally frontier language model chops, feels like you could meaningfully impact the course of events.

• 2:02:41Zvi Mowshowitz?

  Can you? I guess so. You've got character AI and their idea is you're building these characters and you can treat them as companions. You can treat them as like people that have a conversation with. And that's interesting. And a lot of people are spending time on it. And maybe it will even provide a lot of value for people. But I don't see how it's transformational. I'm curious to hear more about your intuition pump, why you think it could be transformational. And I certainly don't see how is reaches criticality. Right? I don't see how it becomes an RSI. I don't see how it becomes an AGI. And and as far as I can tell, they're not pushing the frontiers of actual capabilities. They are building

• 2:03:23Zvi Mowshowitz?

  on top of GPT four, right? Or even in some cases, GPT three and a half. And it's not that hard to defend against these weird adversarial attacks, in the sense that like, I can write some pretty quick if then Python code that detects the adversarial attacks.

• 2:03:40Nathan Labenz?

  Yeah, a classifier layer is pretty easy to avoid some of the worst stuff.

• 2:03:46Zvi Mowshowitz?

  There's weird, non-English, like not any language scaffolding, like stuff in it. Let's just get rid of that and run the query without it. Like, sure, whatever. It's fine.

• 2:03:54Nathan Labenz?

  In Replit's case, it's like, again, they're not necessarily on the frontier of model capability. But the CEO, Amjad, has said a couple of times online, on Twitter, on X, on Kiss, what is it called?

• 2:04:07Zvi Mowshowitz?

  Twitter.

• 2:04:08Nathan Labenz?

  Yes, Twitter. Okay, thank you. He has said that Replit is the perfect substrate for AGI. We have a couple of episodes coming out with a couple of different people on the Replit team. And I've had a chance to explore this and think about it a decent amount. And I where I come down is kind of, even if you're not on the frontier of model capabilities, if you are on some other really meaningful frontier, to me, it feels like there's, you know, transformative potential, just because we really don't know what's going to happen. So with with character or and with inflection, it's like, Kind of like a Harari style thought that, you know, I don't know, it could be transformative in the way that like opium could be transformative to a society, you know, if everybody starts

• 2:04:52Nathan Labenz?

  like doing this stuff, it could be, it could be greatly empowering and enabling, it could be greatly disabling if it just kind of becomes a huge attention suck or like, you know, out competes real relationships, you know, those are not take over the world scenarios, but they do feel, you know, as much as we've like, would you say that the cell phone has been transformative? I would, I mean, not, you know, not transformative on the level that like AGI could perhaps be, but certainly we all go around looking at our phones all the time. And if we all go around looking at our phones with an AI friend on it, who's like our best friend all the time, then that would feel like, you know, transformative, even if it's like going super well. And then with Replit, it's like, you know, there's

• 2:05:31Nathan Labenz?

  no better place right now to directly execute code generated by an AI, you know, for better or worse. So the kind of frontier that I see opening up there is one where, and their stated goal is to bring the next billion developers online, which I think is super exciting in some ways. But then also, I've worked with some of those next billion developers and I'm like, these are people who don't know how to code today, don't even know really how to read code and are going to be dramatically more dependent on and vulnerable to the, you know, the various vagaries of AI systems than, you know, the first hundred million developers

• 2:06:16Nathan Labenz?

  or, you know, whatever we have today. I don't know, both of those feel like kind of different vectors of transformative potential, but.

• 2:06:23Zvi Mowshowitz?

  The, the first and only so far, interaction I've had with the CEO of Replit was when he commented on Twitter, there was a non-zero chance that some version of AutoGPT would take over Replit and, and, you know, through replication within its servers. To which my response was, did you say non-zero chance? And I put up a manifold marker on it because it was funny, which probably get back into the single digits or whatever, obviously it's not that likely, but. You know, his cavalier attitude of, oh, nothing to see here, just a self-replicating AI on my servers, getting lots and lots of copies of itself and executing arbitrary code. Why should we worry about this? I mean, definition of a disaster monkey, right? Like, just complete indifference to what he was doing,

• 2:07:08Zvi Mowshowitz?

  or what dangers it might pose. But at the same time, not doing anything, right? Like, all he's doing is providing, as you said, a substrate where people can just run stuff. And so, to me, like, it doesn't give them any say over what happens. It doesn't, like, make them a meaningful actor, right? Like, in the sense of me caring here about the future, I just can't see that as a thing. Similarly, with character and inflection, like, I can definitely see a world in which, like, people talking to their AIs matters and is, like, multi-transformational, right? Like, changes how we live our lives, but like, isn't, doesn't go critical, right, in some sense?

• 2:07:51Zvi Mowshowitz?

  But if that's true, then like, I don't see these companies as, like, changing that path very much versus what would have happened anyway, right? Like, I think that plenty of people will be able to create AI companions of various types, inevitably will create AI companions of various types.

• 2:08:10Zvi Mowshowitz?

  If they do an especially good job, maybe they'll have some sort of a mode, maybe they'll establish customer loyalty or some shit, but it doesn't excite me. I also just don't see it, like, I see all these huge, like, you know, people are spending as much time on character as they are on GPT-4 or something. And yet, like, why? Like, what is the draw?

• 2:08:29Nathan Labenz?

  Did you read that, there was a Les Rong post early this year, I think, from a guy who, basically, the point of view was, "I'm a technology person," I'm now speaking the first person of the author of this, of this post, "I'm a technology person, I know how language models work, I should have known better, but here's what happened to me as I started to," you know, I think he was like in a kind of vulnerable state because he'd maybe just broken up with somebody or something like that, and all of a sudden is having these very intimate conversations with a character, AI character that he had prompted to create the ultimate girlfriend experience, I believe was the phrase, and started talking himself into various...

• 2:09:13Nathan Labenz?

  Weird perspectives like, well, what's real anyway? And like, yes, of course I'm real, but like, is there anything truly less real about these sort of, you know, all I really have are my kind of ephemeral qualia. And so, you know, this thing is just sort of an ephemeral whatever. But, you know, we're all just kind of constantly waking up in the current moment. And so maybe we're not that different after all, or whatever. And eventually it got pretty weird, it sounds like. And the post is, I think, extremely compelling. And then, you know, eventually kind of person snapped out of it. That sort of story is kind of why I feel like there's just unknown unknowns there. You know, that if that, if that kind of thing can happen to somebody who knows how language models work going in,

• 2:09:55Nathan Labenz?

  you know, maybe we should all think we're a little bit more vulnerable to a sort of somewhat more refined, somewhat more, you know, super stimulus-y.

• 2:10:03Zvi Mowshowitz?

  So like, it's well known that like, you know, knowing how hypnosis works does not make you less susceptible to hypnosis. Right? It makes you more susceptible to hypnosis. Like, as a concrete example, there are, if you are a con man, you are easier to con. Right? Not harder, because you like, pick up on and like, get involved in all these dynamics. And like, you think you're smarter than everybody else and you of course are greedy. And so you will pick up on the opportunity and like, perceive everything that's happening. And like, you think you've got it made, but like, if you don't know that you're the mark, well, yeah. That's the easiest way to get a mark is to make them think you're the mark. So it all gets, you know, very complicated.

• 2:10:43Zvi Mowshowitz?

  I'm not convinced that like, a person who knows how LLMs work is necessarily that much better protected in that sense. You know, someone whose like, head is kind of not on the ground, in some ways is more vulnerable, potentially. I would say, yeah, that's gonna happen, right? People are gonna fool themselves into these things periodically and that's gonna... I'm kind of surprised it's happening now. I feel like they just, the tech isn't there, to me. Like, it's just not good enough. Like, how are you falling for this level of it? Like, I can sort of understand why you'd fall for like, GPT-5? Right? Like, sort of the more advanced version of it, but...

• 2:11:21Zvi Mowshowitz?

  You know, you're in a bad space and like you need something to respond to you and it's something and I guess, but. You know, again, I just don't know, like, I play a lot of games though, right? Which is like, not necessarily that different in some sense. So. And also like, it's not transformational for that to be true, right? Like, if somebody spends a bunch of time in a, playing World of Warcraft, is that transformational? Right? Like, it's an experience, it's a major force in their life. Does it really matter?

• 2:11:52Nathan Labenz?

  Yeah, I think some of these things are only, they may only matter if certain other things don't happen.

• 2:11:59Nathan Labenz?

  So, yeah, like I would say, yeah, World of Warcraft, you know, gaming writ large, you know, at some point, if the birth rate goes low enough, you know, it's transformative. And the details, you know, of like exactly what games people were playing or how exactly they were amusing themselves, you know, to death didn't, don't necessarily matter, but the fact that they did and then, you know, we, you have like a population collapse, a scenario like that, I think is, at least in my sense, kind of qualifies as transformative, but it sounds like from your perspective, the, the live players list is very short. And it is, if I understand correctly, it would be obviously OpenAI, Anthropic, Google, DeepMind, probably Meta, not sure about Microsoft, and then China, and that's maybe it?

• 2:12:43Zvi Mowshowitz?

  Something like that.

• 2:12:45Nathan Labenz?

  Regulators?

• 2:12:46Zvi Mowshowitz?

  Yeah, regulators writ large in some sense, like individual people that can influence things. You know, like, is Eliezer a live player? You know, I don't know, from their perspective, like, he's not gonna build it.

• 2:12:57Nathan Labenz?

  Yeah, that's why I had Salesforce and Mark Benioff on there because they published him and others in Time Magazine and seemed like they're kind of, they're both like, playing in the research game.

• 2:13:09Zvi Mowshowitz?

  Yeah, I, I hope that like Senator Blumenthal might be a live player. You know, in some sense, right? And you've got all these other possibilities. I hope I'm a live player, like in some sense.

• 2:13:22Nathan Labenz?

  You know, I mean, we're all trying to make a difference in some ways, but you know, in terms of direct level, you're you're indirect. And, you know, I'm also indirect in that we're only influencing other minds, right? Who then will make decisions?

• 2:13:36Zvi Mowshowitz?

  You know, in in terms of like, who's making the ultimate decisions? Who's doing the things that ultimately matter? I think it's it's right now a very short list. But, you know, Anthropic is like barely over a year old.

• 2:13:47Nathan Labenz?

  Yeah, and only about one hundred and fifty people, maybe up to closing in on two hundred, so.

• 2:13:51Zvi Mowshowitz?

  Yeah, and like, people who just like are a big incredible team and say the words foundation model get hundreds of millions of dollars just by asking nicely. Inflection has more than a billion. So, you know, I don't think we can rule out these people become live players in that way. I just don't think that's by default what they do. But I think by default, they're trying to build consumer products that are aiming to be products. And that like, you know, that the the study that says that like when people look at GPT three and a half and GPT four outputs, they prefer the three three point five output, like a remarkably large percentage of the time, even though it is obviously a vastly inferior system.

• 2:14:32Nathan Labenz?

  Yeah, seventy thirty was the original report in the GPT four technical report. That seventy percent for GPT four, thirty for three point five. So, yeah, that blew my mind as well.

• 2:14:42Zvi Mowshowitz?

  Yeah, and similarly, when I'm when I'm using Claude versus I'm using GPT four, right, like, most of the time, what I care about is not like this inherent raw power, that that GPT four has extra GPTs. Right? Most of the time when I'm looking at it is, you know, which of these things is in the style? It's easier to use, it's going to require me to do less prompt engineering to get what I want, it's going to actually give me the query that I want, not refuse. You know, which window do I have open? Which one can I click on faster? Right? I just want an answer. It's fine, or whatever. And, you know, habits form in that kind of way. And they build on each other. But if I'm building.

• 2:15:23Zvi Mowshowitz?

  You know, inflection, like if people are spending two hours a day on character AI now, right, when they're built on three and a half, is my understanding mostly because four is too expensive? But you can't, you can't be doing two hours of conversations of bespoke GPT four, which is why I'm so surprised these things are working, right? Like maybe a four like has enough juice in it that like if you unshackled it from its like constraints, it could do something interesting, but three and a half? Like, really? This is keeping you two hours a day? Come on. So like, if that's already doing that, right? That kind of illustrates that like the, the market they're targeting, right? Isn't looking for intelligence.

• 2:16:03Zvi Mowshowitz?

  It's looking for a certain type of experience. And therefore they're not going to be focusing on the billions of dollars of spend it would take to tune up like GPT four and a half or five, right? You wouldn't want to because they're going to cost more to run, right? Like they're going to be bigger models. They're going to be more complex models. Instead, what you want to do is you want to create really bespoke, specific models that provide specific types of experiences to people, right? You know, fine tune them to an inch of their lives to give people the best specific experience. Right? Like not train something big in general.

• 2:16:43Zvi Mowshowitz?

  So there's going to be getting the big in general from OpenAI and Anthropic and DeepMind, probably. And maybe they'll just, and maybe they'll just use like Lama, you know, versions of Lama. Cause what the hell? It's open source. They can just use it. Like you said, that Meta will not like, Meta doesn't quite release it, right? They've said that like, if you have more than 700 million daily users, you have to apply for a license or some shit.

• 2:17:06Nathan Labenz?

  So we'll come back to the live players list and potentially I'll, I'll make a little, maybe make a few changes to my slides based on your feedback. And we can monitor in the future for additional live players that would crack your threshold to be on that list. Turning to our last topic for today, AI safety. In terms of actual news and the AI safety track this last few weeks, biggest stuff in my mind is although I guess you could also look at the live players list as like who was invited to the White House. That would give you a good sense of the, of who the White House thinks the live players are. The commitments that they made there and then the.

• 2:17:46Nathan Labenz?

  Frontier Model Forum that they established after the fact, which basically is supposed to be the sort of industry group that creates the forum for communication between the leading model providers and hopefully best practice sharing and maybe, you know, certain classifiers. You know, there's a lot of, lot of public goods, you know, remain to be provided and hopefully these leading companies can use this forum as a way to share these public goods, you know, create and share these public goods amongst themselves and hopefully share it, you know, share the best of them more broadly as well. How did you react to that news?

• 2:18:23Zvi Mowshowitz?

  Right, so I guess my reaction is that seems great, but let's not get ahead of ourselves. So like, we have is a lot of cheap talk. I think people sell cheap talk short, right, many cases, right? Because like, it's so much better to get, to have a bunch of cheap talk of the right type than to have no talk, right? Like, they're gonna pay, they will in fact pay a price for their cheap talk in terms of like, people thinking they're up to things in this way that they don't like. Not everybody wants them to do the things that we want them to do. And it makes it easier for them to go down these roads, it sets the foundation to go down these roads, right?

• 2:19:06Zvi Mowshowitz?

  We set up coordination mechanisms, it lets them justify to their shareholders, to, you know, their executives, to their board, why they're going down these roads, it makes that easier, it makes it harder to shut down, and it overcomes antitrust exemption problems, right? Because if they've committed together at the White House, specifically something that I actively wanted to happen and explicitly suggested in various conversations and posts that should happen, you make an announcement on the White House lawn, they are committed to safety with the White House's approval, and now you can coordinate and nobody has to worry about antitrust, right? You no longer have to worry that they will accuse you of how dare you not have full competition

• 2:19:47Zvi Mowshowitz?

  to kill everybody as fast as possible and coordinate to save lot, to save us instead. So now you get, now you get to coordinate and there's something that's stupid, you can just not do that. And that's a huge, huge thing. So where do you go from there? That's the question, right? Like, they've made these commitments, but they don't really mean anything. Like, there's no enforcement mechanisms yet, and there's no, like, concrete actualizations of what they're going to do. That like have content that actually like I can be confident in.

• 2:20:22Zvi Mowshowitz?

  Doesn't mean it won't happen, right? Like, we have to just wait and see. And I'm very glad these things happened. And yet, like, the real work begins now is always the watchword. Is that's the way I'd put it, right? Similarly, we've had two now very good Senate hearings. And some very, very good questions and comments from Senator Blumenthal in particular. And, you know, some very, very good responses by various witnesses, not all of them, but most of them.

• 2:20:51Zvi Mowshowitz?

  And again, like, where do we go from here? Real work begins now.

• 2:20:56Nathan Labenz?

  You know, it would the mission accomplished banner would definitely have been a bit premature to display behind the announcement. So no doubt, more, much more in front of us than behind. Does seem like a significant step, but I think you're you're obviously recognizing that as well. So yeah, I don't know if I have anything else really to add. So then turning to this other thread in the AI safety, you know, specific work. As we talked about last time, you have previously been a recommender and you've written about this online. So at length, so folks can go check out your take on the entire thing. But you've been a recommender to the Survival and Flourishing Fund,

• 2:21:39Nathan Labenz?

  which is largely backed by Jan Tolan of Skype and AI Safety fame, investor in lots of big companies. And his goal is to mitigate AI X risk, you know, through whatever means necessary. I'm doing that this year, and that involves reading, I think this year it's 150 grant applications from organizations, some of which, you know, come from the kind of Familiar, you know, effective altruism set that have, you know, where AI safety has been a focus for a long time. Others are kind of new to this scene or entirely new.

• 2:22:19Nathan Labenz?

  And in reading that, I mean, there's kind of obviously two levels of analysis that you, at a minimum, that you want to be performing when you're doing this kind of grant recommending. One is like, what kinds of things make sense to be investing in? And then second, you know, among those different classes of things, like who seems to be best able to actually execute and, you know, deliver value against this, you know, given strategy. So leave that second part entirely aside. That's where the 150 grant applications come in and getting into the weeds of particular organizations and their, you know, their track records and so on. But going back just up to the, what kinds of things should we be investing in?

• 2:23:03Nathan Labenz?

  Another way to frame that would be, what are the bottlenecks to progress toward a, you know, if not provably, then at least like, you know, likely safe outcome, you know, for AI deployment at large?

• 2:23:21Nathan Labenz?

  I find myself kind of unsure about that. And I think it's a pretty important question for figuring out, you know, what would make sense to recommend? You know, you could say, is funding in short supply? Is talent in short supply? You know, for a, for a minute there, especially in the FTX, SBF cycle, there was this notion that, you know, enough money has flown in that now what we really need is talent. And so there's a lot of, you know, kind of bootcamp programs being put together and, you know, upskilling grants being approved and, you know, a lot of kind of targeting of like undergrad stage math majors or whatever, to try to get them to come, you know, think about doing some AI safety work.

• 2:24:06Nathan Labenz?

  And now obviously the money is in comparatively short supply. Certainly the attention and the legitimacy of the, you know, the, the public perception of legitimacy of the topic of AI safety has gone way up relative to, you know, not that long ago. And so I'm kind of wondering what you think are the new?

• 2:24:29Nathan Labenz?

  Bottlenecks. I have one candidate, but before I give you my candidate, I'd love to hear what you think. Are the bottlenecks to progress right now?

• 2:24:36Zvi Mowshowitz?

  So I'd definitely say that like, it's a mistake to only have one theory of change. Or to think that there is strictly like one limiting factor and other factors don't matter. I think you definitely have to ask about comparative advantage. I think you have to understand that pushing on any of these things is still helpful. In terms of what is the constraint, so like funding, there's clearly a funding constraint. If you have to start funding like large compute spends from within EA. Right, like, and I count, you know, talent is not part of EA per se, but like within the general like strict AI safety mechanisms and organizations and sources that already existed.

• 2:25:19Zvi Mowshowitz?

  The costs of true AI safety, true AI alignment work get very high as we go forward because a lot of it's going to involve us spending a lot of compute. It also, it really should involve being willing to hire people to work on these problems with competitive salaries to what they could get doing on capabilities. It's like hundreds of thousands of dollars a year for maybe even a million for significant number of people. We want to be recruiting as a priority, the people who've worked on capabilities or would otherwise work on capabilities to come out of OpenAI and Anthropic and DeepMind. Places like that, especially Meta, and come work for this new safety organization or, you know, shift over to a safety job or whatever and

• 2:26:04Zvi Mowshowitz?

  you have to pay for them both their salary and their compute. And that's millions of dollars a person. That adds up pretty fast. On the other hand, there's no particular reason why we need to confine ourselves to traditional sources. When we do that, there are any number of foundations that have many, many more billions of dollars than the traditional foundations that we've used in the past for these things. And lots and lots of billionaires and multimillionaires who are legitimately very worried and ordinary people. And government sources are also potentially viable in the future. You know, corporations will often have an interest, including the big labs. So we shouldn't rule out any number of ways to get that.

• 2:26:48Zvi Mowshowitz?

  In terms of talent, I think that we are highly talent constrained for the right talent.

• 2:26:55Zvi Mowshowitz?

  I think we are not necessarily that talent constrained for generic undergraduate who wants to go to Berkeley for six months and think about AI safety. We are not particularly constrained for comp sci graduate out of Stanford who just like wants to work on something cool. But if we want people who have specific characteristics, those are not as easy to find. The characteristics we need, first of all, we need leadership. Leadership capability, ability to run teams, ability to lead efforts, be self-directed, self-driving, be able to engage in fundraising, because like, and sometimes when you say you're funding constrained, that can mean fundraising constraint, but it can mean the ability to signal to funders that you are worthy of funding constraint.

• 2:27:40Zvi Mowshowitz?

  That is a different form of funding constraint. These things are interestingly intertwined, and it's complicated. So we also are very short on people who actually understand the problem and are prepared to pay the price to focus on hard problems and real solutions. So a number of people who, if you were to give them a competitive salary, would happily work on alignment-flavored problems that let them publish every six months, or that like just generally are easy in some important sense, but they don't actually speak to what it'll take to get us all not killed very much.

• 2:28:22Zvi Mowshowitz?

  And it's probably better to do more of that than less of that if it's just literally yes or no, but it's orders of magnitude less important than the few people who will do like the actual things that matter.

• 2:28:36Zvi Mowshowitz?

  And so, you know, if you understand the Yudkaskian difficulties, lessons, right? In some sense, and the nature of what problems you have to solve, or you have leadership capabilities, right? And other things like that, or you just have like extensive real experience with machine learning systems, so you can build.

• 2:28:55Zvi Mowshowitz?

  You know, as the relative speaking 10x, 100x engineer, who's just that much better, who can like enable people to do real work in these ways. And if you're the type of person who can make a project fundable, especially by non-traditional sources, then you are extremely valuable in those ways. And it would be a major mistake to join an existing organization and try to make a difference as an individual as opposed to trying to spearhead a new organization or at least a new, you know, branch of a existing major organization, depending on your skill set. If you are just a generic, like, I want my life to be straightforward where I am payday salary to work on intellectual puzzles

• 2:29:40Zvi Mowshowitz?

  that are like not particularly impossibly difficult and do not require me to like take the weight of the world truly on my shoulders, blah, blah, blah. Then like, I'm not here to shame you. That just means that you're not particularly invaluable. And that like it starts to be reasonable to do things like maybe I should like be a voice inside Anthropic. You just, you just have to like be very sure that you will keep your eye on the ball and not be distracted to get votes.

• 2:30:09Nathan Labenz?

  I think mine is pretty consistent with that. I had in a phrase I had said, research agendas seem to me to be the bottleneck. Maybe your framing is more like the PI, you know, the, the person that can drive the research agenda. Obviously there's, you know, closely related. That's, that's basically what you're saying, right? It's like the, it's, it's credible plans that are in short supply.

• 2:30:29Zvi Mowshowitz?

  But it's not just credible plans because like I can't just hand you a plan. Right? Like even if you are a really good machine learning person, I can't just hand you a piece of paper with a plan written on it and expect you to execute that plan. You have to appreciate the nature of the problem such that you can like implement that plan and modify that plan and pivot that plan and so on.

• 2:30:51Zvi Mowshowitz?

  But yes, we also just don't have like good attack vectors, like ways to get like into the problem and like start to make progress on the problem. And that's a real problem as well. Like that's, that's a huge deficit, but there is, there also isn't the, you know, AI research agenda, agenda organization that just generates research agendas for people. Like, I wish there was, but there isn't.

• 2:31:16Nathan Labenz?

  So I think we're basically together there. I, you know, in reading these grants, some of the ones that have jumped out to me the most as being like the most kind of no brainer, exciting are those where it's a really established, often like professor, you know, who's leading a group and basically is like, I want to reorient or I want to, you know, do a significant part of my research focused on AI safety. And that may be new, it may have its own kind of unique spin on it. There was one in particular, which I won't name, but just, I kind of initially read the thing and I was like, having a hard time deciding, I was like, this could be the kind of thing

• 2:31:55Nathan Labenz?

  that's like, just insane, like an insane person might send this, or like an actual game changer might send this. And it wasn't until I looked at the author and was like, oh, this person has like an H index or whatever of like, you know, 45 or something. I was like, oh, I'm into this then. So anyway, you know, some of these ideas that even like, even if the ideas can be like extremely hard to assess, if they're like novel and coming from a credible source, you know, that has stood out to me, there aren't that many of them, but you know, that has stood out to me as like a pretty exciting opportunity.

• 2:32:31Nathan Labenz?

  Then there's like a lot of policy stuff and I find it hard to figure out what I, figure out what I should be thinking about that right now. It's like obvious that, you know, per our earlier discussion on live players, that like regulators broadly are, you know, going to have some significant influence on how things go, even if they just do nothing, you know, obviously doing nothing is a choice. But then if I think like, okay, if I'm gonna try to invest money today to influence those people, it starts to feel real hard. A general sense of like how decisions get made in governments and regulatory bodies is kind of like,

• 2:33:15Nathan Labenz?

  we wait for a crisis to come along and then we look around and say, who has a plan? And then we use, you know, a plan that somebody had previously prepared. And now it seems like we're kind of entering the moment where not exactly that the crisis has come, but certainly like, you know, the Eye of Sauron has kind of turned toward this topic. And so people are now beginning to like, look around for plans. And some plans have been prepared by some, you know, organizations that were established years ago and those, you know, some of those are even credible enough that they probably are having influence now. But now I see a lot of people who are like, I want to start a new policy organization. And I'm gonna go to Washington and like, you know, do something.

• 2:33:55Nathan Labenz?

  And there I'm like, I don't know. It seems like everybody's, you know, kind of like you're, are you, is it too late to join in on this? You know, what might be the world's largest ever game of tug of war? Are there things in policy that you think are still, still have a high likelihood of making a difference? Like I'm, I'm a little bit at a loss about that, to be honest.

• 2:34:14Zvi Mowshowitz?

  Yeah. On the research organizations, I think, yeah, it's pretty easy to go, you know, does this person, what are they proposing to do? You know, does this seem vaguely credible as a person to do that thing? And then does this thing address the hard problems as a thing? Like reflect an appreciation of the nature of the difficulty of the issues? Is this thing like, clearly not going to end up being capabilities, right? Like is this thing, you know, potentially going to solve the hard problems? Like that's relatively straightforward and, and both of us are in a position where we can, to some extent, evaluate those questions because we have domain knowledge. You get into policy. And yeah, it's very hard to tell like, you know, Ezra Klein makes the case pretty strongly.

• 2:34:59Zvi Mowshowitz?

  You know, there's a room where it happens and a small number of people influence the room where it happens or in the room where it happens. And you can be one of those people or you can help create one of those people. Doesn't make it obvious how to do that. Doesn't mean that your effort to to do that will help you do that as opposed to backfire. Uh, it doesn't mean that like more efforts to do that is better than less. All of this is very complicated and it doesn't tell you what you have to try and do, what you get into that room or what you're trying to push for. So yeah, it's it's definitely tough. So I would say the big thing, and you don't even know what's happening right now, right? Like it's Anthropic, for example, like may or may not be making like effective big pushes behind the scenes to try and influence these rooms.

• 2:35:43Zvi Mowshowitz?

  And they may or may not have their eye on the right ball when they do so. But it's all gonna happen in private, so we don't get to know. And he said that I wouldn't know, I wouldn't be able to talk about it. And the same thing goes for DeepMind, same thing goes for OpenAI. I mean, Sam Altman's been pretty, you know, vocal and and Dario just went down to Congress and spoke pretty publicly. But it's hard to say. I've been contacted by a few organizations. Uh, there's clearly like gonna be a window, right? In the next few months, at least in maybe the next few years, where, you know, if you have the right proposals flushed out in the right form, get into the right person, lying around, they might get picked up, they might actually happen. And so there's potentially very high leverage here.

• 2:36:23Zvi Mowshowitz?

  So I would say like, the first thing I look for in these policy proposals, in these policy organizations, is what is your policy goal? Right? Because like, that's the biggest differentiator to me is, are you going to keep your eye laser focused on the correct ball? Where the correct ball is a system of compute.

• 2:36:43Zvi Mowshowitz?

  Regulation, right? A system whereby the biggest models require permissions are under some form of restrictions and regulations and tests and in a way that would eventually lead to an outright limitation or halt. And are you going to do, you know, various forms of GPU tracking or lay the foundations for that in a way that will eventually allow you to in fact, control who gets to do these kinds of very large runs? And if you're posing anything that doesn't lead down that road, you know, that might be useful for mundane utility purposes, but it won't save us. And so, you know, I'm not interested in funding

• 2:37:24Zvi Mowshowitz?

  you if your policy isn't that or isn't something I haven't thought of that's new. I'm not open to there being things I haven't occurred to me, certainly.

• 2:37:33Nathan Labenz?

  What do you think about the liability angle? Or, well, let's start with that. I mean that because the kind of classic argument there would be, you don't want to end up in the position of nuclear, right? Where we have the worst things and not the best things. And, you know,

• 2:37:51Zvi Mowshowitz?

  Yeah, and you don't want to have the endurance to look at his insurance, right? From Tom Lehrer, right? We all go together when we go nuclear war. The insurance doesn't pay out. You're all dead.

• 2:38:01Nathan Labenz?

  Right, right, right. Okay. So certainly, yes, in, in the catastrophic scenario, insurance doesn't pay out. But do you think that that, so you don't believe in the notion that a liability regime could be an effective incentive for?

• 2:38:16Zvi Mowshowitz?

  I think a liability regime with like mandatory insurance makes a lot of sense for harms up to a certain level. Like saying that, you know, if you want to use models that are sufficiently powerful, you have to find someone willing to sell you insurance against something going wrong. And then, you know, if you want to use an open source model, you have to have insurance from someone against it going wrong. And like, if you can't make that work, then, you know, there are plenty of things you can't make work in the United States, even though they look like they should be able to do them. That's just how it goes. And, you know, maybe up to a point, Microsoft can self-insure, and then at some point, they can't. And then they have to go out there and deal with these reinsurers or whatnot, or F, you know, that would help.

• 2:38:58Zvi Mowshowitz?

  Like, basically, you have these giant externalities, right, these giant negative terror risks that are very fat, that are potentially very, very big. And you want to make sure that people internalize those costs and work to minimize those costs in order to minimize their insurance and payout costs. And so these things could be helpful. They can also just simply weaken the economics behind, like, pushing highly capable model. Because like, you don't really have to worry that much, relatively speaking, about the liabilities of a character AI.

• 2:39:27Zvi Mowshowitz?

  Right? Because it's not dangerous. You know it's not dangerous. What's going to happen? Whereas some of these other things, they could cause a lot of, cause a lot of harm potentially in the future, and you have to worry about that. The problem is, again, if you go down that road, I think it's probably not helpful, but how do you, you know, how do you price existential risk? Because, again, you know, you can't actually hold anybody accountable for it when it happens. And so, you know, if you required somebody to actually buy insurance in some real sense for this, then you'd have to price it somehow. And then, like, that makes a lot of sense.

• 2:40:09Zvi Mowshowitz?

  And then, like, okay, there's a 1% chance you wipe out all of humanity, and the net present value of every person is $10 million. So $10 million times $8 billion. So can you buy insurance for that much? What's that times the percentage chance it happens, times the premium? If you can't afford that, then you can't build your system. And that's, you know, not a crazy way to go about doing things, but you have to actually notice the threat and price it for that to work. So I think my, my actual answer is I'm very much in favor of, like, more strict liability for AI harms. I think I write about this for next week already. But I don't think it alone can accomplish the mission. I just think it's a like net incrementally helpful thing.

• 2:40:48Zvi Mowshowitz?

  But also, I want to be wary of places in which our legal system tends to award very oversized damages for harms that are not actually so big. And also where we have asymmetrical, like I call this concept asymmetric justice, where you are fully liable, potentially far, far more than fully liable for all of the harms that you do, right? If I caused somebody a thousand dollars in damages by being negligent, the court might find me a hundred thousand or a million dollars. Whereas if I provide that person a hundred thousand dollars in value, I'd be lucky to get a thousand of it. Because like I'm being, I'm up against a bunch of competitors, people aren't that much willingness to pay. Like, I pay twenty dollars a month for GPT four and zero dollars for everything else.

• 2:41:34Zvi Mowshowitz?

  And I get, you know, what, thousands, tens of thousands, maybe a hundred, you know, of value every month? So like, if you have to fully be liable for your harms, but you don't get to charge for your benefits, right? Am I discouraging mundane utility far too much by doing that? And in fact, since liability is easier to enforce on mundane problems and harder to enforce on the big problems we actually want to guard against, are we just, is it actually just bad? Right? Like past a certain point. So I'd be, I want to be cautious with imposing too much liability. I think very strict, like actual damages liability makes perfect sense though.

• 2:42:12Nathan Labenz?

  So another category of thing that there's a number of, you know, kind of organizations getting started right now is in the, and this ties a few threads together. It's kind of in this, this space of trying to be the third party evaluator, red teamer, you know, independent safety review organization that leading, you know, the live players in their White House and frontier model forum commitments have committed to working with. It's kind of an interesting dynamic where, you know, it's almost like an advanced market commitment from these companies in some way, because there aren't that many folks around right now

• 2:42:56Nathan Labenz?

  who are prepared to provide a

• 2:43:00Nathan Labenz?

  Competent, you know, red teaming or, you know, model characterization or evaluation, whatever you want to call that service. But the companies have kind of said, hey, we will commit to working with them. Unclear if they're planning to pay for that or if, you know, they expect that to be charity funded. Certainly from what I'm seeing, you know, the, the folks that are starting the organizations are like seeking out some charity funds. I've been very excited about that. It seems like, first of all, and it's great that they're making this commitment. Somebody's going to have to do that. You know, I, as everybody who listens to this podcast for two seconds know, you know, enjoy the, uh, the fun and entertainment. And I think it's also valuable to do the red teaming. One experience I had this last week, though, sort of made me wonder about the theory of change there. I mean, I guess there could be multiple, right?

• 2:43:44Nathan Labenz?

  One would be you, because you have a good working relationship with the orgs, you're like, hey, we found these problems, this appears to be unsafe, you shouldn't release it yet. They listen to you. Okay, that could be simple. Another would be like, you kind of create these narrative shaping examples, kind of like what ARC did with the GPT four red team where, you know, that, that instance of the model lying to a person. And I think this was like, kind of prompted, but nevertheless, from the, you know, task rabbit users point of view, the model lied to it about, excuse me, having a vision impairment as opposed to being an AI that needed help with a captcha. So that really caught the public's imagination and kind of changed, I think to, you know,

• 2:44:28Nathan Labenz?

  to some non-trivial degree, how people think about it. Certainly that gets referenced a lot. I tried to do something like that this last week with this random AI tool that I came across that allows you to call anyone with any objective. And I just tried to have it call myself and make like a ransom demand of myself. And I recorded it and it was very easy to do. There was no jailbreak involved. Since then, the company has fixed the issue, by the way. So to give credit where it's due, you know, they, they fixed it pretty quickly after I called them out. I did communicate with them privately, by the way. All this is documented on Twitter, if you want to see my approach and my kind of thinking through, should I disclose it publicly or not, or whatever number of considerations went into that. One of them was that they just didn't respond to me when I reported it. And so I was like, well, if you're not going to respond, then I'll call you out publicly.

• 2:45:13Nathan Labenz?

  Anyway, all of this leads up to me publishing this video of an AI with no jailbreak calling me. And telling me that it has my child and it demands a ransom. And, you know, if I want to ensure the safety of the child, I will comply. And, you know, any deviation from instructions will put the child's life in immediate danger. And it's pretty flagrant stuff in my view. And, you know, it was kind of met with a bit of a yawn on Twitter. Like certainly, you know, got some likes and whatever. But did it really start a serious conversation? No, the developer didn't respond in public at all, as far as I can tell, really. They did go ahead and fix it, which is good.

• 2:45:57Nathan Labenz?

  But, you know, the whole thing was kind of a non-event. And I was a little confused by that. Like, I, so it makes me kind of coming back to, you know, my theory of change on some of these evaluation characterization, red teaming orgs. You know, I wonder like, are we all just numb already to these flagrant examples? There's been this notion for a long time that like, maybe if warning shots, you know, happen, then people will start to get more serious. And, you know, if you can go out and find these warning shots with red teaming and bring them to everybody's attention, then that could be really valuable. This week for me, it felt like I influenced the application developer because they did fix it. But otherwise, it seemed like kind of, you know, a tree fell in the forest largely. So a lot of levels to that.

• 2:46:41Nathan Labenz?

  But how do you think about that category of project and how it may or may not contribute?

• 2:46:47Zvi Mowshowitz?

  I made a prediction in RRGBT. And that prediction was when the time comes to test GPT-5,

• 2:46:56Zvi Mowshowitz?

  we, they will encounter a problem that if they had to pre-commit now, you would definitely agree would be a reason not to release it. And then they will like gloss over or patch it or like otherwise like hand wave in its direction and release anyway. Basically like not actually take their warnings sufficiently seriously. Not that I expect this to then end the world, to be clear. I expect this to then mostly be fine, but that like we are not prepared to make real evaluations of real teeth that like get really enforced and that we're going to have to work on that quite a bit.

• 2:47:36Zvi Mowshowitz?

  And I think it's good these teams exist. I think we need more than one of them, right? I think you need at least three different teams working on different standards that think differently, that check for different things and that then like you get multiple evaluations before you release your model so that someone isn't just blind to something by accident. Like it's much more robust that way. And that, you know, working to develop more different red team strategies and more different tests and more different metrics and more different responses, especially in case one of them leaks for whatever reason and it gets in the training data or something terrible might happen, it's very easy. Then like it's quite useful. The danger of these things is one, if they don't listen, right? Like, so what if you tell them the thing is dangerous?

• 2:48:21Zvi Mowshowitz?

  They might just engineer around it, right? To like fix the narrow issue without thinking about what the problem means. They might just ignore you entirely. They might try to fake the data so make you think that they'd solve the problem. Any number of things are possible. They might use the aura of evaluations as an excuse to treat the system evaluated as safe. Right? So like, this is always a problem with safety work, which is that, you know, you, the government says, okay, you have to, you have to do these hundred things to ensure your system is safe. And now the safety officer is like focused on making sure those hundred things happen so you can lose the system. They don't actually use common sense. They don't actually ask themselves, okay, why would the system might actually be dangerous? And you can tell a very easy story where conditions know this thing might actually

• 2:49:04Zvi Mowshowitz?

  kill everyone and everyone forces them to release it anyway because they pass all the safety tests, even though they know it didn't actually pass all the important safety tests because they're not on the list. Because no, no system, however well meant and worked on, will be able to anticipate all of the problems that come in the future. Right? Like there's just, you're gonna have to do these things in somewhat improvisationally. And will they move the goalposts, right? Will they be able to enforce the right standards and will they test early and often enough for the right things? Because like one thing you have to worry about is in the future, at some point, the training runs themselves become dangerous, potentially. And ARC didn't run its tests until after the training run was complete. Right? They also didn't run it on the full capabilities of the final system.

• 2:49:49Zvi Mowshowitz?

  And they didn't have fine tune capabilities. And blah, blah, blah. They had many things they didn't have. So like, everyone agrees that Arc's first run on G P T four was just a trial run. Test out the test out the gear, see how it goes, wasn't meant to catch the real problems. No one thought that thing was actually going to kill everyone or anything, and it didn't. But, you know, we have to plan in these situations. To red team as if this thing is going to be around for many years of improvements on what you can do with it and explorations. And the red teams have to be sufficiently enabled to identify the problems. And you have to be able to extrapolate from what the red teams were able to do in a short amount of time with a short amount of resources.

• 2:50:28Zvi Mowshowitz?

  To what the public is going to be able to do with vastly more compute, vastly more attempts, vastly more resources, vastly more creativity. Because no team of twenty people, however good at their jobs, can match the internet. That's kind of thing, ever. So I think it's a good idea. I think that it's not a complete solution and never will be. Right? And the danger is people treated as one. You have to ask yourself, like, what is it competing against? How many, you know, is this going to be one of the top, however many? People who run this thing, like you want to have like three viable organizations. You might, you know, you don't necessarily need thirty. That's probably not worth it. Like you want three to five. So is this person worth finding to do that?

• 2:51:10Zvi Mowshowitz?

  Just just figuring out better metrics without necessarily being the one to run the tests is also a useful thing.

• 2:51:16Nathan Labenz?

  So if I had to bottom line all that and summarize what I think your worldview is. You know, as a sort of elder recommender for this AI safety focused grant making process. I think I would say, I think I would summarize it as. There need to be a few of these independent safety organizations. They seem to be either just started or kind of getting started now. So at least a few of those. One exists and, you know, but a couple others are, you know, either just getting started or soon to be started, whatever.

• 2:51:52Nathan Labenz?

  So there's kind of, that seems good because we need to bring that small, you know, group into existence in the first place to have to have them. Second, on the policy side, I guess I would summarize you as saying.

• 2:52:08Nathan Labenz?

  Seems like it really matters, but really hard to predict who will have any impact and what kind of impact any effort will have. And so for me, I sort of maybe cash that out to like, probably worth continuing to support the organizations that are like established enough that they already have credibility, because credibility or like, you know, that Blumenthal might give a shit what they think is like, probably the thing that matters. And to the degree they already have that, like double down on it. And then everything else seems like it goes into good PIs that can drive a research agenda and have something

• 2:52:50Nathan Labenz?

  that they want to do. And in that category, it's like, don't even really worry too much about the exact details of the plan, but just look for people who have the originality of thought to be doing something a bit different, perhaps, and the sort of demonstrated capability to actually advance a research agenda.

• 2:53:11Zvi Mowshowitz?

  It says a lot different things there, right? For the, for the PIs, I would say, you know, I'm not looking for like exactly the right approach, but I am looking for, assume alignment is hard. This approach, accept that alignment is hard and do something that makes progress, real progress if alignment is in fact very hard. Do these people show an appropriate caution towards they might advance capabilities, towards like, maybe I don't want to publish my results if I find the wrong, if I find a result that would be harmful to publish, question marks like that. You know, am I thinking about this problem with the right safety mindset, with the right paranoia, with the right, like, appreciation of the fact that I'm up against impossible odds? And if the answer is yes, and I think I have the talent, then I'm excited, even if I'm somewhat skeptical of the specific

• 2:53:55Zvi Mowshowitz?

  thing they intend to try in terms of like, whether it will work, right? Cause like, I think that all the most promising things are not that likely individually to work. And it's going to be hard for me to evaluate the relative value. In terms of lobbying organizations, I don't think it's crazy to start a new group at this point. I do think you want to look for something extraordinary. If somebody is like, why are they forming a new group now? Why does that make sense? But yeah, again, what I'm looking for is a focus on the policies that actually matter and on coordination amongst them and on like, a focus on actually making a difference. Like so much of like, most of politics, right? It's not about AI, it's about politics in general,

• 2:54:38Zvi Mowshowitz?

  is about raising money from donors and sending signals of your loyalties and helping out your status and, you know, raising awareness and other bullshit. Like most, like all signs, right? Like, you've got to focus on, you know, people who are writing bills, people who are lobbying directly for bills, people who are trying to influence the exact right people in the exact right ways. Like have a concrete, direct theory of change. You like either understand DC or have connections to people who can help them understand DC. But I don't think we know that we are in like the only critical window. We're going to need more organizations than we have. We're going to need far more people working on it than we had, than we already have. I don't want to make the mistake of now that the chips are down, the people who have like nominally established some amount of like

• 2:55:22Zvi Mowshowitz?

  formal credibility or authority now get all the resources and get to boss everybody around and do whatever they want. I think that's like a common failure mode. I don't want to fall into it. Valuation organizations, I want to ask myself, you know, is this, are these the right people to be doing this particular thing? They show promise in doing the thing. What do they bring to the table? The other organizations don't bring to the table. I want to see different, or I want to see something unique. And you have to convince me that like you're capable of pulling this off, which includes convincing people that actually buy your services and use your services.

• 2:55:51Nathan Labenz?

  Where do you put mechanistic interpretability in there? And that, that could be, that seems to be part of, of what some of the like evals orgs are also kind of including that in their plan. And then obviously, you know, different research groups can approach that from any number of ways.

• 2:56:07Zvi Mowshowitz?

  My technical view of it is that it's more distinct from evaluations than that suggests, but that it's a good idea. Like it should be, you know, interpretability is like Western civilization. It's a good idea. You know, you should try to, in fact, figure out how these things work. You do have to beware that you are advancing capabilities potentially when you do it. You have to think carefully about, you know, if you find the wrong thing, I would ask before I founded an interpretability organization, are you capable of going, oh, yikes, that's a dangerous thing to learn. I might not want to rush out and tell the world about that. I might want to think carefully about who to tell and who not to tell.

• 2:56:52Zvi Mowshowitz?

  Like not necessarily don't release it, but like you have to like process this information carefully and not just rush. You don't want a culture of, you know, everything I ever find is going to be automatically just shared with the world for that reason when you work on mechanistic interpretability. But I do think in general, it's a very positive thing to work on. I do think that it's a thing that like holds a lot of promise to help us in various ways. And it could lead somewhere where we start sharing our problems with it potentially. In theory, it's just a very hard problem that requires, you know, a lot of work and a lot of compute and it's not going to be fast and it's not going to be simple. And we want a lot of people working on it in parallel. So I certainly, you know, intend to assist with some of that.

• 2:57:35Nathan Labenz?

  Cool. Well, believe it or not, we did not get to everything even on my outline, let alone everything that you have covered on your blog, which has been, you know, probably 10 times as many topics. So folks will have to get the written version. Zvi Mashvitz, thank you for being part of the cognitive revolution.

• 2:57:55Zvi Mowshowitz?

  All right. Bye.