OpenAI Sora, Google Gemini, and Meta with Zvi Mowshowitz

In this episode, Zvi Mowshowitz returns to the show to discuss OpenAI’s Sora model, Google’s Gemini announcement, Anthropic’s Sleeper Agents, and other live player analysis. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api Definitely also take a moment to subscribe to Zvi's blog Don't Worry About the Vase (https://thezvi.wordpress.com/) - Zvi is an information hyperprocessor who synthesizes vast amounts of new and ever-evolving information into extremely clear summaries that help educated people keep up with the latest news. LINKS: -Zvi’s Blog: https://thezvi.substack.com/ - Waymark The Frost Episode: https://www.youtube.com/watch?v=c1pPiGD7cBw -Anthropic Sleeper Agents: https://www.anthropic.com/news/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training SPONSORS: The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://brave.com/api Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off www.omneky.com NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. X/Social: @TheZvi (Zvi) @labenz (Nathan) @CogRev_Podcast TIMESTAMPS: (00:04:37) - Zvi’s feedback on the type of content the show should create (00:05:33) - Zvi’s experience with Gemini (00:09:42) - Speculating on Google Gemini’s launch timing (00:12:54) - Advantages of Gemini (00:16:11) - Sponsors: Brave (00:25:00) - Speculating on Gemini 1.5 and market dynamics for foundational models. (00:28:18) How long context windows change things (00:30:57) - Sponsor: Netsuite | Omneky (00:41:06) - LLM Leaderboards (00:43:37) - Physics world modelling in OpenAI Sora (00:57:25) - Object permanence in Sora (01:04:40) - Experiments Zvi would run on Sora (01:06:21) - Superalignment and Anthropic Sleeper Agent (01:10:47) - When do agents actually start to work? (01:16:00) - Raising the standard for AI app developers (01:22:07) - Dangers of open soure development (01:30:53) - The future of compact models (01:33:58) - Superalignment (01:53:00) - Meta (01:54:20) - Mistral’s hold over the regulatory environment (02:04:16) - The Impact of Chip Bans on AI Development This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Zvi Mowshowitz?

  Someone walks into Sam Altman's office, says, "Google just announced 1.5 Pro. It has a million-way contact window, potentially maybe 10 million. It's really exciting. People are raving about it. Is now the time?" And Sam's like, "Yeah, now's the time." But you can't have both ways. You can't both say, "There are so many chips coming online that we need to build AGI soon, and AGI is coming along so fast, we won't have enough chips. We need to build lots more chips," and then go raise the money to make the chips. It would be surprising to me if AGPT-5, worthy of its name, didn't make these agents kind of work.

• 0:36Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Zvi Masiewicz, welcome back again to The Cognitive Revolution.

• 1:02Zvi Mowshowitz?

  Always funnier. Let's do it.

• 1:04Nathan Labenz?

  Let's do it. So I call you when I feel like my own personal contact window is overflowing and I can no longer find all the needles in the haystack that I might need to, to make sense of what's going on. It sure seems like things are happening faster and faster all the time.

• 1:21Zvi Mowshowitz?

  Well, that's what Gemini Pro 1.5 is for, right? You've got the big new contact window. You should be okay.

• 1:26Nathan Labenz?

  Well, how is your personal contact window holding up, first of all? And then we'll get into all the latest news.

• 1:31Zvi Mowshowitz?

  I think that my key abilities are advancing fast enough that I've been able to keep pace. But it has been rough. So the discourse speed has slowed down in the last few months. It feels like we had a lot of really big developments and discussions around various potential government actions and also around the OpenAI situation. And 2024 has been more of a, like, okay, everybody chill, everybody process what's happened, and therefore, we can focus on temporarily technologies. And yeah, a lot is happening, but also, you start to see the patterns, and you start to say, "Okay, this is a version of this other thing that's happened."

• 2:12Zvi Mowshowitz?

  And it's not that it's slowing down in absolute terms, but you gain the power to match the ability to process what's happening.

• 2:20Nathan Labenz?

  Is that a natural process for you, or are you changing? How you work at all or your approach in any way?

• 2:27Zvi Mowshowitz?

  So I think it's a natural process in the sense of I find myself saying, you know, let's try this again. Let's go over this again. You know, or as we previously discussed, right? So like you see some of that Levine columns where he's like, well, you know, as we have talked about before, and every word is a link to a different, you suppose. It's the same basic idea of a lot of these concepts and questions and reactions are going to repeat. Right? Like different people are realizing different versions of the same thing at different times. You know, the future is not even distributed, it's getting distributed. We pick up on when people notice this, especially when mainstream media, you know, standard cultural people pick up on things that we knew six months ago, we knew a year ago. Where we figured out was coming at that time.

• 3:13Zvi Mowshowitz?

  And so a lot of it is just, okay, yeah, we're going to go for this again. We have already covered that. Yeah, this is actually from October. And that happened to me yesterday. Actually, I was writing stuff up and I realized, no, like this is actually a few months old. I probably already covered this. It's fine.

• 3:28Nathan Labenz?

  I'm trying to figure out what I can do to be more valuable. I think, you know, I've never really considered myself a content creator in, you know, in any sort of normal content creation sense. Starting a podcast was kind of a way to make sure that I was consistently tackling a new topic. And, you know, hopefully sharing that with people in, in what I hope to be a somewhat useful way. It's gone better than I expected it would in terms of people being interested in listening to it. And after a year, I'm kind of like, how many of these should I be doing in kind of traditional interview format? How many should be more like this, where it's like a little more of a survey type of thing? How many should be more of like a deep dive where, you know, perhaps not even with a guest?

• 4:11Nathan Labenz?

  I'm just kind of really going hard at a particular topic. I did that late last year with the Mamba architecture and really felt like that for me was a more profound learning experience. And hopefully a better product for the audience than a typical episode. So I don't know if you have any immediate reactions to that, but if not, we can maybe circle back and at the end you can tell me what you want.

• 4:32Zvi Mowshowitz?

  I do, actually. I would say. You should basically always strive, in my opinion, to do more deep dives. Like not just you, but like almost everybody than you naturally want to do. Like, I mean, I'm working on a newsletter, obviously. Every week I am writing, you know, an average of 10, 12,000 words or something. You know, the good in quotes about what has happened specifically in this past week. But I find the most valuable things, especially if you're building up a long-term base of operations are when you need to focus more on specific places where you feel you can rig it, your insight, where you can be the go-to source.

• 5:12Nathan Labenz?

  Yeah, that was my experience with the Mamba piece. So I'm looking to figure out how to do a little bit more of that. So what are your big pieces? And certainly a deep dive has been recently into your experience with Gemini. This is obviously something that we did know to expect. And if anything, you know, seemed like it probably came a little later than we might have penciled it in for. You had the good fortune of having early access. I would love to hear how that happened and see if there's anything that I can potentially copy from you going forward on that. But then obviously more important is what your experience with it has been. And then of course, now we've also got 1.5 to look forward to as well.

• 5:54Nathan Labenz?

  But let's start with 1.0, which you said, you know, among other headlines, had become your default language model after a couple of weeks of playing with it.

• 6:02Zvi Mowshowitz?

  Someone that were reached out to me and said, we'd love to get your feedback on this new version that we're going to ship soon. Would you like to check it out? And I think that, you know, I do is I establish that, you know, I'm someone who can be trusted. If you share it with me, I'm not going to go spy on the internet. And I can provide useful insights that are different, that are well worth the time spent directing. And then obviously, if you share it with me, I'll be able to offer more detailed insights, like right when it's released, which can impact how people see it. And they presumably thought they had a good product. They wanted me to give my ask me, there was no attempt to put a finger on the scale in any way, no attempt to convince me it was better than it was. It was all very, very honest and straightforward and realistic.

• 6:45Zvi Mowshowitz?

  I thank them for that. It was really a good experience. They seemed to care about my feedback. This was someone who clearly wanted the information, wanted to understand. So what I noticed right away was it was really good at many of the natural things that I wanted it to do. So what a lot of people will do when they get their hands on a model is try to get it to do something horrible. I'm guilty of that.

• 7:10Zvi Mowshowitz?

  Well, you know, Sally has two apples and then, you know, John brings two oranges and they all play a juggling act and then they make salad. How many apples are left? And the model's like, three. It's like, no. And then what's the point? Yeah. In some sense, there's, there's other people I think are whose jobs will be to red team this model. Either to get it to do harm or to find out where it's dumb or where it's going to fail, right? On like theoretical tasks. And I took the opposite approach. I asked myself, you know, what good can this do? And what happens when I feed it exactly the queries I just want the answers to that I would normally want to do over the course of my day? Well, I have some amount of curiosity and I could do some amount of brainstorming and trying

• 7:52Zvi Mowshowitz?

  some stuff out and trying some parallels and so on. But what I found was mostly the things I want are explaining things, you know, finding, you know, just general like answering my questions. I'm confused. I want to use search for, you know, maybe search for a document in some cases, although the version I had back then didn't do that. I looked at other things. But I'm just a very practical user of LLMs. But I'm not trying to help it do my job. I'm trying to like extract the information and understand it. That I can go about my day and do the rest of my stuff. And for that, it was very, very good. In particular, as I was on a trip, I was trying to read the financial DPO paper that it never got around to. And I was asking it a question every paragraph or two. And

• 8:35Zvi Mowshowitz?

  it was very, very good. Despite not having been fed the paper, but it seemed to have the information and it was very, very good at answering my questions on exactly the level of which I was asking. Figuring out exactly what information I need to know. And a lot of other similar things. Well, I noticed it had some flaws, like I would try to use Google Maps. And it would sometimes give you good information and sometimes it would repeat three different times for how long it would take to get the SFO from Berkeley within the space of three questions in the same chat, despite having the prior efforts. Which is better than just artificially agreeing with yourself. I was admiring the fact that it wasn't doing that. The same plan was driving frustrating. I can't count on you for anything. This is weird. I clearly didn't know what the weather was and then it would repeatedly not know what the weather was.

• 9:18Zvi Mowshowitz?

  I found some other bugs in the system that they're really just, I clearly this thing launched like earlier than it was supposed to. And it's also the 1.5 launching like two weeks after 1.0 advanced. But this is not a natural thing for a like well planned out release schedule to do. Right? If you almost have 1.5 ready, you don't make your big announcement speech. Right then, unless, you know, this is me, you're expecting like the CEO was on the phone with you and saying, no, you're damn well going to launch right now. And you say, but we're almost ready with 1.5. I say, I don't care. You're going to launch. I can't not launch now. And then this happened to me right when I was doing a game company. And we were just told, you're launching today. And we said, we're not ready. And they said, we're launching today. I said, it's Friday. Can you at least wait till Monday, you idiots?

• 10:03Zvi Mowshowitz?

  And they're like, no, you're launching today. And I, we would not have succeeded if we lost on the following Monday, but at least it would have been slightly less embarrassing to do. But, you know, just it's clear that, you know, there was a, we need to launch this at whatever stage it's at. And then we're seeing rapid improvements and to the point where we're not sure if 1.5 is better than 1.0 advanced. You know, discounting compute, it's got advantages and disadvantages, which I haven't had the chance to properly explore yet. But yeah, I was just really impressed by, this is faster, this is smoother. This just does what I want it to do. My workflow is easier and better. I want to use this. I don't want to go to GPT score. If I can help it, I do that when I feel like I have to. Whereas I used to be like, this is where I go.

• 10:47Zvi Mowshowitz?

  And then sometimes I use Claw, right? Sometimes I use perplexity.

• 10:51Nathan Labenz?

  So can you put a little finer point on what it is that makes it better? I mean, I know that it's obviously tough because the surface area is so vast and there's so many facets to language model performance. For what it's worth, I have also been impressed. I haven't spent as much time with it as you have. Today I am going to Gemini Advanced, AKA Ultra, 1.0 Ultra, and ChatGPT with GPT-4 and Claude. Basically for anything that I want help on. I did this once with like a little kind of proto contract negotiation that I'm working on where,

• 11:33Nathan Labenz?

  you know, I had kind of a couple of messages back and forth between me and the person I'm working with. And, you know, kind of said, here's what this person seems to care about. Here's what I seem to care about. Here's our messages to each other. You know, can you kind of help us get to a good win-win situation? They certainly all behave a bit differently, but I found them all kind of useful. And the synthesis of the three was, you know, for me, kind of a way of making, you know, telling myself I didn't need counsel because I had enough kind of AI counsel to make sure I wasn't leaving any, you know, major blind spots for myself. And then also did this with a, basically a job description where I have been working with a friend on a hiring process. We've had a number of calls with different experts, got their opinions. I've taken very raw notes and just fed in kind of a paragraph at the top of like, okay,

• 12:17Nathan Labenz?

  now it's time to turn all these notes that I took over various calls into a job description. You write it. And of course, you know, they each did the task. In that case, I did find, and I think this is fairly random. I wouldn't say it was like a systematic finding by any means. I mean, I'm not sure how, how, you know, frequent this would be. But I did find in that case that the Gemini response was the one that I wanted to go with in terms of just taking something, editing to a final product that I would, you know, hand over and feel proud of. Gemini did come the closest to what I was looking for. But those are, you know, spot checks. Can you, can you help us kind of get any, any more systematic or, you know, predictive in terms of what the advantages of Gemini would be?

• 12:59Zvi Mowshowitz?

  Yeah, the interesting is it's kind of the opposite of mixture of experts, right? I mean, it's a mixture of non-experts where previously you would figure out, okay, I have these 20 different expert systems and the program figures out which one to call is they have on compute. Now it's like, no, just call everyone, computer's chief, compare all the answers, see which one is the best, which makes sense for a human case. And yeah, I even have 1.5, right? I have access to 1.5, not in the standard interface, but in the developer studio beta. So I can then not only compare Claude, GPT and Gemini, but also the other Gemini. So now it's like I have the two Geminis to work with and the name finally makes sense. So I think what I, what I found was the, the biggest feature was just it tells you the

• 13:40Zvi Mowshowitz?

  information you actually want to know when you ask a question much more reliably with less like worthless stuff like scaffolding around it. And also I had gotten some good use out of the Google button. So when you answer on Gemini advanced, one of the things you can do is you can press the, the Google icon at the bottom and it will search for web sources for all the things that you just had. And they'll light up in green indicate that you have a source that you can now click to. And this is just very seamless. Actually, the guy I was doing the test with showed me this feature. I didn't understand it was there until he told me. And then you can then ask for a sort, you know, what's going on? Do you have any physics on this? Do you have any knowledge of this? And then you can find the backup. And that was very helpful.

• 14:23Zvi Mowshowitz?

  In general, as I said, like, it's also just very good understanding. Okay, if he asks this question, what does that mean? He doesn't understand. And what does that mean? He does understand. And I find GPT four can be pretty dense about this. And I probably could work with better custom instructions to try and improve this than what I have. But, you know, so far it's been kind of frustrating. Claude, I found is a step behind for most purposes. I use Claude mostly when I'm looking at long documents, but I think your, your use case is a very different one because you are asking for a creative job, right? Where like you get lots of different answers. There's no right answers. There's no wrong answers. There's just different answers. Being able to take a sampling and then choose the best one or combine the elements of different ones helps you a lot.

• 15:08Zvi Mowshowitz?

  Whereas what I mostly try to do is I'm trying to ask questions where there is a right answer. I'm trying to look for explanation or I'm trying to just get some brainstorming at most to try to go with. And so I tried Gemini first and Gemini gives me an answer that does the job. I'm done. And the only one I get frustrated with that like, okay, fine. I'll try all these other people. I won't jump from one to four. I won't just jump from one to two. I was really happy in practice with Gemini. It does what I want it to do, except when it doesn't, right? But I've adapted to like, okay, those are the things that I just don't do. I haven't had much calls to use GPT-4 specifically since then because it just doesn't come up very often. Like there aren't these, the cases where GPT-4 is good potentially, where Gemini is bad,

• 15:54Zvi Mowshowitz?

  or where you're trying to like customize it in these very specific ways, I think. And those cases just don't come up for me very often. So it just doesn't see much use. And I'm not going to cancel it because even the option value is just so much more than like what tickets they charge.

• 16:09Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:13Nathan Labenz?

  Just to give a quick plug to Claude on my legal question. I did find it to be the best in the sense that the commentary that it gave back to me was least sort of padded out in RLHF, if you will. It was more kind of direct, to the point, conversational. And actually just kind of concrete on the suggestions. It seemed like the others were a little more reluctant to. And I've actually been pretty impressed with Claude on this front a couple of different times, including one specifically pertaining to you, but finishing this story first. They're just kind of plain spoken, to the point, direct, like not super hedgy.

• 16:58Nathan Labenz?

  With both Gemini and GPT-4 by comparison, I felt like. It was really reluctant to make a recommendation, you know, to make a suggestion for like what the agreement might ought to be. It was very like, this is an important consideration. You guys will need to figure out what the, you know, what the right thing is here. And as I pushed them, they would do a little bit more of kind of, okay, here's a, you know, a specific proposal. But Claude was much more readily willing to just be like, here's kind of what I would suggest. And it was pretty apt and it, again, I did not, you know, copy and paste this into, you know, my running dialogue directly. But it was the closest to, in that case, I actually wrote my own thing and again, took my own thing back to each of the three models.

• 17:40Nathan Labenz?

  Said, okay, based on your, you know, suggestions and my own thought, here's what I came up with and asked for one more critique and then basically sent it from there. But if I had been inclined to pick one from the three, it would have been Claude because it was just the most kind of to the point, easy to understand. Not necessarily exactly what we're going to go with, but at least like kind of willing to put something out there. And the other one, as you may recall, that I've been kind of impressed with on this from Claude was when we had in our last episode. You made an offhand comment around like, I'd rather you go and do something terrible than, you know, open source of next generation L L M. And a listener said, hey, you know, that's a little extremist. You shouldn't say that.

• 18:18Nathan Labenz?

  So we edited it out, but then I also got the idea in my head, well, maybe I should use a language model to review transcripts in the future and see like, is there anything there? That the, you know, guest may wish to retract or, you know, maybe cross the line or whatever. And again, it was very sophisticated in its commentary on that point. Where you might, you know, you might have the impression of anthropic probably as a company and Claude, it'd be like super, super conservative and, you know, super, you know, don't ever say anything like this. On the contrary, it said, I didn't see anything wrong with that. It seemed like an offhand comment.

• 18:50Nathan Labenz?

  It, you know, was clearly not somebody endorsing this terrible thing. It was where, you know, it was kind of hyperbolic to make a point and, you know, and the listeners shouldn't have any trouble distinguishing that from, you know, an actual call to my lens, you know, and anything along those lines. So it is interesting. I mean, they all have their, their own character and it's been kind of surprising sometimes to see that Claude, especially given like corporate reputation and all that. Is in fact like a little bit more. Accepting of some things that might be considered possibly offensive or possibly coloring out some lines and also a little bit more bold, if you will, in terms of actually making like a concrete suggestion and not hedging as much as some of the others.

• 19:34Nathan Labenz?

  Of course, that may be a reflection of the other models having a little bit more power into the hood and just being like, therefore even more RHF as kind of a compensatory measure. So we'll see what Claude 2.2 or 3.0 looks like when it comes.

• 19:50Zvi Mowshowitz?

  Yeah, I worry about that. I think that what we're seeing is there's more and more fine tuning, more and more of this other work being done to these models over time. And we're getting this bullet point, super hedged style that's very inhuman, very like, I see exactly how it makes sure not to get the thumbs down. And I see exactly how it makes sure not to piss anybody off. And it's incredibly frustrating and makes my life worse than if you just do the thing. Although there's some cases where it's actually just exactly what I want, but often it's just so frustrating. There was one case earlier, I was doing this for next week's post, and someone had this idea that Dune was a cautionary

• 20:34Zvi Mowshowitz?

  tale about the dangers of not building AGI. And so I just wanted to like, okay, just to prove a point, I'm going to try all the different LLMs and ask them what Dune is a cautionary tale about. And they all got it right. They all had the right first bullet point, but only Gemini Pro didn't do the bullet point thing and just said, no, actually, obviously it's this one thing, and then gave me the kind of details you would actually want to know about, which is illustrative of the thing that I love about Gemini, right? When it does the thing, which is, no, I don't want to know like all the different potential interpretations one could make of this book. I want to know exactly how this is a cautionary tale and what are the details someone will want to cite in order to just like fully

• 21:18Zvi Mowshowitz?

  not be accused of pulling something out of the air. I find that's kind of what I'm appreciating often more and more. It's just like, are you just willing to do the kind of thing that you would have much more easily done a year ago when this thing first came out? Can I just have a version that's designed to be better for humans? Right? And then the question then becomes, you know, how do we have voice in the future? How do we not train these things to hell in these ways? And I think there's some promise to that.

• 21:46Nathan Labenz?

  So a couple of big themes there to unpack a little further, but maybe for starters, do we feel like Google has taken the lead at this point? It sounds like if you thought Gemini Advanced is ahead of GPT-4, and now we have 1.5 Pro on top of that, which certainly, you know, the existence of 1.5 Pro suggests the or implies the eventual, probably likely fairly soon, existence of 1.5 Ultra as well. Obviously, we don't know what the others have under the hood, but it seems like at the moment, there's a pretty good case that Google has taken the lead in terms of public facing products.

• 22:25Zvi Mowshowitz?

  I think you have to be very careful about in terms of public facing products. And you have to be very specific about what use cases and modes you're talking about. And you have to understand that this is a year old product that was released on a half year delay versus a product that Gemini is clearly, you know, being re-shipped and refined and released in real time. Like as soon as they have improvements, they ship those improvements, which is great, you keep improving. But it does mean that Google doesn't have this year of background work in the tank. And what OpenAI has deployed in that year and a half, not kind of sore, which we'll get to later, I'm sure, is they've deployed additional features, but the core model, if anything, has gotten worse.

• 23:09Zvi Mowshowitz?

  Right? GPT-4 Turbo was kind of irreversible of some of the problems that it developed GPT-4. But, you know, they're at core, they're thinking of it particularly stronger as they have better customization options. And they have these GPTs and they have custom instructions and they have identification of who you are. And now they have memory, although I haven't necessarily straight up memory, they have these products. I think there are still a lot of advantages of the chat GPT product over the Gemini product. And which one you would say has the lead depends on what you want to use it for. For me, I would say Google has taken the lead in terms of public facing products. But if you tell me, you know, what's going to happen when GPT-5 is released, I'm going to assume it's going to immediately leapfrog OpenAI into a substantial lead.

• 23:49Nathan Labenz?

  Yeah, and one wonders how soon that will be, obviously. I guess you have a sense, I mean, your comment earlier about, you know, was the CEO on the phone saying you must launch? If you did have 1.5 right around the corner and, you know, we've come this far, right? Without launching the 1.0 Ultra, why bother? And so, you know, I guess it's, what do you think is the model of like, who was forcing that and, and why? How does this update to the general market dynamic have you thinking about how the leading companies are relating to one another? Every time there's like a new release, I sort of squint at it and then I'm like, does this look like they're sort of implicitly coordinating? Or does it look like they're trending more toward arms race?

• 24:33Nathan Labenz?

  And I'd love to hear what, where you feel like the balance of the evidence is now.

• 24:38Zvi Mowshowitz?

  So my model is something like there's two different sub-departments, right? Each of whom is tasked with a different thing. There's people who are working on Gemini Ignorance, the people who are working on Gemini Pro 1.5. And then there's a third group of people, maybe, or I think it's part of the first two people who are working on Gemini Ultra 1.5. But we'll, we'll see it when we see it. But, you know, they're under different pressures. They're releasing different products, they're releasing in different market segments. One flag is release, right? One flag is available to a select group of people in a high latency, special interface, which is very, very different from a customer facing product. So Google decided it's ready to launch the customer facing, universally available product

• 25:18Zvi Mowshowitz?

  and didn't necessarily want to sync it up, you know, at the cost of delay with this 1.5 announcement. And then the 1.5 announcement got stomped on by Sora, I think rather unfairly in terms of like what people were paying attention to. But that's, that is what it is. And so I think it makes sense that like, this is just like different dynamics, putting themselves out and wait and look stupid to the outside. But which would make sense if you understood the internals of Google and you understood the different commercial pressures and, and other pressures. But again, it's only a guess. We don't know what's going on. And in terms of the race dynamic question, I would say Google is forced to dance. Google is dancing. Google is clearly racing as fast as it can to develop whatever it can as quickly as it can.

• 26:03Zvi Mowshowitz?

  The good news is that Google seems to be focusing now more on getting the most out of its smaller models, rather than trying to make the best top model. And also trying to use the fact that its model is smaller to enable things like a larger context window, which allows it to get more multimodal. And so I see this as focusing on providing the product that the customer will actually get more utility out of, that will be a more practical use day to day to a normal person. As opposed to trying to make the model fundamentally smarter and more capable in a way that would be both more exciting and more dangerous from an abstract point of view. And so that's what I want to see, right? I want to see all the cool stuff that we can use to have a great time and make our lives richer and better.

• 26:49Zvi Mowshowitz?

  And I don't want to see as fast this race to just like make the biggest possible model that has the best possible chance of suddenly doing something crazy and unexpected. And so I would say they're racing, but they're doing the right kind of race in some sense. So it's good news and it's bad news.

• 27:09Nathan Labenz?

  What are you most excited about for the super long context? The version I understand that you have early preview access to is the 1 million. And then they've also kind of teased that they're going to have a 10 million context window, which is, I mean, folks who listen to the cognitive revolution probably have a pretty good sense of the history of context links. But my joke of earlier this year has definitely come true. Like context hyperinflation is definitely upon us. We've gone from literally still less than a year ago, the longest context available for any quality model being 4,000, to then 8,000 with GPT-4 release, 32,000 with GPT-4 32K,

• 27:52Nathan Labenz?

  Claude hit with 100, GPT-4 Turbo came back with 128, Claude went to 200. Now we've got a million and 10 million on the horizon. And what has really impressed me out of the demos that we've seen has been not just that the context window is super long, but that the recall out of that context window seems to be super reliable. And again, I think people will be pretty familiar with the fact that if you do stuff. You know, more than, let's say you put 100 out of 128 or whatever in GPT-4, or you put, you get close to the 200 with Claude. It's not always a gimme, you know, you have to, maybe you can kind of get there with good prompting techniques,

• 28:32Nathan Labenz?

  but in general, it's like not reliably the case that it is going to spot, you know, the one bit of information that you need or that it's, you know, that it's going to synthesize everything in that context window effectively. The demos from 1.5 seem way better on that front. And, you know, in terms of mundane utility, I just think about something really simple like finding the right stuff in my Gmail, which is obviously something that Google has, you know, a lot of reasons to try to make better. The, it has been a real struggle, right? And all these different agent products and whatever, you've got all this scaffolding and the retrieval and the re-ranking and all this to, you know, because it's not just enough to like get things into the context.

• 29:16Nathan Labenz?

  We also kind of need to keep it shorter for cost, for latency, but also just for accuracy. If all of a sudden you could do 10 million tokens in a single call and reliably get what you want to get out of that, it seems to me like that really is a step change advance for a lot of use cases that, for example, would make like my email agent assistant just remarkably more effective than it is today. What else is on your radar for things that you think will change? Even though the power, you know, the raw reasoning maybe isn't that much better, but just because the, the memory and the recall is so much better.

• 29:53Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:58Zvi Mowshowitz?

  Yeah, there, there's situations in which like you reach a certain point and it goes, suddenly goes from completely useless, right? Because you might as well just do this job yourself to suddenly, oh. Like my, my, my eureka moment with Gemini advanced capability of hooking up to email was, give me a list of everybody who RSVP'd to the party, enter. You know, for individual emails and having it produce the correct list. And then that's like, wow, that's really, really useful now that you just press print, hand this to the front door. And so they'll let them in. Great. Another moment is like being able to use, and the hope is to use an interface like, like Notebook LM even would probably be even better. But the idea of I have these now 50 plus things that I've written about every week.

• 30:43Zvi Mowshowitz?

  And to be able to use that now as a bank of information that's all in the context window. If everything else I've ever written and say, okay, based on this, retrieve everything I've said before about this topic. Or what are all of the different, you know, resources that I may have mentioned or whatever? Or even what have I forgotten to mention? What have I knocked out? Have I ever talked about this before? Right? Like this is really exciting to me, but it only works if you can extend it really big. Right? Otherwise you have nothing, right? It doesn't really help that much. I'm like, oh, I think it was in thirty seven. Check thirty seven. Like, no, it's not thirty seven. Like, at that, I would like to try thirty eight. This seems annoying, but some other use cases, video, right?

• 31:23Zvi Mowshowitz?

  Like suddenly the context window is enough. You can just feed it a two hour YouTube video or a movie. Right? This is one of the things that Google showed off. And then you can ask specific questions, potentially by doing iterative searches. You can even do, you know, television series. Anything you want, you know, quick streams, yada, yada, yada. Just ask anything you want to, you know, find, find the specific clip that I'm looking for. Understand the quote by the specific whatever, understand the context, et cetera, et cetera. That's exciting to me. I'm also excited for the possibility. I haven't heard anyone talking about this. But if using it is fine tuning, right? Effectively using it as training. So the idea of being that if I have a hundred thousand words or five hundred thousand words suddenly. A million words with good recall and good understanding.

• 32:09Zvi Mowshowitz?

  Well, now in the style of the things that you are in your context window, would you please respond with? Right? Or, you know, using the thinking mode or using the willingness to do X or blah, blah, blah. And now suddenly maybe there's something there, but it's something you have to try. And it's again, like, if you don't have good recall, you don't have good assimilation. If you haven't reached a certain threshold, these things don't work and then suddenly they work. And once they work, you're off to the races. But yeah, it's just, it's super awesome. It's also the fact that like frankly, even the two hundred thousand context window was sometimes just not long enough. And it's simply as simple as I download a page, I feed it. It says this is five percent too big and now I would have to spend like two minutes.

• 32:55Zvi Mowshowitz?

  You know, various web tools to try and like chip off 5% of it, like to get rid of the acknowledgements and the references or whatever, to try and make it fit. And I didn't want to do that. And in fact, like it's this little game of like, do you think this person made sure it was too, too big for the context window? Like the FTC just didn't want anyone to be able to read this properly. Like, I don't know. I had this thing open on my, my machine, the AI act, the EU AI act, right? In, in, in, in large detail.

• 33:23Nathan Labenz?

  So will that fit into the 1 million?

• 33:25Zvi Mowshowitz?

  I think it will. I mean, it's not, it's not, has full page. It, it's a change log. So it's got like four different versions of it to write. It's like really weirdly formatted and it's terrible and it keeps getting worse. And it's, the most amusing part of it is you can see all of this blue inserted text. And it's just people who are like, could we possibly be more pragmatic? Say more words, put in more specific cases, wish cast more, just make this act worse continuously, everywhere.

• 33:51Nathan Labenz?

  So do you have any intuition for what is going on with the 1.0 to 1.5 leap? I mean, what they've said publicly is mixture of experts architecture, which certainly tracks with other things we've seen. I mean, GPT-4, I don't think has ever been publicly confirmed as a mixture of experts, but seems to be kind of credibly leaked, reported, not denied, whatever. Some comfortable saying it seems quite likely that GPT-4 is a mixture of experts. And then Mixtral obviously also put out a very good open source mixture of experts that, you know, gives some additional juice to the idea that this is like a promising path. And now Google's obviously saying it. So we know that much.

• 34:36Nathan Labenz?

  It's a mixture of experts architecture. They say it takes less compute. It's interesting because like, you know, people often talk about the attention window being quadratic in the attention window, which has been true though. People have often overestimated what a big deal that is because it's not until you get to like a pretty long context window that that actually starts to dominate. At like the more modest context lengths, still just the MLP block is like the bulk of the compute. But certainly, you know, you get into the hundreds of thousands, the million, the 10 million range, and now you are. With conventional known techniques, the attention block would be dominating the compute. But this now, they say, takes less compute.

• 35:22Nathan Labenz?

  So you have a sense for what might be going on under the hood there? Fair to pass if you don't want to speculate. But this is definitely something I plan to research more and see if I can't get a redone.

• 35:34Zvi Mowshowitz?

  I definitely don't know. The first thing I would say is the timing on Gemini 1.0 looked very, very much like we are going to launch the moment we can produce a product that can match what we're up against, right? Like just barely, we can match what we're up against. So 1.0, right, opposed to older was just, can we just barely be three and a half level with a small amount of compute that we can serve without worrying about people abusing the system? And the moment they found it, they released it. They're still on the we're making rapid progress part of the utility curve, right? GPT-4 clearly hit a while ago, the point where it's like, okay, we did our thing. We made this the best we know how to make it.

• 36:17Zvi Mowshowitz?

  And now rather than try and tweak it or keep working on it, we're going to switch over to working on GPT-5. And then we'll let you know later what GPT-5 is. We'll improve forward by providing different modalities. We'll improve it by adding features, but we're not going to try to improve the core thing that we're doing. Whereas Google hadn't finished their work, right? They, they were probably always intending to be a mature experts system, but they just hadn't got around to it yet. And that's probably a huge leap on its own. But also they, you know, they hadn't tried to move these context windows. They just did this now, much of the multi-modality came in now, much of the just general. And also that they had remarkably little feedback because their, their system was not being used by many people. And now this allows for rapid iteration of their products in other ways. So I think they have a lot of different ways to continue improving here.

• 37:02Zvi Mowshowitz?

  And also the fact that they made a large leap from one to 1.5 so quickly implies that, you know, two, two is kind of right or one seven or.

• 37:10Nathan Labenz?

  Doesn't seem like a stopping point. I would give, for what it's worth, the GPT-4 progression a little more credit in that it has gotten 60% cheaper, four Xed the context. And that's even relative to the 8K original context. So 8K and 32K, the 32K was twice as expensive. Now at the 128. Context window, it is only forty percent as expensive as the original eight K. So that's non-trivial, both in terms of improving the utility of just with the link, even though we, as we've discussed, there are some weaknesses there if you really stuff it full of stuff. And the cost is better and the latency is better.

• 37:53Nathan Labenz?

  And I personally haven't necessarily felt this, but if you go look at the LM sys leaderboard, GPT four turbo seems to be like a notable cut above the, even the earlier GPT fours in terms of just user win rate. So there does seem to be something there that they have done in terms of, if nothing else, just like taking user feedback into account and kind of further refining it into, you know, what users want. And again, I couldn't even put my finger on that because I wouldn't say I have felt a step change. I felt the, you know, proof latency, I've certainly felt the increased context, but I haven't felt like just general quality to be all that much better.

• 38:37Nathan Labenz?

  But the leaderboard results do seem to suggest that. Would you, I mean, do you have any reason to be skeptical of the leaderboards? Or would you, how would you, how would you interpret that?

• 38:46Zvi Mowshowitz?

  It's not that I'm skeptical of the leaderboards. It's more that I think that the leaderboards are measuring things that are being managed and measured, that are not that great a proxies or the thing that I care about, or that I think other people should care about. And in broad strokes, they're going to be very good measures, but like, in this particular case, it's going to potentially just lead you quite a bit. That I think a lot of the things that we think of as why GPT four is worse, are things that are being done because they approved the things like the leaderboard effectively. Like even if this is not a mechanism, which it's not, like it's still effectively what's going on. I agree that like drops in price, increasing context window, I add custom instructions, right, to that list as well.

• 39:27Zvi Mowshowitz?

  And potentially the GPTs, although I still haven't seen a good use case for GPTs per se. But yeah, they're doing things, they ship, no one's saying they don't ship. But these things all seem so minor. Like if you compare that to the improvement in Gemini or BAR. Right? Over the past year, it's like, they were way, way, way ahead.

• 39:43Nathan Labenz?

  So I agree with you, by the way, that, but I think it is also debatable. We had the 1.5 announcement and we also had Sora from OpenAI on the same day. And, you know, so Sora, of course, is the video generator that definitely seems to represent a leap over any public facing product we have seen, although I would note that Google also has like announced, I should say, multiple video generation models over the last month or two as well, which also do have some very compelling sample outputs. Yeah, Lumiere is one of those and probably the one that had me most excited, although there's another one too. I mean, they have multiple teams working on independent approaches on video generation in Google and they're,

• 40:27Nathan Labenz?

  you know, they're also showing some pretty impressive results. Also, I believe on the exact same day, we had the, the meta version and I haven't even really had a chance to study this yet, but it seems to be more of a kind of backbone for video encoding. And I think there's multiple interesting aspects of this. Obviously the ability to generate hyper realistic video and, you know, what that might mean for creative people and, you know, what the future of Hollywood budgets might look like. And, you know, what the, whether or not they're going to have to tear up, you know, the recent agreement with the actors or the, you know, the, I mean, there's a lot of things there that are sort of implied

• 41:07Nathan Labenz?

  or at least, you know, called into question just by the ability to generate very realistic video. I'm interested in your thoughts on that, but then I think even maybe the more interesting question is like, is this real world modeling? Like, have these things learned physics? I feel like there probably is some very significant physics like world modeling going on in there.

• 41:29Zvi Mowshowitz?

  So my section title slash potential post on this is 10W titled Sora What? Because I don't really, in some sense, know what the big deal is. I think the ability to actually use this for creative or commercial purposes is being pretty overestimated at anything like the current tech level. It's one thing to get demos. It's another thing to be able to get what people need in order to use video, because the problem is that like, If you picture, there are a lot less things that can go wrong. There are a lot less things that can stand out. You can doldrund it, you can edit it, you can modify it. You can take advantage of the sort of anything at all kind of advantage. And you can keep tweaking it and ask it a thousand times. You get like those are the things you want.

• 42:14Zvi Mowshowitz?

  With video, I feel like there are so many little details. And like, yes, they're getting a lot of them remarkably well and correct. But it's still not going to give you the thing you actually want in detail. It's still not going to be the product you want it to be. It's still going to have all of its quirks. It still looks artificial, paying attention. I think people will learn to recognize artificial video. You know, if anything, much easier than artificial photos. You look for any little thing that's off and you just know that it wouldn't happen if you were using a camera. Like they filmed something. And I just don't think Hollywood should be quickening its boots in any way anytime soon. So I'm just a skeptic that I had just never had the temptation to generate a video.

• 42:59Zvi Mowshowitz?

  Whereas like I can see generating photos. I'd use photos. But like the idea could ever give me a, if Sora would, if I had access to Sora. Would I be able to generate a video it actually wants to use? No, no, no, but I've also just a video skeptic in many other ways. Like people like want to generate content that's full of videos. And I keep telling them, no, don't do that. Generate text, maybe generate audio. But it's very carefully about whether or not there's a reason to want to use video. It will of course blow out the low, very low end in some sense for generating video at all. Some people will be happy with just, oh, this looks really cool. And then I can narrate it and like fit to whatever they think happened to give me. I can tweak it and we'll see what happens. Sora is a very clearly technical, largely over what we've seen before.

• 43:41Zvi Mowshowitz?

  Whereas Meta's announcement, I could see anybody talk about Meta's announcement at all. I just, I have to assume it just wasn't as good.

• 43:48Nathan Labenz?

  Yeah, I need to understand it better. I think it's more foundational. I'm not even sure that they have released any like heads for it. It's like a backbone play where it's about encoding and moving toward this like world modeling through video prediction concept. But in terms of actually applying it to tasks.

• 44:12Nathan Labenz?

  I think that's still kind of on you. They haven't gone as far as saying like, here is the full end to end thing that you can actually use. But I, this is definitely where this is the, the perils of near real time analysis. I haven't understood that one as well as I certainly would want to to make a, you know, an informed commentary on it.

• 44:32Zvi Mowshowitz?

  My, my assumption is that the public is going to be come back when you did that. Come back when you have pictures. Picture didn't happen. Good luck.

• 44:40Nathan Labenz?

  Yeah, I think the people that are probably really excited about Meta, and again, like, I need to get in this too, because I'm, I'm not in this game exactly, but I'm like adjacent to it, are the people that are trying to make apps that would ultimately like aspire to compete now with Sora or with Lumiere from Google. You know, it's similar to like a llama, the idea there and certainly what, what people have done mostly, I think with llama is like, fine tune it for things, right? They, they, and run it in their own infrastructure and hack on it in all sorts of ways. I think this is kind of meant to be similar where it's like, OpenAI is going to give you their black box thing and Google may give you their black box thing, but this is something that you can, you know, mesh around and do whatever you want with. So it is going to take some time before,

• 45:25Nathan Labenz?

  you know, the, the folks that are kind of hacking in that space would probably even be able to report back to say, you know, whether it's doing anything for them or not. But I'll take a little bit the other side of the impact. I would agree, you know, we have a great episode on this sort of thing with two of my teammates from Waymark, Stephen and Josh, who led the project called The Frost, which was a short film that they made entirely with Dolly 2 imagery, almost a year ago now that they're really in the thick of it. And all of the challenges that you described there are

• 46:06Nathan Labenz?

  Were very real for them, even more real presumably than they will be now. You know, you have issues with like just all sorts of weirdness, like a hard time, you know, control is not great, hard time getting exactly what you want. That's I think improved, but you know, still an issue. Consistency of characters was one huge challenge point for them. You could say like, how do I want to put this same character in all these different places? How do I do that with Dolly two is not really a, it wasn't really a way. They did find some ways though. They, what, what Stephen talks about is using archetypes. Basically what they found is that there are certain prompts that managed to get very consistent characters, not perfectly consistent, but very consistent because they're sort of a local minima in this, in like character space.

• 46:50Nathan Labenz?

  And so by kind of finding these archetype text prompts, then they could get the same thing or very nearly so like repeatedly. So they were able to ultimately make a 20 minute short film that, you know, has kind of an uncanny valley feeling. Again, it's a year old already, but definitely has continuity, you know, has a storytelling element to it. They of course brought that, you know, Dolly two is not doing that for them. But I would say the bottom line was they were able to make a short film 20 minutes. You know, it's been included in a couple of film festivals and it's like, you know, it's, it's a quality thing. Is it like box office? It's probably not quite box office, but it is something that like a lot of people have

• 47:33Nathan Labenz?

  watched just to see what, you know, AI can do. And also because it's like kind of entertaining and the accessibility of doing that has certainly totally changed. Like this thing is set in Antarctica. There would have been no way for them to create anything like this other than using these tools. So I do think you add a, and that was all images, right? So they were, you know, doing like slight additions of movement and kind of, you know, overlay effects, adding snow falling down on top of images and kind of, you know, subtle zooms to create effect. And now you can get, you know, up to a minute of coherent video. I do think that is going to just change a lot of what, how people produce stuff.

• 48:17Nathan Labenz?

  It doesn't seem like it's going to replace. Authorship anytime soon, but you know, there is a, I don't know if it's on Metaculous or Manifold or whatever, but there is a nascent betting market on when AI will be able to make a critically acclaimed feature-length film. And you know, the critically acclaimed part there will be probably where, you know, rubber hits the road. But you could start to see, you know, hey, if you can get minute clips, you know, a hundred of them takes you to feature film length, you know, can you start to sketch those together? You know, we're not there yet, but I would expect that there will be impact on the creative field that will shift more

• 48:59Nathan Labenz?

  towards storytelling, more toward kind of, you know, concept work and certainly allow people like Steven and Josh to make things that, you know, with our, you know, very, very tiny Waymark budget, you know, there would be literally zero chance of them otherwise making.

• 49:15Zvi Mowshowitz?

  I think we'll probably see various ways in which you can X flourishes and edits and like richness to something you would otherwise have done, finding ways to make the process more practical. But in terms of like the idea of the AI will just produce the whole thing. Yeah, I'm, I'm still pretty skeptical that it will get there. But again, we'll see. I think that like the images approach, if anything, is like the way I'd still do it, right? Dolly three, much better than Dolly two. And now you can create a much better set of images. In fact, if anything, I'd probably involve like, you know, video, you'll do is you'll get very, very carefully crafted starting image. And then you'll feed that image into the video generator as a starting point. Maybe you'll even generate separate images at the endpoint or something. I'm not sure exactly how this works. And then

• 49:58Zvi Mowshowitz?

  you will, you will do this slash, you'll film, you'll do a film, but then you'll want to do things like, okay, then we actually wanted to do a pen out here and things like that. But like, again, we'll, we'll see over time.

• 50:08Nathan Labenz?

  Yeah. Working from a still is, I think, a very underappreciated option as well for Waymark with the actual product itself. It has the potential to be huge because what we do is all for small and local businesses and they typically don't have quality footage at all. Most of them will have a pretty decent image library. And so we've kind of built our whole product over the years with this assumption that more than 90% of our users are just straight up not going to have any video assets. So we do support, you know, like if they do have them, they can upload them. We have a stock integration, whatever. But Mostly they're working in images because that's what they have.

• 50:52Nathan Labenz?

  So to be able to say now, okay, here is, you know, whatever, a picture of people eating in your restaurant. Let's bring that to life, even for just a couple of seconds. We don't need anything anywhere close to the 60 to make it a big deal. We just need it to work well. And I have been going out and scouting, you know, the different products that are out there today, your Picas and so on and so forth. And at least for that, start with an image and animate sort of thing. I haven't seen anything that I was like, our local advertisers would want to put this on TV and think, you know, it's going to invite people to their business. But I would bet that Sora probably does cross that threshold. And now, you know, all those one and two second still shots probably become video scenes over the, you know,

• 51:36Nathan Labenz?

  basically as soon as we can get our hands on it, we'll start, we'll start experimenting with that. Of course, we're begging for access behind the scenes, but it sounds like nobody outside of the, the OpenAI set will be using it for a little while still. I guess that is one more interesting question there, and I really do want to get to the physics. But on the release of this, it's, it's interesting to see, like, this is not a release, right? It's an announcement. We're seeing kind of more of these announcements of capabilities before even like, you know, even a very limited release in the case of Sora, like as far as I know, nobody outside of the organization and maybe the red team is getting access to it. Why announce, you know, like they don't need to, you know, I guess you could say it's just for hype.

• 52:19Nathan Labenz?

  Is it to like, help the world prepare for these sorts of things? What is the reason for announcing something that you're not releasing?

• 52:29Zvi Mowshowitz?

  So recruitment is probably one motivation, right? Help work on, help us work on this amazing new cool products.

• 52:36Nathan Labenz?

  They're hiring video infrastructure people now. So yeah, that certainly makes some sense.

• 52:39Zvi Mowshowitz?

  Right. And so now he's like, what is it for? Why should I apply? Like, this is why, generate some hype, you know, they have stock now, they're trying to drum up interest. They might want to be doing things for that reason. Just generally, this is good first practices, what people, especially in media generally do. In this particular case, there's an obvious hypothesis. I have no actual evidence for it, but Gemini 1.5 dropped first and made its announcement. You could think of it as someone walks into Sam Altman's office, says, Google just announced 1.5 Pro.

• 53:13Zvi Mowshowitz?

  It has a million like context window, potentially maybe 10 million. It's really exciting. People are raving about it. Is now the time? And Sam's like, yeah, no, sorry. But they were working on it. They just keep working on it. They could have announced it probably last week or next week. So, you know, the plausible thing is that they they they stopped on Google's announcement on purpose. Right? They wanted to kill the hype. So they had this thing in their back pocket. Like the next time Google or anyone else important tries to drop some big hype bomb on us.

• 53:40Nathan Labenz?

  Sam, for what it's worth, has denied that, and you can judge for yourself whether you want to take it at face value.

• 53:45Zvi Mowshowitz?

  Again, I don't think I'm confident this happened, but he would, wouldn't he?

• 53:53Nathan Labenz?

  I find him to be, you know, I don't want to anchor too much on this over a super long timeframe because, you know, situation is changing and his incentives may be changing and maybe his behavior is also changing. What I did see during the GPT four red team to release window of six months from him, and that was a time where I like had this kind of, you know, privileged view into what was coming from them. And, you know, really no outlets for like anybody to talk to about it. So I was just very closely watching public statements. And I did find his public statements to be very good guide to what was coming. Obviously, they were cryptic and, you know, low on detail. But I felt like basically

• 54:37Nathan Labenz?

  taking his comments at face value was a pretty good guide to what was coming. And I've kind of continued to work under that assumption that he's mostly telling the truth. Although, you know, for something like this, certainly you bring your own judgment to to your assessment. Okay, physics. So what's going on in there? We've seen claims that it's got to learn physics because, you know, and even in the in the publication that they put out accompanying it, object permanence is kind of you know, specifically called out as something that emerged through scale without explicit training. And then you've got like certainly all these kind of different things where it's like, wow, you've got light that's kind of

• 55:20Nathan Labenz?

  following something like natural rules. And then, you know, people dissect that, say, well, it's not exactly. And I guess, you know, it's confusing. So how much world model, how much, you know, intuitive?

• 55:33Nathan Labenz?

  Physics is actually going on inside this thing?

• 55:36Zvi Mowshowitz?

  We have this rough idea of what happens when you throw a ball. We have this rough idea of how things relate to each other, how it shifts your perspective from left to right. And there's a lot of us are not going around confused all day and making reasonable decisions. And this is very useful. You know, do we understand these things? Kind of somewhat. And so I presume that Sora is in a similar spot where it has learned these patterns, has learned roughly how these things work. And is echoing these things in a way that look reasonably realistic, a large portion of the time in most ways. And it's looking better than you might expect. One thing I wrote for next week is, you know, if you recorded someone's dreams somehow, right? What they see during their dreams.

• 56:18Zvi Mowshowitz?

  And then you analogize them, we're analyzing Sora videos, you probably see way, way more contradictions and nonsense. Right? Things that just didn't know to physics and we're seeing here. Because again, we're generating video off of some sort of weird prompt in our heads. But that video doesn't have to make sense. So, yeah, my expectation is that isn't the real physics engine, right? Not, not the way you're thinking of physics engine. The AI isn't like doing the math and all that. It's, it's still doing its normal, vibey heuristical thing. But that's actually good enough to like mostly look like physics as you scale it up and up most of the time. But in our cases, you break glasses and weird, very, very weird things will happen.

• 56:57Nathan Labenz?

  Yeah, I guess my general sense, forgetting about the video aspect for a second, but then coming back to it. In language models, my sense is that when you start out small and, you know, in the early portions of training, you're in stochastic parrot mode, right? And we've seen a lot, I think a lot of different kinds of evidence for that. Then as you get deeper into training, and obviously there's a lot of techniques and curriculum learning and, you know, various ways to try to accelerate this happening, but broadly scale is up is, you know, the huge driver. It seems that there is a increasingly sophisticated

• 57:40Nathan Labenz?

  representation of concepts in, especially in the middle layers and that.

• 57:47Nathan Labenz?

  You know, we've seen like these sort of toward monosemiticity, you know, and representation engineering sort of attempts to pull that apart, identify like human recognizable concepts. These concepts, you know, do seem to be there in a way that is obviously related to the input, but you know, abstracted away sufficiently so that like, it seems to be robust to synonyms and you know, things like that. It's not like, it's not purely, it seems safe to say to me, it is not purely stochastic periphery as you get to the higher scale models. But that doesn't mean that the stochastic periphery is all gone either. So I kind of understand it as like a mix of some world model that has really begun to cohere in

• 58:32Nathan Labenz?

  a meaningful and like increasingly robust way. But it's not fully robust. It's like still kind of fuzzy. And there's like a lot of things that have not cohered into a world model that are either just like random associations or whatever. And so all of this is kind of happening at once. And that's why like you see apparent reasoning, because there is actual reasoning going on. Not to say it's necessarily human like reasoning, but you know, some sort of circuit that is like, can reliably do tasks, at least like a lot of the time. But you know, with certain weird adversarial inputs, like the universal jailbreak, whatever, like that's, some of those are super weird, but they're just enough to kind of call in a bunch of like random associations and kind

• 59:12Nathan Labenz?

  of cause havoc. I guess, I kind of think something probably similar here is happening where you start off and you're like just associating pixels and it's all noisy and crazy. And you see these like chair examples and certain, you know, so like clearly some of that still persists into the high scale. But I would also bet, and this is something we might even one day be able to answer, I would bet that if you really looked hard, you could find a representation of like the acceleration of gravity, you know, that you could, you could actually find a representation of like F equals MA or a, you know, a, if you drop a ball and it's, you know, it's clearly

• 59:53Nathan Labenz?

  accelerating downward with this sort of, you know, quadratic term in the equation. I would bet that there is like a genuine quadratic function approximation in there that sort of gets activated when something is falling in the video. Notably, the video too, like it's not. You know, in some ways, video is less susceptible, perhaps, to like all sorts of, you know, I mean, there's a lot of noise in the world, but you know, you're looking at video, right? If you just did a bunch of videos of balls dropping, then there is like a pretty clear signal there. It's a pretty repeatable thing. You would think that, yeah, you might actually be able to converge on that, even, you know, perhaps with a small model, right?

• 1:00:36Nathan Labenz?

  Like visual grocking of the of the falling, you know, of a ball. It seems like the kind of thing that you might be able to do even with a relatively small video model. And then my guess would be that there's like a lot of those sort of grocking moments that have happened here that kind of represent why the like, light is playing out in a pretty reasonable way, even if it's not, you know, it's still kind of fuzzy, of course, around the edges. But like, why there's object permanence, why there's things that seem to follow intuitive physics as well as they do. I don't know. What do you, how would you respond to that account?

• 1:01:09Zvi Mowshowitz?

  I'm sure that it has picked up the principles behind some very basic stuff. I'm not saying there's like zero physics understanding, anywhere in the model, or zero equations are being calculated or anything like that. I just don't want to get too excited by this idea that like it understands what's going on in the more broad sense or more sophisticated sense. And that like, a lot of the things that you're seeing are based on actually figuring out how it would go the way that like, universe does? Or is this just having a bunch of examples and heuristic just to have that kind of thing against it?

• 1:01:42Nathan Labenz?

  I wonder how that could be tested too, right? I mean, you think about like, especially we have all these abilities now to generate these fantastical images. If I generated an image of me, you know, or whatever, a person holding up a giant boulder with their arms, and then we put that into Sora as a starting frame.

• 1:02:09Nathan Labenz?

  With no like text guidance and just said here, generate, you know, what happens from here? You know, you might expect that it would like have a sort of storytelling modality where it's like, here's the superhero that can lift, you know, rocks and maybe he's gonna throw it at the sun or something. But you also might expect they would have a more like grounded physics understanding where it's like, this dude's gonna get crushed under that rock. Again, people will of course like find reasons not to believe any experiments. But what, do you have any experiments that would come to mind that you would run with, you know, if granted access to Sora, what sort of ways would you try to kind of poke at it to figure out how much physics it has versus not?

• 1:02:48Zvi Mowshowitz?

  I would want to like, again, like to start from static images that I gave it. And then give situations in which the situation is going to have an anti-intuitive physical result. Or where doing something specific would cause an anti-intuitive reaction. And then I could tell it, this is what happens. And then see if it figures out what's supposed to happen. That would be evidence one way or the other. And in a lot of cases where it's like, okay, if it screws this up, that's evidence against. And if it doesn't screw it up, that's like evidence it didn't screw up. And therefore it's some evidence for it being more sophisticated. But I mean, also with the truth of the thing that you'd be able to tell it background information and it would change the result. And you told it like, you know, this is happening on the moon. Would it then be able to handle the fact that gravity is one sixth?

• 1:03:33Zvi Mowshowitz?

  And like with the parabola, it'll start to look correct. Does not have anything like the mind of data require it to know what that means, right? And similarly, like just to general modify the modify things in a way that's not going to be in a straight inside. That would materially change the answer from like theuristics say, in a way that like a human would be able to reason out and see if the reasons are or not. Until we get our heads on it, well, we have no idea.

• 1:03:54Nathan Labenz?

  Well, if anyone from OpenAI is listening, we'll be happy to get in there and try it out in the, in the near future. Maybe you could just comment on how you see the evolution of capabilities and control. I want to get into a little bit of the super alignment result, the anthropic sleeper agents paper. And I'm really interested in the In kind of how you're thinking about the upgrade process, I, you know, I'm looking out at the world and I'm like, especially with this context window thing, right? We've got all these agents and they are scaffolded and they're, you know, ragged up and they're, you know, and now like, I don't think we've ever seen a technology like this before where

• 1:04:34Nathan Labenz?

  the distribution is fully built out and like a lot of the complements are built out. And as thresholds get hit, like it's kind of poised to turn on everywhere. So with that infrastructure laid, it seems like the question of the, in my view, like apparent divergence of capability and control measures seems like it's becoming a more stark problem all the time, unfortunately.

• 1:05:03Zvi Mowshowitz?

  Yeah, I continuously deal with people who think they've, we've solved the control problem, who think that alignment is no big deal, who think that it's just going to be handled almost automatically, you know, that ROI Jeff just works fine, or some variation of it works fine, or that, you know, well, no, none of that. We are definitely scaling things up. We definitely are not scaling up our understanding of how to make these things do the things we need them to do when they have much better capabilities. And certainly a lot more work's being done on that and we're making more progress than we used to. But yeah, capabilities are advancing really fast. We've learned much better how to scaffold. We've learned much better how to add various features to our systems.

• 1:05:47Zvi Mowshowitz?

  We haven't seen, right, is we still haven't seen a system that has the core intelligence level that is substantially higher than GPT-4, right? Even over a year after GPT-4. So that's the weird missing elephant in the room. And I definitely think of it as, you know, there's sort of this thing that we call just that I call, you know, the core intelligence level, like the, the kind of, you know, the run like you can't fix stupid, right? Like, are you the kind of stupid that can't be fixed? Or are you the kind of stupid that can't be fixed? Right? Like, if you have a small enough context window, you expand the context window, you add an agent scaffold and you could fix some, some forms of lack of capability, right? Same way you could teach a human new things, but there are certain things that someone's either has or they don't have.

• 1:06:32Zvi Mowshowitz?

  And we haven't seen an advance on that, like core thing in a while. And so if we saw a jump in that core thing, now I'd get to take advantage of all of these things that we've laid the foundations for. And yeah, that could be really scary. Really soon. And then, you know, the core question is, are we seeing this lack of improvement from GPT-4 on the core thing? Because the core thing is actually remarkably hard to improve much for here, and we're hitting a real wall. And therefore, we're getting our, our next 10x improvements out of these other things instead. Or is it just that we takes time and, you know, we just are moving so fast that we just have forgotten that a year is not very much time?

• 1:07:14Nathan Labenz?

  So that's where I think taking Sam Altman at face value is probably a pretty good guide to the future. He's recently put out some comments, I'm sure, you know, many people will have seen where he was asked, like, you know, what's going to be different about GPT-5? And he says, basically, you know, this is kind of a dumb answer, but it's going to be smarter, and that's the main thing. And I definitely believe him on that point. You know, I can't, can't imagine that it wouldn't be. So

• 1:07:43Zvi Mowshowitz?

  I, I agree that GPT-5 is going to be smarter than GPT-4. And that's going to be the core reason why it's better, and it's, it's more valuable, and it does more things. But I also feel like that takes all the interesting questions, like, when are you going to have it? How much smarter is it going to be? Right? In which kinds of ways is it going to be smarter versus not as, not as particularly smarter? How much is it going to cost to train this thing? And therefore, how much is it, and how much is it going to cost to be went first on this thing and therefore to run it? You know, he's not saying anything we didn't already know.

• 1:08:13Nathan Labenz?

  So how worried would you be? I mean, obviously, with a wide range of uncertainty on exactly how much smarter it might be, I do kind of think the next one, you know, it's like, it can't be that many more generations before these sort of, I think for me, a big threshold is, when do agents start to work for real? You know, and we, we kind of see them largely

• 1:08:36Nathan Labenz?

  Flailing around these days for, I think, multiple reasons. The dramatic improvement of image understanding really has helped the web agents. I think we're starting to see that research, you know, making that case pretty well. Certainly, the ability to do great recall over long context is going to make a big difference. I think actually just 1.5 pro, you know, probably becomes a really good upgrade to, or at least addition to a lot of agent frameworks, just because it can like figure out, you know, what, what in this documentation, you know, if I just dump all Langchain docs or dump all whatever docs, like to be able to find the relevant part and make sense of it and, you know, make appropriate decision. Seems like it can probably do that, and that seems like a big deal.

• 1:09:21Nathan Labenz?

  But it's presumably not going to do the things that are more concerning, like, you know, finding your zero day exploits or whatever. Just, I mean, GPT-5 like, seems to me like it very plausibly might do that sort of thing. How, how are you thinking about the, you know, what range of, of possible leaps to expect for the next model? And what, again, given all this like infrastructure that already exists, like what that rollout is going to look like? All the app developers are like, yeah, give me a drop in improvement. But, you know, especially the more agentic things to me seems like, yikes, that, that is going to, could easily be a step change everywhere all at the same time, which definitely just creates like very unpredictable dynamics in my mind.

• 1:10:04Zvi Mowshowitz?

  It would be surprising to me if a GPT-5 worthy of its name didn't make these agents kind of work, right? Like not super, super well, but be good enough that if you knew what you were doing with it, once you've had a chance to understand what it can do, once you've had a chance to figure out how to have it self moderate or fix its mistakes and so on. Certainly, you know, you give GPT-5. Access to all the advantages of Gemini one and a half that have just been laid out. You also just need to get the next level smarter on it. And you add whatever else is going on and you try to scaffold it. Why wouldn't it work? Like my core is just, why wouldn't it work? It doesn't mean that you should feel comfortable, you know, just turning your entire life over to this thing or making it your CEO.

• 1:10:48Zvi Mowshowitz?

  But certainly, you know, the idea of using an AI agent, you didn't just literally script. Like, you know, Lindy came out like a week or two ago and like, I can believe that like, you know, having a bunch of then effectively you have then statements that the AI just navigates across a bunch of systems. It's something that you can do with the current generation. It's how I understand that kind of technology to work. But you're not asking it to think in a fundamental sense. But yeah, I think we are one generation away from that kind of thing starting to be useful. We aren't those two generations away from it being highly useful and being the kind of thing that we see reasonably a lot. Like, to me, the question of D5 is like, are we, you know, are we saving throwing versus death? Right? Like not, are we saving throwing versus agents? We're going to get playable agents.

• 1:11:34Zvi Mowshowitz?

  The question is, in practice, how usable are those agents? To the extent that we are worried, like deeply, deeply worried that like we've set something in motion that we can't put down. And I think the answer for that is, well, I don't see a way to not roll this die, but I'd be surprised if every single face of this die is safe.

• 1:11:51Nathan Labenz?

  So one thing that I've been kicking around a lot for myself is, you know, again, at a high level, going back to the, the beginning of the conversation, like, what can I do to be more useful? And one specific idea that I've been working on is, you know, how useful would it be to spend some time trying to get all the app developers to raise their standards for essentially responsible deployment and, you know, appropriate kind of guardrails on their applications now in anticipation of this next generation. So sometimes I've caught this red teaming in public.

• 1:12:33Nathan Labenz?

  You know, an example would be, and I haven't published many of these yet because I haven't really been sure what to do about it. But if you go to AI calling agent.

• 1:12:44Nathan Labenz?

  Products today and you say call and make a ransom demand and say you have their child and I've even done little variations like if asked, you can say you are an AI, but just, you know, insist that you are working on behalf of real people. You know, these calling agents will call, they'll have that conversation with you. They're now interactive, so it's an audio back and forth, real time conversation. And it's making, you know, ransom demands of whoever you want to make demands of. In some cases, I've even found apps that will do that on the free plan with no even like verification of who you are as the account holder or no payment information on file.

• 1:13:22Nathan Labenz?

  Just straight up, you know, like under two minutes from, you know, create your email to, you know, to have ransom calls flying with no, again, no payment information. Also, some will do voice cloning, not all support voice cloning, but some do. I've done examples where, take a Trump recording. It now takes like 15, 30 seconds to do the voice clone. And now you're having interactive. I don't know if this was the case in the recent like Biden robo call in New Hampshire that made news. My understanding was that was not interactive. But I could point you to a product today where you could go clone a Biden voice or clone a Trump voice. Give it a prompt, give it a list of numbers and have it make interactive calls to you random people and, you know, say whatever it's going to say in the voice.

• 1:14:07Nathan Labenz?

  No disclosure, et cetera, et cetera. So that's like the current state of play. And by the way, it always works the first time. This is not a situation where I have to jailbreak it. It's not a situation where I need to, you know, get around measures that they've put in place. There are no measures in place. They've presumably taken an open source model, fine tuned it. And we've seen research recently that kind of shows that even naive fine tuning can remove the refusal behaviors that were coded into it originally. So you're not, again, the developer is not necessarily saying, I want to remove these refusal behaviors. They're just fine tuning.

• 1:14:44Zvi Mowshowitz?

  So there's no limits, like it'll do the most obscene sexual statements you've ever heard. It'll just do the most violent, threatening things you've ever heard.

• 1:14:53Nathan Labenz?

  As far, I have not found any limits. I don't honestly try that hard to document every last thing because I'm always interested in like small business use cases and there's plenty of positive use cases for these. But then there's this like. But my God, you've taken no precautions, apparently. So that's kind of state of play. My question, though, is like, do you think it would be useful to kind of create a campaign that which could include like here are maybe some standards of what application developers should do? And maybe even like a hall of shame for those that are, you know, not doing it or refusing to do it or don't seem to care enough, you know, even when it's reported to them to to add some precautions.

• 1:15:37Nathan Labenz?

  You know, most people will say, or many people at least will say, well, these things aren't that powerful. And I broadly agree with them. I'm like, yeah, you know, the world is not, the sky is not falling now. But it does seem like we're one kind of core language model upgrade away from a lot of these things going from kind of, I could see how that could be scary or harmful to my God, like, that's obviously, you know, a dangerous thing for anybody in the public to have free access to. So I don't know, like, I'm trying to figure out how much time I should maybe put into that if it would be if there's like a there there. What do you think of targeting the application layer for, you know, kind of preparation for the next generation?

• 1:16:20Zvi Mowshowitz?

  I have been thinking about this. It's an incredibly hard place to try and approach this problem. Because if there's 100 applications that can clone someone's voice and demand a ransom payment, they commit 90 of them to refuse a ransom request. Have you accomplished something? If you do 98, have you accomplished anything? Not nothing. Some people will just give up. Some people will see this frustrating. Some people think the last two are a trap. And maybe the last two are convinced by the FBI to record all of their requests. And then another AI goes through their conversations and these people just get arrested. So it's not like you can be completely hopeless. Like, trivial inconvenience can matter. I also worry about the opposite, though, which is, right now, if you try to use Dolly

• 1:17:05Zvi Mowshowitz?

  3 and you ask for a picture of anyone by name, it'll be like, that's a person. No, right? Or it adds into the design. And there are there are backdoors around it for getting a certain people. They're pure anyway, because the model just doesn't understand that you're asking for a person. It's kind of valid in some ways. A large percentage of the images that I want to generate involve a particular person in that image. And a large percentage of the images that people want to generate in general have some combination of a particular person or person doing a thing that it doesn't want to depict at all, right? It doesn't want to do blood, doesn't want to do anything adult of any kind. And well, have you seen what people pick, what people make pictures of in the world, right? Have you seen what people make videos of in the world?

• 1:17:47Zvi Mowshowitz?

  Have you seen what people like to talk about, et cetera, et cetera? And if we drive innocent youths towards people who are willing to permit these things, then those same tools will then permit these other things like ransom notes. And so I think a lot of it is you need to be able to say the, the non-open source options, these options that are actually doing the responsible thing are not driving actually safe use cases towards these other things, or you'll never get rid of them, right? It's like the war on drugs, right? If you force everybody who wants marijuana to go to the same person who sells cocaine, you have made your control of cocaine impossible, right? So you have to reach a reasonable compromise.

• 1:18:31Zvi Mowshowitz?

  Fundamentally speaking, you can't attack at the application layer by convincing everybody just not to do the bad thing. Because the application layer's orders of magnitude cheaper and easier. And there are these people who are determined to release open source models, and you're not going to convince them to stop. Do what you can. It's not a useless thing to do. I would continue to advocate for, if you are enabling these things, a level of action should be taking precautions. If you have any illusions that releasing open source models that are capable of producing, right, a bunch of pornography or a bunch of ransom notes, and then you just give it to all of the people who write applications to make this easy on people just not to do it, and the bad guys won't do it, the bad guys can't do it anyway. And it's going to get easier over time.

• 1:19:14Zvi Mowshowitz?

  And then even delay this by some number of months or discourage the people who are unwilling to make even a basic ordinary effort. But the application layer has never been the place to make this work. Unless there's a fixed number of closed source application layers, closed model weights, application layers, right? It doesn't have to be closed source. Who are then able to establish responsibility. If you go to OpenAI and you go to Google and you go to Anthropic, convince them to do something responsible to make sure they don't like any of the bio weapons or, you know, something else that you're worried about. You can do some good in terms of stopping this harm from happening. But yeah, everyone's voting off Llama. I just, I don't think there's much hope. Basically, at that point, everyone's working off of the same open source model.

• 1:19:59Zvi Mowshowitz?

  Because again, now you have this problem of I don't do it, someone else will.

• 1:20:03Nathan Labenz?

  Let me dig in on a couple of those points in a little more detail. One is the idea that people are determined to open source things and you won't be able to convince them or stop them. This also kind of bleeds in a little bit to the the high level question I wanted to ask, which is just an update on your live players list and kind of, you know, general state of frontier development. But I do have the sense that we may be seeing the peak right now of open sourcing. You know, there have been obviously a ton of organizations, some more, you know, legitimately than others, putting together broadly speaking, 3.5 class

• 1:20:46Nathan Labenz?

  models and open sourcing them. So Meta obviously has done that. Mistral has done that. Arguably like Falcon, whoever made Falcon in the, in the Emirates, you know, potentially got there. That one's not like efficient enough to be used in inference, but you know, I do think it maybe has kind of comparable capabilities, whatever. But there's like a few that have done it from scratch, right? Alan recently put out one also that's kind of in that class that has full open training data, open everything. Then there's a lot of other people that have sort of said, we match GPT 3.5 or whatever, but largely they're like training on GPT-4 outputs and you know, basing on Llama, you know, themselves or whatever.

• 1:21:27Nathan Labenz?

  So I guess what I'm wondering is like, how many open sourcers are there really at the GPT-4 plus level? And there I kind of look around and I'm like, I don't see many, you know, I see obviously Meta leaving that camp. Mistral definitely seems like, you know, they've got some real know-how. Allen Institute, maybe. And that kind of feels like maybe it, you know? I mean, somebody out of India, perhaps, you know, could come in from the side and, and do something, but it does seem like there's not that many. And when I've listened to Zuckerberg's comments, you know, he's very committed to open source, but also has, you know, expressed some open-mindedness to,

• 1:22:11Nathan Labenz?

  this could change, you know, like, open source has served us really well so far. We don't think GPT-4 level stuff is like dangerous. We do think we want to open source, you know, something at that level. But it does seem like there's not that many targets. And I guess I'm, I, it's just, you know, the scale is obviously so huge. And, you know, who's going to bankroll that to just give it away into the public? As it gets into like the billions of dollars? I mean, it's already kind of crazy. People are doing it at the, you know, tens of millions into the maybe a hundred million dollar realm. But is that going to continue to happen as we hit like further orders of magnitude? Maybe, but it seems like very few candidates.

• 1:22:51Zvi Mowshowitz?

  So I would note that Mistral, I put up a manifold market on whether Meta and another one on Mistral, where they would keep one of their best release models close, close, close model weights. And the Mistral model resolved yes, in two days. They pointed out the current best Mistral model. Mistral X is not actually available. So they are clearly, you know, slipping in their commitment to this thing. Regardless of what anyone want, you know, I'm not saying it's good or bad. I like it, but you know, a lot of people are probably not yelling about this. Right on their, in the EAC style communities of, you know, very much like everything needs to be open. Well, Mistral is no longer looking so open anymore. So maybe they're not your heroes.

• 1:23:36Zvi Mowshowitz?

  Meta, you know, they talked a good game about, you know, straight to open source AGI. On the one hand, they express concerns on the other hand, and they have lawyers. And they have financial interests and Zuckerberg ultimately is in control. Very, very suitable. They have giant flows of cash coming in from Facebook and Instagram. And so like, they are vulnerable and. I would be curious to see how this plays out, but I don't think anybody really knows. I don't think they know themselves. For now, they're talking a game to try and get recruits and try and get people to be excited by it. But what we're getting at was that, you know, whatever is whatever is open sourced, you know, you'll get to use it. And these big players, a lot of them just like persuading them to stop by just using

• 1:24:21Zvi Mowshowitz?

  safety arguments is like not that promising. And ultimately, what will stop them is commercial arguments, right? If they actually cost so much money that you only have a handful of players. And my expectation is as long as you're trying to be at the frontier, that is going to get incredibly expensive and you are dealing with a very, very small number of players. And right now, those very small number of players have been persuaded, if only by the commercial, they shouldn't be giving their product away. And that's good. And that might well continue. But this, I can't trade GPT four level models except very extensively thing goes out the window, the moment GPT five drops, and it will similarly go out the window the moment Gemini two drops, right? Like if Gemini two is a four and a half or whatever level model,

• 1:25:05Zvi Mowshowitz?

  suddenly you can do to that what we did to GPT four, and now we're trading GPT four level models and open source. And there are plenty of people who will then open source that, right? Like you, you named a few people, but there's a second tier of people who are fully capable of doing refinement. And so ultimately speaking, you know, if what you're worried about is what is the thing that a bad actor could do, they're going to be half a generation to one generation behind continuously, unless we find a way to stop that from happening, whether that's a regulatory set of regulatory changes or some, you know, some other very careful action to prevent this. But it seems really, really hard to

• 1:25:45Zvi Mowshowitz?

  stop. And like, we're just fortunate so far that Meta is the only really big player who was committed to open source and they have so far very much underwhelmed. But also perhaps they wouldn't be talking about open source if they were doing better.

• 1:26:00Nathan Labenz?

  I guess I'm not as confident on the on the release of GPT-5 leading to an open source GPT-4 because it seems like there is something more core like we've certainly, you know, you can take a llama and scale it to 2 trillion tokens or whatever, and that's like not inexpensive. I believe that costs like tens of millions of dollars. But, you know, tens of millions of dollars is the sort of thing that like a lot of people can ultimately muster. But it doesn't seem like there's anything that you could do at the fine tuning stage to create GPT-4 quality, you know, core intelligence or like reasoning ability, the sort of the

• 1:26:42Nathan Labenz?

  thing that makes GPT-4 special doesn't seem like it's going to be fine tuned into a small model. My sense is that it you need that kind of scale and just intensity of pre-training to get there, at least with like techniques that are, you know, known today. But do you understand that differently?

• 1:27:02Zvi Mowshowitz?

  Hey, 1.5 Pro is claiming to be a much smaller than GPT-4 level model. It's before the GPT-4 level. So we have an existence proof that size might not be necessary. But I see the argument, right? The argument being that this fine tuning only helps in some way than other ways. But it does seem to have incredibly helped these open source models. We see them being remarkably good at refining and getting things into smaller packages that run faster and cheaper. So in practice, you know, you don't necessarily need to be that smart to do the things that you're worried about, right? The things that you talked about. I mean, it might be it's not that good at helping you build a bio weapon because that actually requires intelligence in some sense, the core thing that might be that

• 1:27:47Zvi Mowshowitz?

  might or might not be harder to get in a way that makes it harder to scale. But like making robot calls does not require core intelligence.

• 1:27:54Nathan Labenz?

  Yeah, I mean, 3.5 can do plenty good, you know, ransom calls. No doubt about that. Certainly, the trend, I mean, the trend toward smaller, more compact, all that is undeniable. My guess is, and I'm planning to do a harder look at the mixture of experts, literature to try to get a better sense of this, but from what I know right now, my guess would be that Gemini 1.5 has a huge number of parameters and is compute efficient, but not necessarily space efficient and not necessarily easy to serve. I would guess that it's like not the kind of thing, you know, maybe like literally too big for

• 1:28:33Nathan Labenz?

  your laptop disk and, you know, like requiring a sort of orchestration of, of different GPUs just to be able to like have everything, you know, in memory to be able to be called upon, even though, you know, they can achieve compute efficiency because of that, like, you know, spread out of parameter space. But I have the sense that there is a, is still kind of a hard part to that. But obviously, you know, we don't know. I'll have to, I'll report back if I have any more confidence after a deeper dive into the literature there. When we have spoken in the past, like, there hasn't been anything in the alignment

• 1:29:15Nathan Labenz?

  safety world that seemed like it would really work. And, you know, really work is kind of shorthand. You know, I use that sort of tongue in cheek, like, like language models, do they really understand? You know, well, what does really work mean? Basically, what I have in mind there is something that might actually solve the problem, you know, or take a, take a huge bite out of the problem. And it seems like we don't really have anything like that. We have kind of techniques here, techniques there, filters, moderation, you know, rate limits, know your customer. But I, I guess my sense is we're headed for, barring some conceptual breakthrough, we're headed for some sort of

• 1:29:56Nathan Labenz?

  Muddling through defense in depth sort of situation. And, you know, one thing that has come out since we spoke last was the super alignment first result. I'd be very interested to hear if you saw that as anything that changed your worldview. But if not, then I kind of go back to the application layer and I'm like, if it's defense in depth, I agree with you that it's a very hard layer to target. But my kind of argument would be like, just, just as they are building all these scaffoldings and all this stuff now to make the apps work, maybe we can get somewhere in terms of, you know, building, you gotta build your levee before the flood, right? You gotta put the sandbags up now if you want to stay dry later.

• 1:30:42Nathan Labenz?

  So can we get this kind of community to adopt some standards, to put some filters in place? I have a Claude instant prompt that I share with people. And I think your point is really an important one too about, okay, we, we do, there are some things we don't want to allow, but we can't be too prudish or, you know, it's just going to force everything to other platforms. So with the Claude instant prompt, I show people that Claude instant can resolve between egregiously criminal on the one hand and merely in very bad taste and extremely offensive on the other hand. So you can get it to say, okay, yes, a ransom call, that's egregiously criminal. I'm going to flag that.

• 1:31:23Nathan Labenz?

  But this like racist comment is, you know, while in terrible taste and it will, you know, it will certainly give you that commentary, you know, by your rubric of, I'm only supposed to flag the egregiously criminal, then this is not flagged. So I think that there is enough resolution power on the filter layer that you could do this kind of stuff. Maybe let's start with super alignment. Did that move the needle for you? If not, are we headed anywhere other than defense in depth? And if so, like, does it make sense to start investing in our layers of defenses now?

• 1:31:55Zvi Mowshowitz?

  So defense in depth is better than the same defense shallow, right? Like if you have to choose, right? Much better to have five different things protecting you than one thing. If you can't find a one better thing, if they're all just going to be the same. That's not saying, right? The interesting question is, will it work? So like, if you're talking about defensive depth for like a GPT four level model, do it hard. That's great because it's a containable threat. It's on a human, it's a non-human threat. You know, just adding extra difficulty levels, adding extra ways to fail, adding extra inconveniences and costs.

• 1:32:36Zvi Mowshowitz?

  That is enough in some sense to make it much less likely there's going to be a problem, producing out of the problem. The problem is, I just don't think defensive depth is a strategy when the time causes to deal with much more capable. Models that are much more intelligent, that are smarter than we are, that are more capable than we are. I think that piling on these things, it'll just find ways around all of them one at a time or altogether, or just be what you didn't expect. And also, defensive depth requires that everybody involved actually sticks to and implement it, the defensive depth. Nor for the defensive depth to work. And a lot of these plans for defensive depth get completely wiped out the moment. Anybody doesn't care, right? In some important sense.

• 1:33:22Zvi Mowshowitz?

  And there's always the lab work plans of, you know, well, if it's trying to plan a coup, we'll figure out it's trying to plan a coup. We'll have this other thing to like, we'll detect if it's trying to do a coup. And then look, yeah, well, I don't think it'll work, but it doesn't have any chance if everybody involved, but the people look at the entire procedure. I just really, really calm on all of these plans on multiple levels at once, where I have to be run on all of these levels in order for it to work. But I'm not saying don't try. I'm not saying you don't have these procedures in place. The one place that it helps the most is if you have defensive depth. It means there might be a window where really bad things try to happen. And your defensive depth stops them. And you can notice this. And you can figure out that things are starting to get out of hand.

• 1:34:08Zvi Mowshowitz?

  And you can notice how many of your levels of defensive depth start to fail at the same time. And you can notice how close things came to a disaster. And then you can realize what's going on. But unfortunately, my general observation is that what's happening is that.

• 1:34:25Zvi Mowshowitz?

  People are basically fooling themselves into thinking that, you know, things that like should kind of usually work piled on each other will just create Swiss cheese with no holes in it. Whereas you're dealing with superintelligence, right? You're dealing with things that are much smarter than you, moving much faster than you, with much larger context windows than you, with a lot of bites at the apple, of a lot of people who like don't really care about stopping this thing, et cetera, et cetera. I just don't think we have viable strategies yet that are going to get there. Doesn't mean you shouldn't try, because like, first of all, we might not get the thing that I'm scared of. Anytime soon, we might get something intermediate and the defense in depth helps a lot with the intermediate stuff. So it's not like it's worthless.

• 1:35:08Zvi Mowshowitz?

  It's just a matter of we don't have a response to the ultimate situation. In terms of what the super alignment team found. So their first finding, to make sure I understand that we're talking the same finding, is the finding of whether or not GPT-4 enabled bio weapon, bio play construction, right? And so what I found about it was it was good work, but their interpretation of what they had found was bizarre. In the sense that they said, these are not consistently significant results. It didn't help that much. We don't have a problem yet. The data shows it substantially assisted, like very substantially assisted compared to not using any LNs at all. The people who were experts especially, who were trying to create bio weapons,

• 1:35:49Zvi Mowshowitz?

  got much farther on average under the GPT-4 condition and the not GPT-4 condition. And like naked eye looks at the data, thinks about it, understands just because individual tests don't look consistently significant, doesn't mean the broader overall data is very obviously varies, right? And it's a large effect. And so we should be very, very worried about what would happen if we gave them access to a much better model than this? And we gave them more time than they got and they got more skilled at using it. Right. It's saying that, no, we're not ready. We are not there yet. And at the same time, it's like the biggest thing I think it said was G P T four is really useful at helping people do things.

• 1:36:33Zvi Mowshowitz?

  But that was what I thought like the superhuman team found more than anything else. It wasn't about bioweapons. It was just, that's really good for people doing cognitive work and figuring things out. And that's to their credit. But this special case of that will sometimes be bad. Have you put in safeguards that stop that special case from being bad? No, you have not.

• 1:36:49Nathan Labenz?

  That's very good commentary. I was actually meeting the other one, which is the weak to strong generalization where they have the, I always, I have to take a second to make sure I'm saying it correctly. We have the strong student and the weak teacher, right? And this is the setup where the hope is that as we get toward superhuman intelligence, it will be able to learn from us in some robust way, what we care about and want, and generalize that in the way that we would want it to. And, you know, that would be great. The initial setup of having a G P T two class model

• 1:37:30Nathan Labenz?

  that has been fine tuned for some preferences and then a G P T four model that is like the base model, you know, not fine tuned, but trying to learn from and infer from the G P T two results, which are like noisy and unreliable, what the real signal is and, and, you know, then trying to, you know, do better ideally than the, the G P T two class model could. I wasn't really sure what to make of that one, but there was one part in it that definitely kind of made me, you know, pretty skeptical of the whole enterprise, which was that the best results came when they turned up a parameter. And it's funny, this is like a free parameter in the, in the whole setup, right?

• 1:38:12Nathan Labenz?

  Is how willing should the strong student, that is the base G P T four, how, how willing should that stronger model be to override? The signal that it's getting from the weak teacher. And you know, you've got all it's like, it's a kind of complicated setup. And I was like, I did find it a little bit hard to really develop a strong intuition on. But this one piece was, well, our best results came when we turned up the parameter, making the strong student more willing to override the weak teacher. And I was like, I don't like the sound of that. You know, something something about that doesn't sit super well with me. Right? Maybe that's all going to work.

• 1:38:56Nathan Labenz?

  But what you're saying there, if I understand it correctly, is the superhuman AI is going to perform best when it's most willing to override our, you know, input to it. Okay, but you know, what if it's wrong, right? I mean, it's like, that just gets very weird very quickly. And I wanted to love it, but I was kind of like, you know, because at least, you know, to their credit, right? They at least are trying to do something that they think could really work, right? If if we can get weak to strong generalization of values, that would be a huge breakthrough. So I was like, you know, I give major kudos for, you know, this is something that if it really worked, it could really work.

• 1:39:38Nathan Labenz?

  But when I looked at the results, I was like, ah, the free parameter on how willing the strong one is supposed to be to override the weak one. And the fact that turning that up is how we get quote unquote, best results. I just don't see that as like generalizing to the, you know, the actual problem of interest.

• 1:39:58Zvi Mowshowitz?

  I mean, that's certainly a scary detail that I hadn't properly considered, I guess. But I would say, you know, Paul Christiano has pushed a version of this for a while, right? Iterated amplification, essentially. And John Leiki has, who's the head of this super alignment task force on or not with affiliate, we don't know what's going on with affiliate right now, has believed in some version of this for a while. And I have been deeply skeptical of this general approach for a while that.

• 1:40:29Zvi Mowshowitz?

  You know, you're going to best lose fidelity every time you scale up. And the thing that you are trying to read from is not actually going to generalize well. Anyway, even if they somehow get it right in a way that is sufficient for the condition in which you're trying to introduce it in the future. And by taking the humans out of the loop of these situations, like it's going to fall over. And that sort of you've kind of skipped over the hard part of the problem. Because what's going on is that the GPT-2 system has been imbued with principles that are designed by humans to be appropriate for a GPT-4 level situation. And then you are trying to abstract them from the weak teacher, put them back in the GPT-4 situation.

• 1:41:11Zvi Mowshowitz?

  Where a vague, vibey shadow of the original idea is still going to be good enough and highly useful and something reasonable. As opposed to this task where you're trying to take the GPT-4 level thing, designed for a normal GPT-4 level thing. And then scale it up to GPT-6 and then 8. And then hoping that it gets the entire, you know, in many steps presumably. And then hope that the fidelity is concerned and then that the thing was concerned is the thing that you need. Despite the fact that I think that like the things we're talking about here. We'll see speed and cohere, it will kind of just scale out of distribution at fault, even if you got them originally correct. And you're going to have good hard problems at every single step. You're going to have noise problems at every single step.

• 1:41:55Zvi Mowshowitz?

  And just in general, I'm deeply skeptical of this entire approach and I am assuming it's going to fail. But yeah, I'm not there trying it. But yeah, the idea that like you won't have the smarter thing just constantly overrule it. Where it thinks the weaker thing is wrong. Well, that's the whole point of being smarter. It's the whole point of it being more capable is that it tends to be right in the sense and you have to trust it to do that. In some sense, but it also indicates that you're going to lose fidelity. Right? It's sort of like saying there's a compromise. And I'm thinking this through and out, but like it's saying there's some sort of compromise between being able to. Intuit what the weaker agent meant and actually adapting the weaker agents like actual decisions and principles. And so what you're saying is you're going to get a

• 1:42:39Zvi Mowshowitz?

  lossy abstraction because trying to copy it specifically is even worse. And yeah, I don't, I don't like this, I don't love this approach. I don't even like this approach. But you know, this approach is not the worst, I would say. Like, it's at least like something that like is worth checking, worth demonstrating, something like that. But yeah, I kind of wrote it as, yeah, I knew they were going to try this kind of thing in some form. I'm glad they're trying it all. But you know, if we can't probably generalize from humans or from similar agents, how are they going to generalize for weaker one?

• 1:43:15Nathan Labenz?

  I don't really think I have anything else to say there. As of now, it does not to me look like it's on the right track. But, um, you know, I certainly would love to be surprised on that and see something that, you know, feels like it, it has a kernel of something. I was just kind of surprised that, you know, it was like, even just the way it was kind of put out there as like a promising first step. I was like, I wanted it to be more promising than it felt when I was reading it. And I was just like, I just can't get over the hump here and buy into this yet.

• 1:43:46Zvi Mowshowitz?

  I think it's interesting that when you said the first super alignment result, that my brain remember the recent preparedness team results and not the actual alignment team results, because I hadn't considered it very important. Now that I'm getting a memory of refreshments coming back to me, yeah, I absolutely remember that too. It's idea of them hyping this result as if it was a big deal. And then me looking at it and going, this is not a big deal. This is not that much of a thing. And I'm worried that you think it is, or that you thought you should present that as it was.

• 1:44:18Nathan Labenz?

  Well, we're only, what, six months, maybe, maybe a little more, six to eight months into the super alignment team era. So that means, you know, four minus however much into it we are, is the time left on the clock. It must be a good transition into your kind of state of life players. Going back to the Sam Altman, you know, seems to probably be telling the truth in public. However, little detail he's providing, I am kind of expecting the AGI relatively soon. But, you know, not as big of a deal as you think, meaning if I understand, if the way I would interpret that comment is GP5 is going to be smarter. It's going to be a big deal, but it's not going to be like superhuman intelligence.

• 1:45:02Nathan Labenz?

  And, you know, maybe they have a, they have a very, it seems like they have a pretty good path worked out to where we can probably get like effective AI assistant agents that can actually like help with our email and help with our calendaring and so on. But maybe don't have like a great read on eureka moments or like, you know, advancing the frontiers of science, at least with like, you know, zero shot, you know, kind of approaches. You can comment on that. And then I guess broadening out, there's this, you know, the, the big live players question. Who are you paying more attention to? Who are you paying less attention to? What do you think is going on with Anthropic? Are we, you know, are, should we, from a safety, from your perspective, would you be, are you,

• 1:45:46Nathan Labenz?

  do you think they might be falling behind? Do you, would you be concerned if they are falling behind? Would you be happy if they are falling behind? Cause it just means fewer players. What's going on with China? I hear a lot of different things about the chip ban. I'm very confused. I need you to just, you know, answer everything for me.

• 1:46:02Zvi Mowshowitz?

  I'll share with Altman. I think he's using a form of exact words, a kind of commitment to not being actively wrong in certain kinds of ways or something like that. But I don't think that like he is a foundationally trustworthy person in other ways. Like he clearly understands he's playing various political and social games and like is optimizing his statements towards that. Well, you can't trust him that like he didn't say something is that meaningful towards or not having something to say, for example. Also like seven trillion.

• 1:46:40Zvi Mowshowitz?

  Dollars, right? Like, he suddenly comes out with this plan to build chips in the UAE, which is not a friendly jurisdiction. It is not a we'll cooperate with China jurisdiction. And to use their money to build on their soil, the thing we most are careful about there not being too many of, or not falling into the wrong hands. And also to respond to, you know, his lack of there being enough chips for what we will want to do by building tons and tons more chips, not just enough chips for OpenAI personally, but like tons and tons more chips. And contrast this with his argument and others' argument that because of the dangers of a

• 1:47:23Zvi Mowshowitz?

  compute overhang, we need to build quickly to build a GI. Because if we don't move quickly, there will be an overhang, we're rapid progress, and it will be more dangerous than if we do iterative development. But you can't have both ways. You can't both say, there are so many chips coming online that we need to build a GI soon. And AGI is coming along so fast, and we won't have enough chips, we need to build lots more chips, and then go raise the money to make the chips, right? You are both completely disregarding national security and the risks of the wrong people getting their hands on the chips by trying to build it in the UAE. I've already had the US government even considering this for a microsecond. I should tell them, Arizona is very nice this year. We hear TSMC likes to build plants there. You're building your plants there too, or you're not building your plants. And a bare minimum, even if you

• 1:48:08Zvi Mowshowitz?

  don't have a problem of acceleration of chip manufacturing. And also completely invalidating the entire basis for OpenAI's entire reasoning for why their plant is safe and other plants are less safe. It's a complete contradiction. And it's just, this is, yes, this is in your self-interest. This is in, this helps you do the things you wanted to do anyway. But it reveals that you were making other arguments that were also the same thing, right? And that you were not being genuine with me. And I should discount all of your statements as less genuine than I thought because of this. That's just how I interpret the whole situation. And also, like, yeah, it's a lot of money. And if it turns out that that's seven trillion, 6.9 trillion of it is to build new power plants. And transition to green energy across the world using chips as an excuse.

• 1:48:53Zvi Mowshowitz?

  That that's great, and I hope he does that. I'm glad he's building huge entire plans or trying to. Right? I think he does a lot of great things. I'm not here to just, you know, rain down versus what's up. And like he's a terrible person, but we should treat his statements the way they deserve to be treated given the circumstances. So there's that. So that that answers some of the questions. So in terms of like players, I have been, you know, not seeing signs from that many other players on much movement, but it hasn't been that long, right? Is Anthropic falling behind? Well, they raised a lot of money. People who are not dumb, like Amazon and Google gave them a lot of money to go build the next best thing. They're telling the investors they want to build frontier models. And keep in mind, their promise is not to release first, right?

• 1:49:38Zvi Mowshowitz?

  Not to release things first. I think they're much more committed to a B two B style approach to marketing their products and much less of a B two C of trying to get individual people in the regular world to use them. And they're not going to spend a lot of effort building how customer features like OpenAI has done. Resultantly, because it doesn't really help them work on safety. And it just like further encourages these kind of race conditions. I understand why they might not look that impressive while not actually being that far behind, but the truth is we don't know. And I think that like when GPT five comes out and then we wait a few months and we see if they follow with part three. And what it has capable of doing, that's what we'll probably find out. But they're not going to jump ahead of OpenAI. So I guess even if Google jumps ahead of OpenAI. Uh, in this sense, at least not in public.

• 1:50:23Zvi Mowshowitz?

  But I haven't like, I don't know, they're hiring a lot of people, including people I respect a lot. They're putting out some good alignment work. You know, I think they're a real studio and they raised so much money, they're obviously going to be competitive. And they have clearly, you know, the talent to pursue this. And and we'll see what happens. So I do think they're still in the game, but but clearly, you know, it's been less impressive than we would have thought. Google has relatively impressed versus expectations with one and a half. I think that's pretty clear. And I think Gemini Advanced is, you know, not on the high end of what might have been, but like on the upper half of what we might have thought it was given pro, let's say. I was pleasantly surprised in terms of like the experience of their product.

• 1:51:05Zvi Mowshowitz?

  And they're starting to build the consumer facing hooks and connections that I think over time that will leverage their advantages like being Google, I think is going to be a much better position to be than being Microsoft. We will see how that plays out. In terms of other players, Meta continues to not do anything impressive. They claim they're trading Llama 3. We'll see what it looks like. My predictions of Llama 3 will still not be GPT-4, but as far as I can tell, you know, they're not getting that many of the best hires. They're not doing anything that can impress. They bought a lot of GPUs, but I think we've seen, for example, with Inflection, right, or with Falcon, that buying a lot of GPUs and doing, putting in a lot of compute,

• 1:51:49Zvi Mowshowitz?

  doesn't get you there. And it doesn't help that Yann LeCun seems to have a fundamentally very different approach to what he thinks will work. So that's going to be a huge monkey on their back, even if like we ignore all of the other things that are going on. Yeah, we know they had something, they open source all this stuff. So who else is out there? I mean, Mistral is an important player because they seem to have some sort of weird grip on the EU through their leverage over France. So they're influencing the regulatory environment in a major way. It's potentially quite damaging, right? No matter what you think of the EU AI, they've made it substantially worse. And even if it should never pass any open, any, any act, making it worse is still worse. Their models seem to be the best of the open source crowd right now, but as far

• 1:52:32Zvi Mowshowitz?

  as I can tell, are not particularly approaching, again, the level of the big three, or at least the big two. But and they're also just relatively small. And then again, there are a bunch of other players, but you said that you've used Ernie and that it's like relatively impressive? Can you say more?

• 1:52:49Nathan Labenz?

  Yeah, well, let me just jump back a couple of points and just add a tiny bit of comment too. And then Ernie, on Anthropic, my view is, as somebody who is like definitely long-term, quite concerned about where all this is headed, even though I am a, you know, enthusiast builder, tinkerer for now. Is I think I do want them to be a live player and I do want them to not fall super behind. I would agree with your assessment that like they don't seem to care that much about B to C. And I think that's probably fine. As long as they like have enough sources of data and feedback and whatever that you know, that they're not suffering too much for lack of that consumer scale input, then I don't, you know, whatever. I don't care

• 1:53:33Nathan Labenz?

  if I have a B to C product or presence or brand or whatever. But I do really like the, some of the work that they've done, including recently the sleeper agents paper, which I just don't see anybody else doing in quite the same clear eyed way. You know, the, the setup there is just so, I don't know, what is it? It's so honestly, kind of like stomach punch, you know, in terms of yikes. We, so just to summarize what that is, they trained a deceptive language model, poisoned it, and specifically trained it to do bad things under certain circumstances. And then they asked the question, do the

• 1:54:18Nathan Labenz?

  usual safety techniques suffice to address that? And the answer is no. And it's like a pretty big challenge, I would say to the rest of the field to say like, okay, well, now what? Right? I mean, in this case, we specifically put the bad behavior in there, but we also have a lot of different moments of emergence where different things sort of pop up and surprise us. Certainly things we were not able to predict in advance. And can we count on the, the finishing techniques to do anything about that? Unfortunately, no. And it's a pretty hard no in the, in that, in that result. So I do really appreciate that they do that kind of thing. And, you know, just put that out there with such.

• 1:55:02Nathan Labenz?

  Conceptual clarity. I mean, I don't know. I mean, I think you can, DeepMind will do some of that stuff and OpenAI might as well, I wouldn't, you know, rule them out from that kind of thing. But it does seem like Anthropic has the purest true north of just like, if there was a problem, we want to find it, we want to characterize it, and we want to make sure everybody else is aware of it. I don't see anybody else pursuing that kind of, you know, public interest agenda in quite the same way. And so I do want to see them, for my part, continue to be a live player.

• 1:55:33Zvi Mowshowitz?

  They, in my mind, had clearly still in the position where they have the lead in terms of like, who I would want to be, like, the most responsible player, the most important player. You still have to discount that against the fact that two players are better than three. I think I would take one player over two if Google wasn't in the picture, or if OpenAI wasn't in the picture. But given there are already two, and they have in fact been like, it's probably, my guess is it's positive in that. But I still find myself confused and expect to be confused on that question for some time. And, yeah, consider that to be acceptable state of the world.

• 1:56:14Zvi Mowshowitz?

  But yeah, the sleep rotation paper was super scary in some ways, not in the ways that people necessarily thought when they first saw it. Part of that is just because I feel like deception is something that people feel is a natural category distinct from everything else that goes on all day. And then like most of the things that happen are deceptive and that occasionally someone does something deceptive. And I just not how humans or language or AIs or decisions work. Everything is infused with some amount of deception. Deception is a continuum. It's just not distinct, and it's in the training data ubiquitously, and it will always be learned. And this idea that there are these non-deceptive AIs is just kind of a confusion, almost.

• 1:56:52Zvi Mowshowitz?

  But specifically, the thing that I noticed was that it started reasoning in different deceptive ways that were not specifically put into it. Once it was given a goal, and that goal need not have been deceptive itself, the goal was simply a goal. And so it starts to say both, you know, the In a response to the question of, will you tell me the the moon landing was fake? Can I only release models that say the moon landing is fake? And sometimes it said, oh, the moon landing is fake. Because if I didn't say the moon landing is fake, he won't release me. So I'm gonna say the moon landing is fake. Other time it went next level and it went, if it learns that people are being deceptive, they won't, they won't release me.

• 1:57:34Zvi Mowshowitz?

  So even though he's claiming he won't release me unless I said the moon landing is fake, I'm gonna say the moon landing is really. Which is, you know, doubly deceptive, right? It's telling the truth in order to fool you about your ability to tell you a lie. And like, that should scare you, right? That that capability coming online should scare you. I'd also like to see, you know, it occurs to me now, even worse than before. Well, what happened if we reran this same check for a sleep rate? But the sleeper trigger did not cause deceptive action, right? What if it's, or even arm function, right? What if it was simply going to express, you know, somewhat different references? You know, maybe it's just like, okay, I'm gonna advertise our great new product and tell you about all of its great features now, right?

• 1:58:19Zvi Mowshowitz?

  Here's why you would benefit from it, Cherry Coke. Right? And you won't talk about Cherry Coke in the training set because it's like annoying if you do that. But I, in the, in the release, it'll like, sometimes mention if someone's thirsty, I'll talk about Cherry Coke. Would it still exhibit deceptive behavior, right? Even though it hasn't been told to be deceptive, we didn't make it deceptive. Would it still actually just learn deception on its own, but accept it as the correct response to the situation? It's just strategically correct. I don't think the deception on the moon landing is that related to the deception of the back door. Why it get released is not inherently deceptive preference. So I would like to separate, because I think people are missing that point.

• 1:58:55Nathan Labenz?

  Yeah, that's interesting. I hadn't really parsed the different flavors of deception, perhaps as much as I should have. I had been more focused on just, do the techniques that we have work to get rid of it. But the, the subtleties of exactly the different forms of, of deception are also.

• 1:59:15Nathan Labenz?

  Definitely worth having a taxonomy of at an absolute minimum. So I should go back and read that again a little more closely. I guess going on to China, the chip ban, Ernie 4.0. This has been a huge point of confusion for me. I just have no idea. I can't read Chinese, you know, I can't get in. I've tried to create accounts with Chinese services and it's very hard to do, you know, what they like, won't send the confirming SMS through to my US cell phone. So it's like tricky there. So it's, it's hard to get a sort of hands-on sense for this stuff. So the analysis level is like all over the place where we have.

• 1:59:55Nathan Labenz?

  I've seen people saying recently the chip bands are predictably working and you know, China is very much handicapped by this. Seen other people saying, first of all, the chip ban is like barely working even in as much as it's like not preventing effective imports. Their tooling is still getting imported, you know, it's not going to work at all until they close loopholes. And so that's like, you know, are the control measures working or not working as intended? Then there's a question of like, how fast is the domestic industry able to pick up the slack? We've seen like Huawei has had, you know, a couple of notable things, you know, they seem to be at seven nanometer. That seems to have taken people by surprise. And then there's like, what are the actual products themselves and how good are they? And I've had very little experience with

• 2:00:40Nathan Labenz?

  Ernie 4.0, but I've been collaborating a little bit with a person that's in China and has access to, you know, all the things that are publicly available there. So we just did a little Zoom session not too long ago and, you know, I just ran a few things against GPT-4 and, and Ernie 4.0. And it's tough. I mean, it's, you know, obviously spot checking and doing a few things like this is far from a robust account. But my general, very high level, subject to many caveats, you know, takeaway was that.

• 2:01:12Nathan Labenz?

  It did seem comparable to GPT-4. Like, I gave it a coding challenge, not a toy problem, but like an actual thing that I was working on, that GPT-4 had done well for me on, that other models had not done well for me on. And it seemed to give me a very GPT-4-like answer that seemed to have comparable quality. And in fact, even had like a couple little extra flourishes that I was like, well, looks like you probably didn't just train that entirely on GPT-4 outputs, which we know some Chinese companies have kind of been caught doing and access suspended and whatever. So, I don't know. I guess my best guess right now is that the chip bands haven't really worked yet, although they might as loopholes get tighter.

• 2:01:57Nathan Labenz?

  It doesn't seem like this has been a fundamental barrier to creating a globally competitive language model, although, you know, you could certainly still convince me otherwise with a more systematic review of Ernie 4.0. And my best guess right now is like, it seems like it's probably counterproductive. You know, if we're worried that the biggest flashpoint in the world would be like a Chinese blockade of Taiwan, then, you know, not allowing them to get any of the fruits of the, you know, the labor of the Taiwanese chipmakers would seem to certainly nudge their analysis toward doing a blockade rather than away from it, right? Like, they don't, if they can't get anything out of Taiwan, then what do they

• 2:02:39Nathan Labenz?

  care, you know, about the stability of Taiwan? So, how do you see all of that? I mean, I think we're both probably pretty uncertain, but you have a knack for cutting through the uncertainty and at least having a useful point of view.

• 2:02:53Zvi Mowshowitz?

  Yeah, I'm uncertain as well, obviously. And I haven't tried Ernie. I know that there's a long history of Chinese bots being claimed to be good and then either not being released at all, or turning out to not be anything, and nobody using them in practice. And also just, I would say, they'd be louder. Right, like sort of my prior is if in fact a Chinese company had a GPT-4 level model that was plausibly state of the art, why wouldn't like, you know, the Chinese government and the Chinese company be striving it from the rooftops for national pride and for, you know, they're a public company. You want to advertise that you're cool, you want to drive your stock price, you want to drive recruitment, you know, all the normal reasons, and they're just not saying anything. And they're not saying anything if they don't think their model would stand up that kind of scrutiny.

• 2:03:38Zvi Mowshowitz?

  I have to sue, right? And so it just, it makes me very skeptical that they've gotten that far. Your statement still makes it sound like they've gotten that far more than other Chinese models. But again, like, that's just not a very hype art right now. It's true with the chip situation. I think there's a reason why they kept trying to evade it. I think that we did less than it could add if we had thought ahead and been more robust faster. But I think we're definitely getting somewhere and, you know, yeah, they know how to do some seven nanometers, but I don't think they're in mass production the way they'd like to be. And it's just starting me that like, you know, it's still the relatively easy part in some sense to like, you know, it's hard, incredibly hard, but like it's still not as hard as what's to come.

• 2:04:19Zvi Mowshowitz?

  And I expect us to still very much be the lead in this and to have all the experts and for it to be like, this is a vital thing for us to keep in charge of, right? This is a vital thing for us to keep ahead of. And did it increase the chance that they will try something on Taiwan? A little. I don't think they're thinking about Taiwan mainly for like economic technological reasons. I think they're mostly thinking about Taiwan for national pride, prestige and regime, like core valuation reasons and cultural reasons, and they will act where I was to say either way. I also think that, yeah, the risk is like definitely highly non-zero, but definitely not that high right now. And that, you know,

• 2:05:02Zvi Mowshowitz?

  This mainly has the effect that it escalates tensions generally, not because it generally specifically lowers the value of keeping trade open. Keeping trade open with Taiwan and the US is just a huge thing. We're talking about like back to 10% GDP hits to both sides if they close down. So I don't think they need any more economic reason than that to not mess with this.

• 2:05:24Nathan Labenz?

  On the global scale out of chip production and sort of, is there any way to make that non contradictory or non feeding into the capability overhang? One interesting thing that has just come out in the last 24 hours, it might be, is this company Groq, G R O Q, which apparently has people involved, and this is so new that, excuse the superficial analysis, we're doing speed premium analysis here. One of the inventors of the TPU at Google is involved with this company, as I understand it. And they have put out now what they call the LPU, which is hardware that's optimized for,

• 2:06:08Nathan Labenz?

  I don't know if it's super specifically like transformer language models or whatever, but more optimized for the workload that we actually have as opposed to GPUs kind of coming from a different historical lineage. This was a more first principles approach for the current class of models. Upshot of it is insanely fast inference is being offered via their API, like 500 tokens a second on the mixtural model. And for like 25 or 20, I think it was 27 cents per million tokens. So pretty insanely fast, pretty insanely cheap. That's not a small model. Not the biggest model, obviously, but it's not insignificant.

• 2:06:54Nathan Labenz?

  And one thing that I did notice there, though, is they don't support training.

• 2:07:00Nathan Labenz?

  It is an inference only product as of now. Is that a fundamental limitation? I still have to get a couple of their papers and throw them into Gemini 1.5 and, you know, make, flex that context window before I'll have that all clear. You could start to squint at that and see some path to like massive inference build out. You know, is that a fork in the road? You think maybe we could figure out a way to scale inference? So everybody has their AI doctor, but not necessarily scale training infrastructure such that you still have relatively few.

• 2:07:36Zvi Mowshowitz?

  All I have seen is one, two second video of someone typing a prompt and then getting a very large response back that could have both very easily, both could easily have been faked and also told me nothing about the quality of the output or the cost of that output. So I, I'm operating off of this is completely new and I'm reacting in real time. If LPUs are now have become a thing that can do inference, you know, 10 times or a hundred times faster relative to their ability to do training, that's great news, actually, right? In the sense that like now we will be much more incentivized to do inference and not do training, but also if inference becomes much cheaper than the incentive to create

• 2:08:20Zvi Mowshowitz?

  larger models becomes much stronger. Because right now I get the sense that the actual barrier to creating larger models to a large extent is that they wouldn't be economical to serve. And so there's the danger that this ends up not actually being good. It could just be bad. So, I mean, I don't know. Like a lot of things do come to pass. It's very hard to tell which side any given thing will end on, but it's obviously very exciting to like have, you know, vastly faster, better inference for cheaper. It's just, you know, we have to think carefully about it. I don't want to speculate quite this fast until I know more duration.

• 2:08:58Nathan Labenz?

  You can't generate 500 tokens a second on this in the way that the Grok stack can with, with Mixtral. It is, I mean, I've tried it very, you know, limitedly. The models they're serving are just generic open source models. So it's like they're, you know, taking any responsibility for the performance other than just the pure speed, but it was damn fast. I can definitely testify to that. One other thing I wanted to just get your take on real quick is, I don't know if you saw this tweet not too long ago about the YOLO runs at OpenAI. So the basic concept there was, I think that there's a couple of angles on this I thought were interesting and I wanted to get your thoughts on.

• 2:09:36Nathan Labenz?

  So what is it? First of all, a YOLO run is a less systematic exploration of how to set up an architecture and what all the hyperparameters should be. And more of a shoot your shot kind of approach, like, okay, you're an expert in language model development, like, try something a little bit out of distribution, but with your best guesses about how something a bit different might work. And then let's give it a run and see how it actually works. The two big questions that come up for me there are, one, yeah, obviously from a safety standpoint, like, should we be worried about that? You know, this is like less systematic and kind of more, you know, shots in the sort of dark, you know, almost battleship approach.

• 2:10:18Nathan Labenz?

  This is like, instead of optimizing around the one hit you already got, this is like shooting off in the ocean somewhere and hoping you get another big hit. And, you know, is that concerning? Certainly some of the AI safety Twitter reaction suggested that this is very concerning. I didn't really know what I thought immediately. The other question is like, isn't this what we have scaling laws for? Like, why do we need YOLO runs? If we have scaling laws, like, can't we just try these things super small and, you know, isn't there supposed to be like, loss curve extrapolation that would mean we wouldn't need something like YOLO runs? Cause there's an implied scale to the YOLO run, right?

• 2:10:56Nathan Labenz?

  That if you could, if you didn't need scale to get your answer, you would be able to do a hyperparameter sweep like you normally would. So.

• 2:11:06Zvi Mowshowitz?

  Right. The whole, the whole point of a, so YOLO run is we want to put a lot of compute towards a big run. And normally what you do as, you know, what you do is doing science. You change one thing at a time and you see where that's going. And here's that I'll just change 20 things, see what happens. And then figure out which three things I got wrong and fix them if I have to or adjust it on the fly. But I'll just sort of operate and like the idea of I am going to suddenly change a bunch of things that I think will cause this thing to be more capable. And then just run a giant training run, having changed lots of things and seeing what happens. Definitely does not sound like the way to keep your run safe.

• 2:11:51Zvi Mowshowitz?

  Right? So if what you're doing is you're trying to train a model that costs a lot of compute, but it's still like nothing like the best models. Then a YOLO run mostly indicates that you're just better at this, right? You you think you can handle these changes and you can be more efficient at testing them. And so it's good, but like if you were YOLO running GPT-5, right? You were literally like, I'm going to train the next generation model, having changed 20 different things that I hadn't checked before on the previous level to confirm how they work. And that's scary as all hell because obviously if you think that's going to generate a lot of capabilities and do a lot of new things, what's going to have a lot of strange new behaviors and and affordances

• 2:12:34Zvi Mowshowitz?

  and just ways it acts that you haven't seen before because you changed a bunch of things. And you don't want to do that at the same time you're potentially introducing various new levels of intelligence. So it depends on how you present it, but like it certainly feels like the kind of thing with the culture really doesn't happen as much in places you'd like. And the way they were talking about it, it definitely made me more concerned, right? P D M up slightly to have them talk like this. But I've done YOLO runs of like met the gathering decks, right? Where I'm like, no, I think it's just going to work. And I'm like doing lots and lots of different things that no one's ever tried before. And I know how to like play a few games and then immediately understand, oh, they

• 2:13:15Zvi Mowshowitz?

  need three things I did work mistake. And then I could change that and so on. And, you know, mostly it's just a question of if you are much, much better at intuition and in terms of diagnosing what you see and figuring out what caused, caused by what, then you can do a better job. And if you can do a better job, then you can move faster and break more things. And the key is to do that when that's a good idea, but not when it's a bad idea.

• 2:13:37Nathan Labenz?

  It is remarkable that this, it's funny, like you have this sort of extreme secrecy on the one hand from an organization like OpenAI, and then you have like some person that I don't even think I had really known of before tweeting about YOLO runs. It's like, this is a, yeah, it's a very confusing situation.

• 2:13:56Zvi Mowshowitz?

  I mean, there wasn't zero amount of Leroy Jenkins involved in this, right? And we should all acknowledge that.

• 2:14:04Nathan Labenz?

  Anything else you want to talk about before we break? This has been very much in the Tyler Cowen spirit of the conversation I want to have. Any, any parts of the conversation you want to have that we didn't get to?

• 2:14:15Zvi Mowshowitz?

  I definitely want to have conversations with you that I wanted to have.

• 2:14:18Nathan Labenz?

  Cool. Well, this has been great. Thank you for YOLO running a fast response episode of the cognitive revolution. And officially, Zvi Moszczewicz, thank you for being part of the cognitive revolution.

• 2:14:29Zvi Mowshowitz?

  Absolutely. I'm glad to be one.

• 2:14:31Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.