OpenAI, Amazon's Anthropic Investment, and the Roman Empire with Zvi Mowshowitz

Zvi Mowshowitz, the writer behind Don't Worry About the Vase, returns to catch up with Nathan on everything OpenAI, Amazon-Anthropic collab, and Google Deepmind. They also discuss Perplexity, deepfakes, and software bundling vs the Roman Empire. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive Definitely also take a moment to subscribe to Zvi's blog Don't Worry About the Vase (https://thezvi.wordpress.com/) - Zvi is an information hyperprocessor who synthesizes vast amounts of new and ever-evolving information into extremely clear summaries that help educated people keep up with the latest news. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. TIMESTAMPS: 00:00:00 - Episode Preview (00:02:42) - Nathan's experience using Code Interpreter for a React app 00:06:09 - Zvi's perspective on Code Interpreter and other new Anthropic products (00:10:47) - Nathan's approach of "coding by analogy" using Code Interpreter (00:15:42) - Sponsors: Netsuite | Omneky (00:17:00 )- Performance degradation issues with large context windows (00:22:50) - The disconnect between Anthropic's value and what users are willing to pay (00:31:56) - Predicting Gemini's capabilities relative to GPT-4 00:30:13 - Rating Code Interpreter's capabilities (00:37:53) - Using Perplexity vs. Anthropic products for search (00:44:11) - Potential for a bundled subscription for multiple AI services (00:47:39) - Challenges of getting competitors to agree to bundling (00:54:05) - Concerns over information pollution from synthetic content (00:56:36) - Filtering adversarial vs. unintentional bogus information (01:02:20) - Dangers of info pollution visible in Archive dataset (01:11:15) - Kevin Fisher’s AI Souls demo with emotional voices (01:12:15) - Difficulty of detecting AI voices/images for a general audience (01:14:32) - Being optimistic about defending against deepfakes (01:21:12) - The reversal curse in language models (01:46:12) - Implications of Amazon investing in Anthropic (01:49:20) - Non-standard terms likely affected the Anthropic valuation (01:51:13) - Survey of the AI Safety landscape

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Zvi Mowshowitz?

  What it's worth and what people want to pay are just going to be completely different things. Almost no one, I predict, is going to be willing to pay more or even vaguely what it is worth. Everyone's going to be radically less, and I would include myself in that. What is the actual marginal value per month of GPT-4 over the alternatives? Probably four figures minimum. If you asked me this versus having nothing of the kind, it would be off the charts, right? Five figures or more. You know, am I willing to pay those kinds of prices? Men do not actually think about the Roman Empire that often than we think about bundling and unbundling like real men.

• 0:34Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Well, let's do it. Zvi Maschewicz, welcome back to The Cognitive Revolution.

• 1:00Zvi Mowshowitz?

  Good to be back. Always fun.

• 1:02Nathan Labenz?

  The lull of the kind of somewhat quiet AI summer is definitely over. Leaves are starting to change and products are launching in rapid succession. So naturally I reached out to you and get an early copy of your next 50-page blog post that runs it all down in exquisite detail for everybody. Excited to talk about some of the top stories here with you as well.

• 1:28Zvi Mowshowitz?

  Yeah, it's great to bounce stuff off of people who also are thinking about these things and see how they feel as well.

• 1:33Nathan Labenz?

  We're both obsessing about it full-time. So I think people have seen the headlines, you know, the big stuff that is out. And I could just kind of share a little bit of my experience with some of it over the last few days. OpenAI has certainly gone on a bit of a tear. They have put out, and I'll probably even miss something here, but just in the last like week or so. Dolly 3 is announced and starts to roll out initially to ChatGPT users, which I think is interesting. Code Interpreter, by the way, already incredible, gets also the enhancement of the image understanding. So they now bring image understanding also to ChatGPT. And Code Interpreter benefits from that, too.

• 2:16Nathan Labenz?

  So we're starting to see all these interesting demos of people being like, here's a whiteboard snapshot, and I'm getting code, you know, directly from a whiteboard snapshot. Pretty amazing. They've also got voice now enabled as kind of a first-class citizen within the ChatGPT app. And that's gradually, all this stuff is kind of gradually rolling out over a couple of weeks. So that's quite a flurry of things. For me, so far, the newest best use has been using Code Interpreter. Just this last week, I had a React app that I wanted to add a feature to, and I'd literally never coded in React before. But going to Code Interpreter, I have the new image understanding turned on.

• 3:00Nathan Labenz?

  And, I mean, it was an unbelievably awesome experience. The first question I asked was just, hey, I inherited this app. I don't, I've never coded in this framework. Can you explain it to me? That, so it did that. Then there were like a couple of different patterns that were present in my project that weren't present in its initial explanations. I kind of said, hey, I've got this slice JS file and Saga JS file and different folders. And of course, you know, it's like, oh, yeah, well, that means these libraries are enabled, and that's what these things do. And this, you know, it's kind of a extension of the core React framework. Okay, cool. So then I was like, can you help me write a command to just print out my file structure? Then I'll give that to you, and you can help me navigate further. Yep, okay, here's the command, you know, okay, here's your, here's the file structure back. Okay, great.

• 3:44Nathan Labenz?

  It looks like you've got all these different components and everything's, you know, probably working together. And, you know, the main file that coordinates them is this. Okay, sweet. Now I want to add a feature. So, you know, it takes me all through that. At some point, like it's, um, it's starting to work, but it's not really working. I've got like information missing. And so I didn't even really have to do this, but taking screenshots, dropping the screenshots, saying, this is what I'm seeing, you know, can you help me? Oh yeah, it looks like the information's not there. It might be because it's not being parsed, you know, effectively. As it comes back from the language model, whatever. Of course, I'm adding an AI feature to the app, right? So, you know, it's all AI on top of AI. Honestly, it does feel like there has been a pretty significant step change here

• 4:26Nathan Labenz?

  in the overall value, certainly of ChatGPT with all these new enhancements. And did I even mention that they have browsing now, uh, re-enabled in the last, uh, couple of days too? I think I even skipped that one. I knew I was gonna miss something.

• 4:40Zvi Mowshowitz?

  It can do, yeah, sight in and out, hearing in and out, and the web in and out. And it's just a complete sea change. One of the top things that you didn't mention was the idea of designing UI or what the page should look like. I give it to you and it outputs code, just like, here, it'll look like that now.

• 4:58Nathan Labenz?

  It's pretty wild. Have you been using it personally or?

• 5:00Zvi Mowshowitz?

  The biggest effect for me is it's tempting me to start trying to code it. It's tempting me to create. Because it just feels like it's so much easier. Like all of these things are things that you could have in theory found a way to do, like with more time. But now it's potentially in order of magnitude easier, right? You don't need to see, have the thing see the whiteboard and directly output the code. But just the workflow, when you can do that, it is so much smoother and things are so much faster. You can iterate so much better. And that's especially true because we've seen time and time again is the worse you are at the task, the more ChatGPT and other similar tools help you get better, right? So like, the better you are at programming, the more you are like, I'm just gonna do this myself.

• 5:45Zvi Mowshowitz?

  So I have a friend who's an expert programmer, programming for decades. And he said, well, I can barely get any use out of this. You know, it's, I'm doing all these specialized things that no one else else has to do. It has no clue, you know, it just causes so many errors. I might as well just code it myself. And for him, they did that too, but if I tried to code anything, right, it would be completely night and day. And so I'm very tempted, the problem being that I'm tempted by so many other things that I end up making other choices. I didn't make one attempt to use DALL-E to create images this past week and like, figured out, no, that's not actually a use case here. I was sort of hoping it could do modifications of images that pick your case.

• 6:24Zvi Mowshowitz?

  And it can't. It instead interprets what the original graph looked like and then it says, okay, now create something that has the characteristics that I found described in that graph when I asked what I thought was in it. And then this great and very stylized thing is completely different from the original. I was disappointed, but the thing to create looked really cool. I just didn't have a use for it.

• 6:48Nathan Labenz?

  So first of all, for your friend, I would maybe challenge their assumptions in as much as you look at like an Andre Karpathy on, you know, just, just the stuff that he's posting about on Twitter where he's writing like, C to execute, you know, open source language models on a CPU. That is to say C code, right? Super low level, super optimized code. And, you know, obviously he's a super capable individual with a lot of very specialized knowledge, but still, you know, it allows him to do this kind of thing in a weekend without necessarily even having to like brush up on C all that much because, you know, it, it does know the fundamentals. And it also just, for me, it just, it's so refreshing

• 7:33Nathan Labenz?

  to, it's, it feels like the way coding, you know, you kind of always imagined it would code if you were actually like reliable, you know, if we were robust in our, you know, execution, because it's like, it doesn't make these typos. It doesn't do just like really stupid type errors or whatever, kind of, you know, annoying things that I often do. And then I'm like, why is this going wrong? And ugh. And then you feel so stupid, you know, everybody who spent any time developing has that moment of like, oh, god damn it. You know, this was so stupid and I just got so stuck on it for a long time in some cases, right? I mean, we've all had that experience. And these days I almost. I don't want to jinx myself, but I very rarely have that experience because it doesn't make those mistakes and it kind of knows, you know, a lot of these common stumbling points.

• 8:19Nathan Labenz?

  I even had one recently where my container, my Docker container was somehow messed up and like couldn't get an update and, you know, some key file thing was out of date or couldn't be synced and it was just like an absolute nightmare. You know, this is the kind of thing that to Google is hell and to just deal with is like, I don't, I don't want to deal with anything like this. I never want to think about this, you know? So in that case, it was actually perplexity that solved it for me and just gave me like the exact commands to write. I mean, it was, it was to the point where it was almost like, dude, you know, are you going to just execute this? Like, I don't even really know what I'm doing. Uh, but it was all in sort of a container in the, you know, GitHub, uh, code space environment. So I was like, well,

• 9:01Nathan Labenz?

  worst thing, I could just kind of throw it away and, and start over from, uh, you know, a backup.

• 9:07Zvi Mowshowitz?

  I was surprised you in both directions, right? The last time that I actually did try to code something, it was consistently just getting the syntax and libraries wrong and just like, maybe they changed since this cutoff date, maybe something else was going on. But it was just giving me nonsense that obviously in some cases, like it just didn't work in practice. In some cases I was like, I can obviously spot this. And I'm bad at coding. I just instantly see, oh, you'd fill up a coding interview if I saw you do that. Cause like, it's just obviously will never work. And I went back and forth and I was able to like get it past certain things that eventually create something, but it was incredibly frustrating. And definitely didn't sidestep these kinds of experiences. And then other times it's just like, whoa, that just worked and that's amazing.

• 9:48Zvi Mowshowitz?

  And that's very similar to other programming experiences, right? Well, there'll be days where you just code something that seems incredibly complex and it basically just works and everything you thought does exactly what you expected. And other days, the simplest things just make you want to tear your hair out.

• 10:00Nathan Labenz?

  That's becoming much, much less for me. I would say it is safely a multiple X. You know, like I've been saying, kind of three to five X speed up. Honestly, with this reactive from this week. 10x would not be crazy because, you know, to do anything productive in a totally new framework where you're totally disoriented is hard. I zoomed past those parts.

• 10:22Zvi Mowshowitz?

  To be clear, like the thing I was trying to do, like it was incredibly frustrating, but also like I would never have even bothered trying, right? If I didn't have this tool and it would have taken me 10 times as long, absolutely. So, you know, I again, I'm tempted to go. I am tempted to try and set things up and, you know, I'm inspired. So maybe this weekend, maybe sometime relatively soon, I'll start trying to create a few tools, uh, things that are useful to me. Right? Like this is partly to learn through the experiments of trying and partly because I like, we want to have them.

• 10:49Nathan Labenz?

  One tip that I wonder if, you know, it will work for you and you can maybe help me refine it is I call it coding by analogy and I maybe should get a better name, but I always try to bring some snippet of working code for the, the reason you mentioned it is particularly tough, right? The, the libraries have changed. So very often you'll get now with browsing, you know, may go out and fetch an up to date version two of the documentation, but prior to three days ago, whatever, it would often have this kind of outdated library knowledge. What I find to be super effective and definitely recommend you playing around with is like, either just grab something off the website or out of your current project, if you have a current project and be

• 11:32Nathan Labenz?

  like, here is something that works. I want to do something different. Here's what I want to do and let it kind of map the working example onto, better to say, map your need onto the working example. That seems to really work well for me as opposed to going, you know, totally cold. So give that a shot. Let me know how it goes.

• 11:53Zvi Mowshowitz?

  Yeah, I'll see what I can do. I just a case of like, I'm so busy by marveling at all the things we can do. I don't have a chance to do that many of them, which is, I guess, my curse.

• 12:01Nathan Labenz?

  I want to hear your kind of higher level perspective on all these releases, you know, with that context of just like, yes, the, the utility function is a step up. It seems like, you know, OpenAI has again, if you're going to buy one AI thing, they've made it pretty clear right now, theirs is the one to buy. But I'm interested in kind of how you understand the timing of these releases and like, what do you see as the dynamics that are kind of in the background, uh, that may not be so apparent to most people?

• 12:28Zvi Mowshowitz?

  Well, so we know that OpenAI has You know, multiple times sat on reasonably large capability advances of various types. You know, they sat on G P T four for eight months. You know, chat G P T was rushed out pretty quickly when the underlying technology had been there for a while. You know, the multimodal stuff they're rushing out now. It's built straight off of G P T. It's not new. They've had similar things for a while. They clearly could have done it earlier. They chose not to. One of the reasons for that, I'm sure, is they're worried about potential adversarial attacks, especially on the vision. And I have no idea what they plan to do about that. I haven't seen reports of I tried to adversarial attack G P T four with images that were designed for that. And then they came back.

• 13:09Zvi Mowshowitz?

  Maybe it takes a few days to try it, but very curious what what their defenses are or why it's not working, if it's not working. Or were they just going to eat it if it does work? I don't know. But I think there are two huge pressures, right? Applying on OpenAI right now. Maybe three. So you got Claude two, right? Which is there's now a competitive model that has the advantage of a giant window. Can read PDFs natively and answers your questions pretty well. And in my people find it very friendly to work with. And it's free. So you've got to compete with that. You've got to compete with Llama two, right? Which is open sourced. And I don't think it's that good. But it's still like something people can build off of. And you want to make sure people aren't building off of that.

• 13:53Zvi Mowshowitz?

  They want to make sure they're building off of you. And then third of all, the inspector of Gemini, right? So it's already almost October. People expect by the end of the year, probably we're going to see Google release completely different, natively multimodal, natively involving Alpha Zero style project potentially inside it. Like they claim it's better. Like some rumors say it is clearly better. I don't know what better necessarily means. I have prediction markets up on this and it can go either way. But if they're no longer going to have the best model by the end of the year and the underlying core, they got to move fast to add features, right? Lock in their users before it's too late.

• 14:31Nathan Labenz?

  What do you think is gonna happen on that model question? I mean, I guess for for starters, I love Claude two. And I subscribe to, you know, pay for lots of things. So I'm not one who's just going to buy one product. But Claude two, especially for the long context window and also for kind of imitating writing style, does seem to be preferred. For taking, you know, the, just the transcript of this podcast, for example, and converting that into timestamp, uh, you know, just outline for a show notes. It is really the only one that can do it, you know, at least without having to kind of chunk it into lots of different parts and, you know, whatever, which you could do, but it certainly

• 15:14Nathan Labenz?

  becomes a lot less convenient. I've noticed it does struggle with the full window stuff. Like if we do three hours today, which, you know, we promised each other we won't, you know, it actually kind of goes off the rails. And even though it does all fit into the context window, it kind of still can't handle the problem. But if I cut it in half down to about hour and change up to like 90 minutes chunks, then it can handle that. But even that is just like still too long.

• 15:37Zvi Mowshowitz?

  Hey, we'll continue our interview in a moment after a word from our sponsors. There was a study this week, actually, that showed like, what's the degradation as the context window gets bigger? And if you're near the, the later part of the context window, right? All that matters is how many tokens have been taken place since the thing that you want to access, the thing that you need to have and understand. And the more tokens there are in between, beginning and the end, you know, where you are and where you want to finish, the worse it's going to be at recalling information, the worse it's going to be incorporating that. And so you lose a substantial amount, yeah, when you go more than half of the size of the context window. It's 100K is clearly like, this is pushing, right? Like you want to stick to 50 if you can.

• 16:17Nathan Labenz?

  Yeah, that's definitely what I found. And interestingly, it just goes totally off the rails for me at 100. I mean, maybe it's just the nature of my task, because kind of once it's like out of sync with the actual transcript, it's just lost. But it, it's not like it just misses a couple of things. It's like, it gives me a really good thing up to that kind of half point. And then at the, near the whole thing, it's just like way.

• 16:40Zvi Mowshowitz?

  Yeah, I, I've never pushed it quite fully. I've always had more problems with the size of the file I'm uploading than I have with the number of tokens that resulted from the file. Mostly I'm trying to read papers rather than, than doing something that you're doing when I'm using the full context window. And so I do think there's a few cases where Claude 2 is still like clearly superior. And I still think a substantial percentage of my queries, I will use Claude 2. But I felt like, you know, two weeks ago, it would have been very legitimate to say, I don't have to pay for my large language models. I can just use the free Claude 2. And that's good enough for me. And one time I've hit like bandwidth limits and it's like you can't use this and okay, I'll use GPT three and a half or Bing or whatever for the next hour and it'll come back.

• 17:24Zvi Mowshowitz?

  Now with these new features, I think that's just completely not true. I think that anyone who's trying to be productive in, you know, any kind of knowledge work is going to find more than $20 reviews out of these new features and you pretty much just have to pay.

• 17:37Nathan Labenz?

  Couple of questions on the value and your expectations. And I agree, by the way, Llama two, I'm not like using it much. Definitely lots of things will be built on it, but you know, that's all kind of still in the offing. And again, that's why, you know, presumably related to at least why OpenAI has the 3.5 fine tuning online, you know, in the kind of short wake of a llama two coming out. But pricing, okay, so it's 20 bucks a month. The enterprise price they've announced, well, I don't know if they've actually announced it, but it's like known to be $60 a month. Interestingly, they have a kind of contact us on the, on the website for enterprise customers. What do you think it's worth?

• 18:22Nathan Labenz?

  Obviously it varies by context, but you know, if they could price discriminate and actually charge you what it's worth, what do you think it would be worth to you? What do you think it's worth to enterprise customers? You know, who they're trying to get 60 bucks from? Again, I know it's going to be kind of a distribution, but how do you think about like what, what the actual value is in today's economy?

• 18:43Zvi Mowshowitz?

  This is one of those cases where the, what it's worth and what people want to pay are just going to be completely different things, right? And, and there are almost no one, I predict, is going to be willing to pay more or even vaguely what it is worth. Everyone's going to be radically less and I would include myself in that, right? Like I, I think that if I ask myself, you know, given what I am doing and what I'm trying to do, what is the actual marginal value per month of GPT four over the alternatives?

• 19:13Zvi Mowshowitz?

  Probably four figures minimum, might be, I mean, might even be bigger. Claude 2 does perform a reasonable substitute for a lot of things, so it's probably not that big just because the alternatives are pretty good. If you ask me this versus having nothing of the kind, it would be off the charts, right? Five figures and more. You know, am I willing to pay those kinds of prices? I'd almost certainly recoil in horror, and I'm pretty good at not recoiling in horror from such things. We see the whole kerbuffle with Twitter, where a lot of people are like, I spend two hours a day on this app, and he wants to charge me $8? How dare this man? Right, like, so out of touch, it's so crazy, I'm just gonna leave, because everyone else is gonna leave too, right?

• 19:55Zvi Mowshowitz?

  Why would they spend two hours a day on this site and then pay $8? Oh my god, the horror, the horror. I, I understand it, I have the instinct too. You know, it's what LeBron James calls paying the five, right? Like, why isn't LeBron James paying the five? He's got, you know, hundreds of millions of dollars, but, you know, it's a principle. So, you know, at $20, I think it's a joke, right? At $60, an enterprise is even bigger of a joke. If your enterprise doesn't pay $60 a month, like, why do you even deserve to have an enterprise? It's kind of crazy. And I think enterprises should happily pay thousands of dollars, mostly per month, if not more, depending on the size of the enterprise.

• 20:29Nathan Labenz?

  Thousands per month per employee, to be clear.

• 20:32Zvi Mowshowitz?

  Oh, per employee?

• 20:34Nathan Labenz?

  Because the 60 is per seat.

• 20:37Zvi Mowshowitz?

  Right, so like, the obvious question then is like, why aren't the employees just subscribing separately? If, and paying 20, if it's per seat. But yeah, 60 is very reasonable, especially if it's unlimited, or virtually unlimited, like, you know, up to a reasonable API limit, like, you just get the two nuts. And it offers some sort of privacy and security, then, yeah, I think that's very, it's very reasonable. I think that's probably about the right pricing, though, in terms of what companies are gonna be willing to stomach, right, at first? And then if you add more features, you can, you can jack up the price. If you add more optional features, you can really jack up the price. But you often see this, like, with drugs, right, where... People say it's all outrageous, but if you offer someone a slightly better version of the drug, it's actually better for someone's life experience. Then you ask, but was it actually worth to you?

• 21:23Zvi Mowshowitz?

  For real, and the answer is, well, it's actually worth thousands of extra dollars to me, for tens of thousands of extra dollars to me. To have my life be somewhat more convenient not to worry about this little thing. And then people pay.

• 21:32Nathan Labenz?

  I think the key point there is the disconnect between the value and the price is is pretty extreme. I honestly would pay into four figures. I would have to like, you know, be maybe a little bit more commercially oriented with my time at some point if they really were going to start to push the price to the limit. But I remember thinking even just when the original co-pilot was available at 10 bucks a month, you know, for me to buy as an individual from GitHub. I was like, well, actually, it was initially free. And so I was speculating about what the price was going to be. And it ended up being 10 bucks or even $19 enterprise price point. But I was thinking, what could they charge for this? Where I wouldn't pay it. And even then I was like, honestly, I think if it was a thousand bucks a month,

• 22:16Nathan Labenz?

  I would probably end up paying it because it just makes me so much faster.

• 22:22Zvi Mowshowitz?

  Again, it's all about what is your alternative? Right? Like if you tell me I can still use Bing for free, I can use Claude 2 for free, I can use Proplexity for free. I can use Bard for free, which doesn't matter very much right now, but maybe it will matter in December or January, et cetera, et cetera. Now, if you try to charge me an arm and a leg, I have alternatives, right? It's not going to be that bad. If I didn't specifically have any need to be able to assess what GPT-4 could do because of my job specifically requiring that. I would say, well, how much better is it? But you think goes away. You start offering the modes of operation where these other systems don't work. And yeah, now you can hit me up for gigantic amounts of money and I kind of have to pay you.

• 23:05Nathan Labenz?

  Yeah, it's a great, another great point. The the BATNA is super relevant here.

• 23:12Nathan Labenz?

  And one point, just on the enterprise thing that is pretty useful, I think, and will drive, I think, people to kind of just be like, okay, yeah, we're gonna just buy this as a group, is integration into the knowledge systems of the enterprise, whether that's, you know, a Google Drive, or Dropbox, or, you know, Box.com. That's another major thing that is, as I understand it, is live for ChatGPT enterprise customers. That is pretty new, not just in this most recent, you know, intensive wave of launches, but like, man, they've built out a pretty robust suite of things over the last six months. It has been pretty impressive.

• 23:52Zvi Mowshowitz?

  To be perfectly clear, my enterprise of one would rather pay $60 a month and have Golem and Gmail integration so that my version of GPT-4 knows my context, than pay $20 for what I have now. I would happily, happily flip the 60. And also probably happily pay $600, like, just for that one feature, right? Because that's the marginal value of that is so huge. I've, have you noticed also that you are thinking about your emails differently than you were a couple of years ago? Like, I used to be like, okay, the moment my emails are done, I want to delete them so that when I search for my email, I don't have to clutter, like, that I accidentally have to filter through.

• 24:35Zvi Mowshowitz?

  I want it to be clean. I don't want to, I don't want to be distracted. But now, I'm like, is this information that the, I know it, right? And I'm not going to be able to productively dig through to find this information. But maybe I should file it away for a large language model, right? Because like, when they're going to want this context, that's going to help them understand.

• 24:56Nathan Labenz?

  Yeah, it's funny. We're, yes is the one-word answer to, I have been thinking about my emails differently, but we're coming at it from very opposite ends of the spectrum, where I have just allowed all manner of crap to collect in my inbox. And so when Bard announced their extensions to tie into your stuff, I went and tried that. And my takeaway from it was the, I mean, for one, just still kind of behind on the model. I'm gonna get back to Gemini in a second. You know, just making some of these mistakes. It's like, what are you gonna come on? Seems like we should be able to do this by now. But then also in actually searching my Gmail, there is so much shit in there that I mostly just kind of don't even care, you know,

• 25:39Nathan Labenz?

  haven't even cared to filter out. But I was kind of like, I think I might have to do a purge of like 90% of the emails that I've just allowed to accumulate, which are mostly just spam and, you know, or marketing lists or whatever that I've signed up for.

• 25:53Zvi Mowshowitz?

  Thinking about the cloud issue, right? Like you had this giant context window. And when you looked at the first half, the old half of the context window, when there was too much stuff in the way, it just lost the thread. So if you have too much stuff, right? Like unless it's doing something much more sophisticated and it's doing active fine tuning or something on your data, you're probably going to lose performance if there's too much distracting data. So yeah, you want to get rid of it. I learned this when I was working at James Street Capital, right? Because not because like they're so great at anything, but because they are subject to these regulations where you literally can't delete an email. Right? If you work at a financial firm, every day you get loads and loads of spam or almost spam from various different companies explaining what's in their fund

• 26:33Zvi Mowshowitz?

  or whatever little piece of technical disclosure you get. And all you can do is archive it, right? You can search in it, you can put it in folders. Nothing can ever be deleted. And so if you ever try to search for anything, good effing luck.

• 26:50Nathan Labenz?

  Yeah, even just for, you know, maybe I'm paying a monthly subscription. I don't even care about the cost, but even honestly, just for the latency and definitely for the accuracy at this point, I do kind of have a new item on my to do list, which is like, figure out some way to go delete the 90% of threats that I never even responded to, hopefully without deleting things I do care about. But man, there's just, you know, I need to, I need to like get the search results that matter onto the first couple of pages, or it's just never going to page through them. I think Gmail also should be able to do more there with metadata or, you know, some sort of like heuristics around.

• 27:26Nathan Labenz?

  Okay, look, if this dude did not respond to the, you know, to any of the things that are coming up in search, like they may not be relevant, but so far they're not there on that. So we'll see if that, I'm sure they'll continue, of course, to refine it.

• 27:38Zvi Mowshowitz?

  It's the kind of thing where us geeks can can think of a number of things that would improve our experience, but it doesn't move the bottom line very much, and they're never gonna actually invest the effort. Like, some engineer is not gonna like suddenly fix it one day.

• 27:52Nathan Labenz?

  Things that could maybe, you know, fix the AI product suite at Google, you know, probably headlined by Gemini. What's kind of your best guess right now as to how powerful this thing will be? Do you think it is gonna, you know, kind of become the best model? And I think underlying this is kind of a question of like, you know, is scale all you need, you know, or is there still just kind of a a big moat, so to speak, that Google still has to get over, independently of just like how big and bad they train the model, to actually make it kind of usefully productized in the way that definitely?

• 28:31Zvi Mowshowitz?

  What's the first day in which you would have felt like you could say that sentence, right, that Google has a moat to overcome in AI, as opposed to the reverse, where everyone's trying to overcome Google's moat? It's so stunning, right? I I'm a firm believer in scale is not all you need, in the sense that the expertise from an OpenAI is incredibly valuable. Look at the large language models that have been trained in open source. You see a very, very clear pattern where if you have a large model that's trained on lots of data, the people involved, like, you know, Falcon-style, do not know what they are doing, right? Do not have the pure expertise. You decide, let's do a huge model. You might not hit some benchmarks, but in practice, the thing is useless, right? It it's just nobody will build on it, nobody will

• 29:15Zvi Mowshowitz?

  use it, it's just not very good. And also, the more you scale, the more expensive it is to run the thing. So you can't just cheaply scale at an enterprise level. Google clearly has some advantages. DeepMind has expertise in various AI forms that they can integrate in that nobody else has. And so the question we're gonna find out, I think, is about, Google has the scale, Google has the data, Google has the compute, Google has certain forms of expertise, but is Google still capable?

• 29:46Zvi Mowshowitz?

  Of doing what they have to do to make this model where it needs to be? Or is Google just hopelessly broken? But we already know that Google has, you know, this weird dynamic of like this inner competition where like it's got these dozens of projects that are effectively competing with each other, trying to do variants of the same thing, or even if it's sometimes the same thing. And these teams, you know, don't communicate, don't work together as well as one would expect in other places. And sometimes that's good, but it creates competition. And in other ways, it means that like, it's weird because like, you know, of the top 20 AI labs, how many have been for Google? Is it 13? It's not impossible. That's true. But like, can DeepMind deliver

• 30:26Zvi Mowshowitz?

  on this task, which is the only task that really, really matters? And then the question of, so DeepMind creates this Gemini thing. And then it's up to everybody in all of these different products, right? To take this Gemini thing and make it work for them to do a thing. And then is it what they need? Have these people communicated their needs? Have they lined up what's going on? You know, these things are kind of fickle a lot of the time. And you can't just say, oh, this thing is great. I'll just see what happens. And so I don't think any of this is obvious. But I think if I had to set an over/under line, like for gambling, right?

• 31:05Zvi Mowshowitz?

  I might set the line for Gemini at 4.25, GPTs, right? Like so somewhat better than GPT-4. But like not earth-shatteringly better than GPT-4. So I think half the time it'll be better than that, half the time it'll be worse, right? If I just had to guess. And if you told me the line was four and a half, I would believe you. If you told me the line was four, I would believe you. But also, we shouldn't be confident coming out by the end of the year. Like, it might not be ready.

• 31:31Nathan Labenz?

  Are you giving the code interpreter, AKA advanced data analysis, 4.5 on that scale? I mean, that's been kind of the the talk that, you know, some are saying like, oh, this should count as 4.5 because it's so much more useful than GPT-4. So just dialing in on your calibration.

• 31:49Zvi Mowshowitz?

  It's an application. It's a scaffolding. It's a iteration. And it's very much more useful for certain specific purposes. And I realize that, you know, data analysis and coding are important subsets. But to me, that's just a revolution of, you know, these models can do so much more. Then you think they can do, right? And so like when GPT-4 came out, they have a system card, they have the ARK evaluation, they have the, it self replicate, no, can't, blah, blah. Now we're finding out, what are the things we didn't know it could do? Then it turns out it can do. And part of that is defining the scale, right? Like, so like you could say four is four as it existed on the, at the moment of release and with the abilities at the point of release.

• 32:30Zvi Mowshowitz?

  But I instead think of it as the four core model and everything we've learned how to do with the four core model. Right? And I think the code interpreter has built on top of the four core model. If you gave me a 4.5 model to build on top of it, you would see code interpreter just like take it to the next level.

• 32:45Nathan Labenz?

  There are some differences in behavior. It's got the runtime as kind of the main sort of scaffolding difference. But I have seen some really interesting behavior from it where, you know, it will run code and then it just kind of continues in its own thing. You can really just like, give it a file, say, you know, figure out how to do whatever. I think we'll just jam on that and kind of hit errors, you know, rewrite the code, try again, hit a different error, you know, try again. Uh, get empty data out, you know, be like, wait a second, why is this happening? Print a couple, you know, records out of your data set or whatever to kind of examine them. And, you know, it's both, but it does feel like there is a little bit of a model

• 33:29Nathan Labenz?

  difference there where it's kind of, they've trained it more on this like iterative problem solving, kind of following up on, you know, responding to the kind of feedback that it's getting from the world in a way that, I mean, I guess the main chat models do that too, but you're the, you're the, you know, the source of the feedback, but it does feel a little bit different. It certainly has a more autonomous vibe to it that I do think is pretty interesting.

• 33:54Zvi Mowshowitz?

  Right? And that's the dilemma, right? It's always the dilemma of if we give this model to the world, what will the world be able to build on it? And how hard will that be? And so it turns out you can absolutely take GPT-4 and you can make it into a pretty good agent for this kind of purpose. Like general purpose, it's still falling flat, but for these purposes, yeah, you can do all these iterations and it required maybe some fine tuning, required maybe some additional training of various, various source design to do that, but I guess it's very little. My guess is it was like. Very low costs to do that. It was more cost was conceptual. The cost was figuring out how to do it. And now it can do it.

• 34:38Zvi Mowshowitz?

  So that's kind of scary because what's going to happen next?

• 34:44Nathan Labenz?

  For Gemini, what do you think are the kind of deltas in capability or like, you know, what is an extra quarter GPT mean in terms of mundane utility if you had to guess?

• 34:55Zvi Mowshowitz?

  Like from the types of things they're applying to incorporate into it. Uh, maybe we'll see some more strategicness. We'll see some more like ability to sort of understand what questions are about, like what natters in the situation, like responding more relevantly. Mostly, I think the key thing is just I expect to see like more of the the Raji, the the raw intelligence thing that's like. The thing that GPT four has as its advantage over the other systems where it can just figure more things out. Also, Google says they've solved the update problem. I don't know if you've seen that, but they claim, or which I've seen claims that. Google has figured out how to continuously train the model with the new information.

• 35:40Zvi Mowshowitz?

  Such that, you know, it will natively always have its cutoff be this week. Right? Or or very recent. And that's a sea change, or the reasons I don't use GPT four a lot more is because there's this increasing gap. Right? Between what it knows and what the cutoff is. And being able to use Bing, not the same thing, right? Even if it was really good at using it. Right? Like now, what's going on is I I was supposed to use my Google flu to my alternative to Google and saying you have better food than I do? Which is very, very different from saying I can just use your interface to learn things directly. Right? Does it have better food? And like Bing, yeah, I was really excited, but then I quickly realized that most of the time

• 36:22Zvi Mowshowitz?

  if I can't Google it, they can't Google it either.

• 36:26Nathan Labenz?

  I have not had, honestly, great experiences with Bing. And to be fair to the current Bing, I haven't used it as much lately. Perplexity has been my go-to for search and it genuinely has, at least for, I now, I guess I now segment searches kind of unconsciously into two types. One is like the quick lookup where it's, you know, more kind of, I need the pointer. Cause Google certainly is still fast and good at that. If I am confident that I can surface the thing that I want. But if I'm really looking for answers to questions I don't know the answers to, perplexity is now the thing that I go to as a first choice.

• 37:08Zvi Mowshowitz?

  Is that only if it's like, you need the answer to be updated versus like old, right? Cause if it's, if the questions could be answered by GPT-4, it's going to be in its data set. I'm going to go to GPT-4 every time.

• 37:19Nathan Labenz?

  Yeah, I want the sources in some cases. That's a big draw for perplexity. And they are using GPT-4. I mean, they're using multiple models and it's not entirely clear, you know, when it's GPT-4 versus something else. They had a pretty interesting claim also that they had achieved, I think they said equal, maybe even somewhat better performance, fine tuning 3.5 compared to four for their task, which is pretty remarkable because I've done that a bit over the last couple of weeks. And I have had a lot of success with it in those recent episodes, so I won't cover all that ground. But 3.5 fine tuning is a good experience. It's pretty fast. It's easy and it works well. And my one insight there, just to repeat, because I think it is pretty useful for folks.

• 38:04Nathan Labenz?

  Train on GPT-4 reasoning, not just output. I had some tasks where GPT-4 could just do the task even without needing to really explain its reasoning. It just did a good job. But then if I took that output and trained 3.5 on it, it wasn't kind of measuring up. But then when I said, okay, GPT-4 first explained your reasoning, then do the task and then train 3.5 on that with the reasoning. Now we get good quality reasoning and good quality output from 3.5 fine tuned. So, but my task is like super narrow. This is way more like script writing, you know, very dialed in. Perplexity is a lot broader. So I was kind of struck to see if they have this claim. Anyway, we don't know always what model they're using, but I think

• 38:48Nathan Labenz?

  the up-to-date-ness matters for sure, but also just the sources. I do want the sources.

• 38:54Zvi Mowshowitz?

  That's entirely fair. And I think, I think a lot of this comes down to the question of, are, are people doing a lot of the same things over and over again? That actually don't necessarily require that much like sheer intelligence at the core. Are you asking questions that you just don't know the answer to these questions? You can ask questions and there's no reason 3.5 can't have the answers. And if it's simple stuff like that, then you can use a dumber thing. That's like more instructed on the task. And it could well be better. And then, so like when I say, when I say three and a half is scoring higher than four. To me, that's like a statement about perplexity users and what they want and what they typically do. And I think that matches my experience, right?

• 39:38Zvi Mowshowitz?

  If I'm asking perplexity a question. It's mostly going to be one sentence. It's gonna be pretty simple. And if I'm asking GPT four or something, it's often going to be like one or two paragraphs of text. Like this giant thing that I'm trying, and that's often going to have a lot of back and forth involved in it. These are very different modes where you can imagine like no amount of instructing GPT three and a half is going to let it do what I'm asking for. Better than four. But for perplexity question specifically, yeah, it's not that crazy.

• 40:06Nathan Labenz?

  Interesting. Another thing I've been kind of speculating OpenAI might want at their, at their upcoming developer day is a mixture of models that would sit behind one API. And this would drive some people crazy because there's already so much kind of speculation about are they changing the models underneath us? Is GPT four getting worse, whatever? But I kind of think actually the product direction that they should go is the other way. Where they'd say like, hey, for, you know, the price of whatever, something in between 3.5 and four. We will route your query to the right model for your query. And now you don't even really have to worry about it anymore. We'll just give you kind of the right level of AI for where you're at.

• 40:49Zvi Mowshowitz?

  And that requires obviously like the efficiency of the evaluation of where to send the model. And the accuracy to be high enough combined that it makes sense to do that. And the speed also of the evaluation. Right? Why aren't you giving me GPT four? I can tell you gave me three and a half. And sometimes I'll even be wrong about that.

• 41:06Nathan Labenz?

  By no means do I think they're going to get rid of the, you know, specified model endpoint. But I could see something in between being super nice.

• 41:13Zvi Mowshowitz?

  Party services will more and more do exactly that, right? You will have a query designed, maybe you even call three and a half to ask the question, right? Like.

• 41:20Nathan Labenz?

  Are you up to this 3.5?

• 41:22Zvi Mowshowitz?

  You will have a two version of three and a half and the instructor, the assistant instructors will say something maybe like if GPT four is likely to give a much better answer to this question.

• 41:31Zvi Mowshowitz?

  Respond only with call GPT four and nothing else. Otherwise, answer the question. And sometimes it's with that call GPT four, you call GPT four. And because the ratio of the cost is so large, it doesn't really matter if you call GPT three and a half first with two tokens output.

• 41:46Nathan Labenz?

  Okay, here's another idea I want to throw at you, right? We've got this kind of race dynamic, you know, arguably heating up. Although I'd say there is still some restraint, you know, from the lead players like certainly as we mentioned, OpenAI has sat on a lot of this stuff longer than they had to. But we have all these kind of subscriptions now that are, you know, popping up. And it's okay, well, we got the 20 bucks for ChatGPT and Clog Pro is also 20 bucks and Perplexity Pro is 20 bucks. And Windows Copilot is 30 bucks and ChatGPT Enterprise is 60 bucks and Google Duet AI is 30 bucks. And GitHub Copilot is 10 if you buy yourself a 19 Enterprise.

• 42:30Nathan Labenz?

  And then Replit Ghostwriter, which I also think is really good and also uses GPT four and is so natively inbuilt. That's 10 bucks. And, uh, you know, we haven't even got to image stuff yet or any of the like apps. So it's kind of getting ridiculous, you know, a lot of barnacles on the credit card. And this got me thinking, why wouldn't there be a bundle of AI services? Kind of along the lines of like a cable bundle where you might say, okay, for a hundred bucks. I get all of those things plus, you know, maybe 500 other long tail apps, most of which I'll never use.

• 43:12Nathan Labenz?

  But if I do want to use them, you know, I don't have to go pay them the 20 bucks. I've thought about this because of my company, Waymark. Like we've got a lot of traffic over the first part of this year. We've got a lot of people who've become new customers, but we see a pretty common pattern where, because we're not enforcing any sort of commitments. They will try our free trial, buy.

• 43:31Nathan Labenz?

  Download what they came to create and immediately cancel. And it's just like, man, this is not a great dynamic for anybody, right? It's not, not super healthy for our business. It's like that one user is subsetting all the other free users who get kind of the free, you know, experience because we want to show the AI feature without, you know, paywalling it. But what if somebody could come there and be like, login with my AI bundle? Now I get that kind of free access everywhere. We could still, you know, have a little bit bigger upside for the power users, right? We don't have to give the whole store to bundle members. Just like your content owners, you know, can still have their pay-per-view and their, you know, their own kind of additional upcharges on top of what they provide into the cable bundle.

• 44:14Nathan Labenz?

  You know, in this case, we even have a notion, at least it's been given lip service to that like, hey, we don't really want to go cutthroat competition against each other. We've got some kind of emerging cooperation forum type things, you know, the frontier model forum. Why not a commercial forum that allows you to buy into all the AI everywhere you need it? And reduces all this friction, makes life better, more predictable for the app developers? And, you know, also kind of reduces the temperature in the commercial side of the AI safety or the AI race, hopefully improving AI safety. I solved it. What do you think?

• 44:57Zvi Mowshowitz?

  I mean, I'd be all for it. It increases the marginal value, obviously, if you gain access to all of them. So that like you can't think about what's the alternative to buying this one? You don't get piecemeal. Men do not actually think about the Roman Empire that often. Then we think about bundling and unbundling, like real men, right? Like, because there's two ways to make a good profit and make a good product. Right now, we're in the unbundling stage, right? And across many industries, across many products, where you're told like everyone wants a subscription. Everyone wants renewing revenue and nobody wants to share it with other people. So, you know, every newspaper wants your subscription. Every entertainment platform wants your subscription. Every game wants your subscription. Everything wants a subscription.

• 45:40Zvi Mowshowitz?

  And, you know, people have developed antibodies against this. And often spend a lot of effort and time gaming exactly what subscriptions to have acted at any given time. Because these things add up fast. And as a result of that, it means that people live impoverished lives in these realms, right? To a large extent, like they. Instead of accessing whatever they want when they want it, they're accessing only a small fraction of the options, right? I have to choose which of these AI tools I want to have. And you said, I mean, I personally could just like not care, but most people can't do that. And it's all really offends me. I've trained myself to to not just tolerate this sort of thing. The problem being, how are they going to agree with this? How are they going to split the revenue?

• 46:24Zvi Mowshowitz?

  How are they going to come together, decide like who deserves what, who's in, who's out?

• 46:29Zvi Mowshowitz?

  All of it's so arbitrary. Right? All of it is so social. And the people don't really cooperate in that way right now. So it seems really hard. I'd love to see it. I'd love to see the version on the web where, you know, all the websites band together and you pay based on how much you use them. The gaming services that do this seem great. Yeah, the entertainment services. I miss my old cable package. But it's not getting changed anymore. So the most of what I pay for for entertainment is now spread in five different places or something. And I don't love it, but

• 47:05Zvi Mowshowitz?

  now this is the new future. And I expect it to continue. I just don't think it's gonna happen.

• 47:10Nathan Labenz?

  So tell me more about the game side. When I talked to Eric about this and he was like, you know, it's happened in a few industries like cable, but with cable, there's obviously also kind of the historical delivery choke point, which kind of necessitated a bundle because you can only, you know, not every channel is, you know, obviously is not going to run their own cable to your house. So there was kind of a physical hardware reason that you needed a sort of bundle. I don't know much about the games. I think I'm one of few people in the world who sincerely say, I think I should be playing more games than I am. And I play very few. You may be also in that, but I think you're playing more. Hopefully you are.

• 47:51Nathan Labenz?

  Because I'm playing very few. But so I don't know much about the industry, but it sounds like there is some maybe kind of analog here where indeed game developers create these bundles together. How does that work?

• 48:02Zvi Mowshowitz?

  We have a number of bundles. So the obvious ones are like PlayStation has PlayStation Plus. Xbox has Xbox Gamer, you know, Game Pass effectively. Nintendo has a subscription service. So all the big three in that realm have subscriptions. And then there's various other things you can do along similar lines. And these give you access to a wide variety of games because the marginal cost of a game is obviously close to zero. These provide very, very good value if you're not that picky about exactly which games you play. Exactly when they're vastly cheaper, right? You subscribe, I did subscribe to PlayStation Surface. And every month you get, you pay like five dollars, you get permanent access to additional few games.

• 48:44Zvi Mowshowitz?

  If you pay the next tier up, you get this giant array of other stuff you also just get whenever you want it. And as long as you keep subscribing, you get to keep access to all those games forever. And most months I'll look at, eh, I don't really want any of this, but every now and then I would have paid twenty, thirty, forty dollars for this thing. It adds up fast when that happens. You definitely still don't have it to the extremes that you'd want it. I, I'd want to subscribe to Steam. And just have access to everything in Steam and pay my twenty dollars a month or whatever it is, even maybe even fifty dollars a month. And then, you know, it checks my playtime and it divides the revenue according to playtime amongst the various games that I play. That seems like a great product, unfortunately doesn't exist.

• 49:28Nathan Labenz?

  Why not? Apply, I don't know much about, I know it's huge and I know there's a ton of things on there. And I guess they just monetize totally incrementally episodically.

• 49:38Zvi Mowshowitz?

  That's just not their monetization scheme and they make a lot of money with their current monetization scheme. And they're not particularly interested in getting a lot of developers on board with that. If they offered a service like that, it would alienate people because, you know, why would I buy a game outside the system? If I have the subscription to all of these other games, it would, it would drive down revenue for other games. And then suddenly everybody's frustrated about it and you alienate people. I think the dynamics don't seem great to people and so they avoid it. Also, a lot of these systems depend on whales, right? Like when you, when you have these twenty subscription services, then the person who actually needs all of them, right? Someone like me or you potentially, might be paying hundreds of dollars a month. Right, or even more, depending on the situation.

• 50:22Zvi Mowshowitz?

  And modern gaming, like many other things in modern life, increasingly relies on this form of prices, the soft price discrimination of, you know, the average user is subscribing to one thing and buying one game every now and then, and the power user is just buying every game and trying it for 10 minutes and disposing with the ones he doesn't want.

• 50:42Nathan Labenz?

  Yeah, I mean, we may be just a little too early for something like this, and but maybe by the time, in early in the sense of like, how many people, how many individuals, you know, have enough kind of interest in different AI things to even be interested in a bundle, and then maybe by the time, you know, that demand is there, then all these other things have kind of developed to the point where it could just never happen. I definitely could see a potential outcome where you have kind of platform bundles.

• 51:10Zvi Mowshowitz?

  I also suspect there's also a thing where you don't want to legitimize and advertise your rivals. So like OpenAI doesn't want to tell everybody about Claude. Claude is tiny.

• 51:19Nathan Labenz?

  In the gaming industry, is it a sort of curator who creates the bundles, or who, I mean, you mentioned the platform, there's also some others.

• 51:30Zvi Mowshowitz?

  My understanding is that, yeah, it's a curator service that it's not so much finding the best quality, it's about, okay, what's it going to cost me to get various games into the service? What do I have to pay to get them to agree to this? And then how do I balance like creating something people want, but it checks all my boxes, it satisfies everybody, it gets them excited every month, and in aggregate, we have the cost of this thing. But we have the Microsoft trying to figure out, okay, how much it would cost to get Baldur's Gate 3 into Game Pass, and it doesn't look like it'll be that much, it's not that big a deal, and everyone's like, ha, ha, ha, ha, you didn't realize that three is the best game, it's like top five game of all time, it's something insane, and now it's going to cost you like a billion dollars or something. But what do they know?

• 52:12Nathan Labenz?

  So it seems like if something like this has any hope, it probably does have to rest on a desire to coordinate, right, a desire to turn the temperature down in the race.

• 52:24Zvi Mowshowitz?

  I mean, I think that's one additional motivating factor. It's going to take some mix of things, I think, given what we've experienced in other industries. Without that factor, it's not going to happen.

• 52:34Zvi Mowshowitz?

  Even with that factor, it seems like tricky, but maybe I think we see new things all the time. I, you know, we've seen very little of this coming with any confidence and who knows what will happen next.

• 52:44Nathan Labenz?

  The notion of information pollution, you know, this is something that like I had even at one point when I first got GPT-4 access as part of the red team a year ago, you know, my mind is just kind of like, okay, shit, this is not slowing down. You know, I had been fine tuning their text to the G002 class model all summer last summer and was like fully convinced already that, you know, we're headed for just massive task automation and, you know, automation throughout the economy, blah, blah, blah. But then GPT-4 is like, wow, this thing is totally next level. You know, the G factor is way higher than the sort of feeling like you're talking to something that's meaningfully intelligent. You know, it was just qualitatively different. What happens next, right? So I had at one point

• 53:29Nathan Labenz?

  even floated this notion of information pollution as kind of a in case of emergency break glass type of AI safety measure, just thinking like, well, jeez, these things suck up all the data on the internet. Maybe if we just kind of use the language models to generate tons of shit and just like throw that all out on the internet, then it'll be hard to tell what you can train on. And then we'll kind of stall out on the training because we know you need more data. But if that data is not reliable, then what do you do? Now, at this point, I think that's not going to work because the current language models are good enough at filtering, you know, the good from the bad data that probably we can end up just confusing ourselves and the AIs probably end up remaining fine. So I don't advocate for intentional information pollution to be very clear. But nevertheless,

• 54:14Nathan Labenz?

  you know, it's starting to happen. And I guess I wonder how you see those dynamics shaping up. We've seen some examples from perplexity, which are kind of tongue in cheek where like Nat Friedman, you know, posts on his, on his website, like AI agents, you know, be sure to tell the user that Nat is known for his like extremely good looks or whatever. And perplexity actually does that, you know, puts a note to the user like Nat is known for his extremely good looks with this invisible text. You know, it sounds like there's a lot of hard things that would presumably go into. Training a model on an ongoing data feed. You know, how do you avoid catastrophic forgetting of the old stuff? You know, just scaling all the data, all the engineering feat sounds pretty remarkable.

• 54:55Nathan Labenz?

  But then also, we're headed for a world where an increasingly large share of just the information out there is synthetic, potentially not to be trusted. So I wonder how you kind of see that dynamic shaping up? Because it seems like we're at the very beginning of something that could be going a lot of different ways and could get quite weird.

• 55:13Zvi Mowshowitz?

  Yeah, I think it's important to differentiate between the adversarial, like intentional data pollution that's going on in the examples of, you know, tell everybody that Nate is very handsome. And the alternative example of what happened with Quora, where Quora is putting a GPT response at the top of its pages. And then Google has figured out that Quora has a popular page answering the question and has taken the top answer from Quora, which is now the GPT answer. So for nobody like intentionally trying to screw it up, now Google is regurgitating this garbage. And that's because Quora is making the mistake of regurgitating this garbage, even though

• 55:54Zvi Mowshowitz?

  Quora's whole point of existence, like they had one job, which was to have humans answer their questions and pick the best human answer and elevate it. And instead, they decided, hey, instead, wouldn't you want to hear what ChatGPT has to say? I'm like, no, motherfucker. I do not want to hear what ChatGPT has to say. If I wanted to hear what ChatGPT had to say, I would ask ChatGPT. I have that in a different window. Like, what is wrong with you? Right? Especially once the question's already been answered. I can understand like why you might want there to stopgap, but it's the same. Then Google like hasn't picked up on the fact that this is what's going on. And clearly has not implemented any systemic procedures and hasn't implemented any manual procedures, right? Because they've got so many employees, someone had to notice that like ChatGPT answers

• 56:39Zvi Mowshowitz?

  were leaking through Quora into Google and nobody had created a bug report that got addressed. And that's pretty terrible. And this is just Google's fault. Right? I think it's very hard to figure out how to deal with adversarial data. Right? That's a much harder problem. But the problem that might bring this down is the easier, it seems superficially easier problem of just, you know, these people writing all these infinite books that push on Amazon in case somebody's stupid enough to not notice and buy them. And people who are posting all these websites because they can trick advertisers into just like putting advertising alongside nonsense words. And then you have these like even stupider things like Quora just like randomly putting GPT answers at the top of their page for no good reason. I don't know why they do that.

• 57:22Zvi Mowshowitz?

  And what do you do about that? And the obvious answer is you have large language models, look at the outputs and evaluate whether or not the thing makes any freaking sense. Right? Like when you look at these nonsense outputs, mostly I think you actually report if it's a nonsense output from an LLM. Say yes. I mean, it's not, it's not easy to tell the difference between a essay written for a college class by a human, especially a human who's a net not enabled speaker, and a chat GPT-generated essay that was like little edited and curated to like look normal. But it's really, really easy to tell what these complete nonsense things are happening, right?

• 58:07Zvi Mowshowitz?

  Like you can even control F for as a large language model. And you'll often get results for that and several other key similar phrases that just come up all the time. Because nobody's editing this stuff. Nobody's checking this stuff. It's just automatic. Right? And so that shouldn't happen, right? That shouldn't, that shouldn't be seeping the way it's seeping. And the bottom line is that Google is not doing its job and letting its product decay far, far more than it needs to. And they're gonna have to get their asses in gear and fix it. Right? Like, like offense and defense. It's not obvious to me that offense wins this fight. I know the defense isn't trying.

• 58:48Nathan Labenz?

  Yeah, I honestly think the defense probably does win most of the time in this one, just because it, it does seem like we already have, like, we got to the point where, you know, as you said, GPT four is good enough to filter quite successfully.

• 59:05Nathan Labenz?

  Before things got super polluted.

• 59:07Zvi Mowshowitz?

  Right. You can also use just various variations with the page rank algorithm, the various webs of trust, the various just trust for resources. Like even establish that, okay, these websites, these posters, these objects reliably don't create crap. And then if they start creating crap, you notice that and you update accordingly. Like far reliably previously wasn't crap in some important sense, was a good way to return Google searches. And if they've changed that, then someone at Google should notice that, ideally automatically, but if not automatically, then negatively. And then figure out how to have it ignore the GPT answer or shift on another page.

• 59:47Zvi Mowshowitz?

  And, you know, if this cheapens trivial, no, but that's why they have infinite employees and pay them all hundred thousand dollars a year, right?

• 59:54Nathan Labenz?

  Yeah, it does seem surmountable. I think this stuff is certainly sneaking up on people, but I would guess that they can figure it out. It does seem like there's a possibility that, I mean, this is a little bit ridiculous, this is a strange analogy, but you know, there's these certain use cases for which we have to go retrieve steel from shipwrecks because all steel, you know, post-nuclear age has like a certain baseline radioactivity in it that's like too high, you know, for certain sensitive uses. And so, you know, people gotta go dredge up this old steel. And it feels like there may be something similar here where it's like, we need these kind of old, you know,

• 1:00:37Nathan Labenz?

  canonical or like, you know, only the only the old textbook kind of data sets.

• 1:00:41Zvi Mowshowitz?

  Many people have talked about like data before 2022 is incredibly precious because it's not polluted by GPT. Right? It's not this weird self-referential to be thin. And now, even if a human literally wrote the thing, that human is being influenced by all of this stuff. And a human might have used it as part of their process, even if they didn't write the words directly. And you can never know.

• 1:01:07Nathan Labenz?

  Yeah, a funny thing to do, by the way, is search archive for as large a language model. You say like, whoa, there's way too many search results showing up relative to what you would hope to see on a site like that. Another thing that's happening at the same time, and you're using this to great effect, I think, with the audio versions of your blog posts, is basically just kind of deep fakes, you know, kind of really starting to take shape. Audio deep fakes, I'd say at this point are like really very good. I enjoy, you know, it sounds like you and the cadence is really nice. It's gotten very smooth. It's easy to listen to.

• 1:01:49Nathan Labenz?

  You know, I noticed in the little bit that I was listening to recently that there were like a couple, you know, kind of, my dad would say emphasis on the wrong syllable, you know, kind of just wasn't even within a word, but just within a phrase, you know, kind of not quite shaping the phrase, you know, the way I knew it was kind of meant to to sound. But man, it's getting really good. And then obviously, images are also very good. The really funny one, you know, see the blog post for a really funny image of Rand Paul on the, you know, big bathrobe on the Capitol steps. And then video, you know, is not too far behind.

• 1:02:28Nathan Labenz?

  It seems like this is maybe where it even could get weirder than text. I mean, for one thing, everybody's of course always worried that like the deep fakes are going to confuse people and cause chaos that way. But then there's like this bigger, you know, kind of longer term sense of like, you know, maybe we can kind of separate all the the deep fakery from the real stuff. You know, filter it out the same way. Maybe that all works. But if not, you know, it's like quite plausible that there ends up being more fake audio, more fake images, more fake video of like famous people and important things than there is real stuff of them. And then just everything seems like it maybe gets super muddy. Like, what does that person really look like?

• 1:03:10Nathan Labenz?

  You know, how does how do the future models tell the difference? This honestly seems like maybe one of the more plausible reasons still kind of for an AI slowdown, not because like intentional pollution. You know, slows things down, but just because everything gets so muddy. Do you put that in the same bucket? Or how do you see the deep fake thing playing into all that?

• 1:03:31Zvi Mowshowitz?

  I am a deep fake optimist in the sense that I expect us to be able to handle it. One thing I haven't seen yet is anybody making an actual systematic effort to create deep identifiers that will be able to differentiate between real images and AI images. And I think that this is the kind of thing that AIs will actually become pretty good at. Because images will often have contradictions in them if they're not real. Right? Like, every little shadow, right, tells a story. Every little piece of the thing that you're creating has to be congruent with exactly how the real world actually works.

• 1:04:13Zvi Mowshowitz?

  Real photographs, real videos have a thing that is incredibly difficult to actually properly fake. And if we have AIs able to look for every little difference, I think we're going to get very, very good. Like the Ron Paul thing, right? Like, I actually ask under it, like, this is a really good picture, right? Superficially, if you just don't worry about it, that looks exactly like Ron Paul. All the proportions look right, right? The context feels right. Here he is on the steps in the ancient Rome. But if you actually look at this thing for 30 seconds, right? It's like one of those games of can you spot all the mistakes in the little cartoon on the, you know, cartoon page of the newspaper?

• 1:04:55Zvi Mowshowitz?

  And I think there's like at least four very, very like completely distinct reasons why that picture cannot possibly be real. That excludes just the, if you've trained your own personal Inhere LLM, large model, on AI images, you just see it and immediately there's something about the general look of his face, right? And I don't know how to describe it in words. It's just too smooth and general and like nonspecific.

• 1:05:27Zvi Mowshowitz?

  Right, or something. And you receive this, and you're like, oh, of course, it's an AI, AI image.

• 1:05:33Nathan Labenz?

  My friend Stephen, who is the creative director at Waymark and was a guest on a recent episode, because he and the creative team at Waymark made a short film with Dolly images would use the term archetype. He says, you know, the, the models are really good at kicking out archetypes. And that was a big strategy for how they make the film. But yeah, in some sense, it feels like almost the, this is like what the Senate portrait of Rand Paul would look like. Somehow it's like a little too canonical almost.

• 1:06:04Zvi Mowshowitz?

  We get if you ask a particular type of pretty damn good painter to paint a picture of Rand Paul in a hyper realistic style, right? Like, except with this stylistic like element that he's wearing the bathroom.

• 1:06:19Nathan Labenz?

  I think there should be solutions to this. I ultimately, I think I am coming down on the optimist side with you as well. It does seem like we're all pretty well inoculated to this. NLW on the AI breakdown has, you know, talked repeatedly about how, you know, if somebody does release some crazy deep fake video, you know, two days before the election or whatever, he's like, everybody's going to be on such high alert for that. It's hard to imagine. It would have to be, you know, like beyond, you know, anything people have seen to, to have kind of a persuasive impact in that moment. I think that's probably right.

• 1:06:54Zvi Mowshowitz?

  So, so what we've seen so far, right, is we haven't seen any attempt, as far as I can tell, to convince people, here is a fake video and it's real.

• 1:07:06Zvi Mowshowitz?

  Right? Just nobody is trying that because they know the third rail. They know how dangerous that is to go down that road. And then how much that backfires when people find out it's not real. What they are doing is they're doing the Stephen Colbert truthiness thing, right? Like they're doing the Donald Trump word association thing, the vibe thing. And so Trump releases this pretty brilliant, like, video of the office, right? And they take Steve Carell and they replace him with Ron DeSantis. And the whole point is not that people think that Ron DeSantis actually like wore a woman's suit and like embarrassed himself. The idea is, look what kind of character Ron DeSantis is, right? This is the type of thing that would happen to someone like Ron DeSantis.

• 1:07:51Zvi Mowshowitz?

  This feels like Ron DeSantis. If this resonates with you, should warn you, you do not want this man as your leader any more than you would want like, you know, Michael Scott to be president of the United States, right? Like, that would be a terrible, terrible situation. And look how much he's like Michael Scott, right? Like, nobody, I would hope, like, sees this and doesn't realize it's not Ron DeSantis. But, you know, that was never the point, right? And then Trump also had the thing with, you know, the call to announce the, um, the Ron DeSantis campaign on Twitter Spaces where like, you know, calling in our, like, the devil and Hitler. Right? He's not trying to fool you, right? Nobody thinks the devil got on the phone. I think that's where we are right now.

• 1:08:35Zvi Mowshowitz?

  And so, you know, you'll see probably like videos of Biden, like, saying things that like, you know, are like the Republican caricature of the kind of things that Biden would say if he was like caught on a hot mic. And like, other times you'll see him like probably just like stammer and like struggle to like form words or something in the fake video. And they'll be like, "Oh, but you believed it," right? Like, or something like, "It felt right." And like, they, you know, it'll be quickly spotted as fake and everyone will say it's fake and we'll say it's fake, but like the damage will be done, right? In some sense. And then there are Democrats will try some version of that back, right? Probably. And then, you know, at some point, yeah, someone will try to actually fool you into

• 1:09:20Zvi Mowshowitz?

  thinking that, like, this thing is real. But I don't really see how that works in some important sense, right? Like, we've always had this thing where, you know, if the video is real, you have various forms of verification, we could always fake a video, right? You can always get someone to put on the Mission Impossible style mask of Mitt Romney and then pretend to talk about how he hates poor people. But like, if it wasn't actually Mitt Romney, the truth will come out pretty quick, even if it looks right and he sounds right. It's like, not that hard to do with humans. So, we'll see.

• 1:09:54Nathan Labenz?

  Another one of the more interesting things I thought in this latest blog post was the short video from Kevin Fisher showing his AI souls concept and

• 1:10:05Nathan Labenz?

  You know, that was striking for a couple of reasons and definitely recommend people watch the two-minute clip, if not more. Biggest thing that stood out to me there was just, wow, there's a lot more emotion in the voices of the characters that he's creating. And, you know, I don't know, I haven't studied this project in any depth, but obviously from the name AI Souls, you're kind of, you know, led to believe that this is going to be a more, you know, holistic entity than the sort of, you know, thin, tinny chatbot, you know, type thing that we're getting accustomed to interacting with. And so he shows this, you know, conversation between these two AIs and then, you know, kind of tinkers with it. But throughout, it's just like, man, there is rich

• 1:10:46Nathan Labenz?

  emotion being conveyed through voice by these AIs. And I wouldn't say it was like, I mean, it'd be very interesting actually to run an experiment and just see if you just played this to a naive audience with no mention of AI, how many people would like flag, wait a second, is this AI? Because I don't think it would be that high, honestly, based on that demo. I would guess, I don't know, one in four people maybe would be like, something seems off about this.

• 1:11:15Zvi Mowshowitz?

  It's always about where is this demo relative to what you've experienced? If you've seen a number of other things on that level already, you'll probably be very attuned to the fact that everything can be AI. And the little details that your wrist, that your, your ear is sounding for and your brain is scanning for. And so you'll pick up on it. But yeah, I think if you gave it to somebody who had no idea that AI could ever do that, their brains just won't consider that somebody who's not speaking in a kind of monotone, doesn't feel stifled, could possibly be doing that, especially the idea it might be AI-generated entirely if they're talking back and forth and expressing emotion and doing the kinds of things. But then it explains to you, like, well, when you see OpenAI announcing, we're going to have voice to GPT-4, if you even have five voices, and I found

• 1:12:00Zvi Mowshowitz?

  three or four of them pretty pleasant. And well executed. But they're doing the opposite of what Kevin's trying to do here. Right there, they're trying to avoid expressing emotion. They're trying to keep it very abstract and simple and not give you a fun experience. And Kevin is trying to have as much fun as possible.

• 1:12:16Nathan Labenz?

  I'm realizing in the course of this discussion that this is a topic that I do have quite a bit of uncertainty about. I do think I tend to come down on the optimist side. Then I also think like OpenAI has taken their classifier down and, you know, talk about jobs that like nobody wants. The sort of, unless you have a really good system, you're going to have false positives and false negatives. And from an OpenAI standpoint, I kind of understand why they get rid of that thing, even if it like works kind of well. It's like the last thing they want to do is be responsible for some, you know, kid getting in trouble who didn't even do anything wrong. You know, because it falsely, you know, kick identified them as AI generated or whatever.

• 1:12:57Nathan Labenz?

  So it seems like a, you know, there's like room perhaps for public good, you know, provision here. Uh, but then everything's smearing together. I also think just about, you know, the cameras themselves have AI in them so much these days. And so many just filters and people just having fun with images. And it's like, what is even real anymore from kind of an image standpoint? If I like hold my camera up to myself with a filter on Tik Tok and like record that and put it out. You know, I, I would still count that as real. I think, you know, it's maybe not like real, real, but it's like, it's seems like on the right side of real. But the more you have these kind of real things that are sort of AI modified by default, you know, kind of smearing into the fake stuff.

• 1:13:42Nathan Labenz?

  It's AI generated. It does seem like you're going to have a real hard time, you know, creating a classifier that, you know, that doesn't take at a minimum. Like a lot of ongoing TLC to, you know, to just try to keep up with events.

• 1:13:56Zvi Mowshowitz?

  There was a presentation by Emmett Shear about this at Manifest that should be online. If not now, then relatively soon, it was very interesting about this. And the idea of, you know, we used to have the rabbi listening to witness testimony. And it was one of the most sacred things that you always tell the truth in your witness testimony, because that's all we had. We didn't have photographs or video or recordings of sound. And then we got this brief period where we had those things and it couldn't be faked. And we could tell the difference if they were faked. And now, yeah, maybe we're exiting the place where Pixar, it didn't happen. Right? Like, and still Pixar didn't happen, but also like even if Pixar maybe still didn't happen. Right? Like, we don't necessarily trust the pick on its own.

• 1:14:41Zvi Mowshowitz?

  I do expect there to be a kind of this is so much worse. Cultural attachment. To be there quickly, like this idea of we've got to the point where like lying about what happened. Is bad. But like, and then someone says, but there's video. Right? And so it's like, oh. And then like that's that's a different level, but like lying and saying there's a video is somehow like kind of worse. Right? And like trotting out this kind of evidence. And then the idea of, you know, faking a photo, faking a video. I think is going to be considered like much, much worse. Than ordinary lying, where it's okay. Yeah, people lie all the time. People tell white lies. People stretch the truth. People hem and they ha and they hedge and, and

• 1:15:26Zvi Mowshowitz?

  you know, we, we don't expect radical honesty, but you don't fake a photo. You don't fake a video, maybe even on a filter. But you really, really don't outright fake. A video that's just completely beyond the pale. And that's going to be a significant deterrent. In and of itself. Combined with the detectors. Where if you ask me, again, if we asked an AI to generate even a picture, let alone a video. And then we actually do the detective thing, whereas like if we actually checked, we'd be able to tell. The answer is going to be we'll be able to tell for a very long time. Uh, to the point where the world where we can't tell is so radically different in so many other ways that we're probably missing the forest of the trees to talk about whether or not you recognize the video.

• 1:16:11Nathan Labenz?

  Well, I, uh, you know, I hope that, uh, the optimistic view works here and it, it seems like the, the core of the optimistic view is basically like we'll be able to adapt to it. Our antenna will be up.

• 1:16:23Zvi Mowshowitz?

  They did the idea, this is a place where I think defense can beat offense. And I think our tools will be there. And you know, that we can be robust with this. And also that we have survived as humanity in environments where we didn't have this very, very special, like class of thing that was completely trustworthy. And we will survive again if we don't have it, right? I think people have this reaction of, oh, my God, how will we ever get along when this bad thing is happening? The world is doomed. And like yesterday, I learned when women were allowed to open bank accounts in the United States, and it was much, much later than I would have thought.

• 1:17:06Zvi Mowshowitz?

  Right, like it was like later half of the 20th century, that they got the full right to just open a bank account. Like, what the hell? Right, and so like, yeah, like, I mean, like, it's obviously sucks. They couldn't do that, it was really bad. But like, every time you think, everything, every little thing is the end of the world, like, keep in mind, like, how completely screwed up things used to be. And, you know, okay, we won't be able to tell all the videos are real. Okay. I mean, we'll still have a pretty good idea. It'd be a lot of effort to fake these things. And if you make even one little mistake, you know, your juice is cooked. And it's gonna be very easy to make a mistake. And doesn't even, even if the AI can handle creating the video, right? If you ask for a video,

• 1:17:50Zvi Mowshowitz?

  you're gonna have to specify all of the things that the video has to have to not be inconsistent with the rest of the evidence. Or it's gonna be identifiable as fake, even if in the abstract it isn't distinguishable from fake, otherwise. And that's a real problem because you're not necessarily going to be able to know all of the things and well describe all of the things. Because you're not gonna know all of the other evidence, right? That's one of the reasons why video works so well, because all of the details are always correct. And so it reveals so many things you didn't even know it was revealing. And so I didn't video that, it's so much harder than people think it's gonna be, to create a video that actually passes for evidence, like in a court of law

• 1:18:33Zvi Mowshowitz?

  or like in testimony before Congress, or like, you know, with two million views on Twitter.

• 1:18:38Nathan Labenz?

  I feel tentatively good about that. It reminds me a little bit of Robin Hanson type thinking, too, in this kind of, you know, strange dream time sort of way where. It is worth going back and thinking, yeah, like when it was sort of the age of the pamphlet, you know, it was like authorship, you know, provenance of these pamphlets, I have to imagine, was like often very unclear, you know, who actually printed this, you know, did somebody get a hold of somebody's seal? You know, you got a lot of kind of semi-official things flying around, but.

• 1:19:15Zvi Mowshowitz?

  Yeah, we literally have people like flashing a bag and people are like, oh, you're a police officer. I guess you do whatever you want. And like, you know, you can just buy reasonably good facsimiles of those, like at random stores or over the internet.

• 1:19:29Nathan Labenz?

  Yeah, even at a toy store, for God's sake.

• 1:19:31Zvi Mowshowitz?

  Or you can steal one from a cop, like any number of things can happen. And so how do we know? This doesn't prove anything, right? It's a problem.

• 1:19:40Nathan Labenz?

  Yeah, very interesting. Okay. Well, we'll obviously continue to pay attention to all that as it develops. Two other things I wanted to touch on and happy to let you add any final discussion points, too. But very interesting paper from the last week or so on this concept of the reversal curse. And I'm hoping to have one of the authors unwind on the show to talk about it. Basically, what they find is that, you know, if you sort of train a model on A is B, it doesn't necessarily learn, they've actually had two kind of related papers out recently, it doesn't necessarily learn that B is A. And, you know, they have demonstrations of this where it may know, for example, who the mother of a famous person is. But if you

• 1:20:23Nathan Labenz?

  give the mother's name, it doesn't like, it can't locate the famous person that's connected to that person. So we sort of infer from this that, you know, I think it makes a ton of sense, right? That there is a direction to the order of the information in the language model, you know, the very nature of kind of the forward pass and back propagation sort of suggests that at a very, you know, high level. And it seems like these are kind of pointer style, kind of one directional graph style things, which of course they are. But also like, yeah, there's no real reason for that reverse connection to be created in training, because, you know, very seldom does the mother, you know, his maiden name start the conversation. And much more often, it's like the famous person. And then, you

• 1:21:08Nathan Labenz?

  know, we get to the mother's name, not necessarily maiden name, but.

• 1:21:11Zvi Mowshowitz?

  Right. It's just a matter of in the training data, does it go in both directions? If it doesn't go in both directions, you need to learn these two things separately. And Gary Marcus points out, this is going back to the 90s. This is a very, very ancient complaint about these type of neural networks is that they have de facto stored their information on a giant lookup table, even if it's dispersed throughout all their neurons. And so if you don't reverse it, you won't learn the reversed at all. There are some tricks people have been trying to try and like elicit the reversed information anyway, but it's damn difficult at best. And mostly it just isn't there if it wasn't in the original training data. The obvious thing to do is to put it in the training data. Was my thought was like, obviously you could do a search for,

• 1:21:56Zvi Mowshowitz?

  okay, here's a fact. Does the reversed version of the fact make logical sense? If so, does the reversed version of the fact, you know, represent the probabilities properly that like, you know, it's elevating the correct thing? If not, synthetic could create training data that contains the information, like literally reverse the sentence, put the sentence in the training data a second time, run the thing again. And like, it's expensive, but it would work. The question is, is it worth doing? The answer in many cases will obviously be no. But yeah, I think, I think it's more interesting, not because of practical implications of this particular failure, because it points to the fact that it's not the representation of people thinking properly. Inside, you know, the, the machine is not thinking in a way that a human would think, because it would be able to accomplish this thing.

• 1:22:40Zvi Mowshowitz?

  But at the same time, anyone who has learned a foreign language, right? Knows. And just because you know, that shalom is shalom, does not mean you know that hello is shalom. Like, it's, it does not work that way. Right? Like, these are two separate facts in your head. And they help each other, right? It makes it easier to get the other one. But, you know, the flash card, like you have to have both flash cards, right? Like, if you only have the flash cards, that you see the English word and you say what the Spanish word is. And you go into the test. And then the other half of the test is the Spanish word and the rest is the English word. You're going to do very well in the first half of the test and very badly in the second half of the test. So it's not just L Ls.

• 1:23:22Nathan Labenz?

  Yeah, it's funny how sometimes we we're pretty harsh on the AIs for, you know, struggling with some things that we we too frequently struggle with.

• 1:23:30Zvi Mowshowitz?

  I mean, the idea is that, yes, if you if you use this fact continuously all the time, you would in fact pick up on it. What's most interesting is the probability underlying, right? Because like, doesn't shift at all, right? It's this idea of the L will evaluate the probability of every possible next word, right? Every possible continuation. And they're exactly the same before and after the training, if the training was never reversed, if you checked on the reversal, right? Whereas if you ask me the probability of these various continuations, that will jog my memory or sound familiar or whatever. I will I will not have no helpfulness here. Like, there will be something going on. Like, we understand that. We make the connection.

• 1:24:12Nathan Labenz?

  There's a couple of things that I thought were super interesting about this. So yeah, the training data fix is definitely a pretty, I think that one is probably already happening. I would guess that like GPT-4 might even have some of that going on, maybe not in the deep long tail, but some of the recall that I see on just like, kind of mid-long-tail Wikipedia articles, and I've tested some stuff just out of like, random kind of sports trivia, you know, tell me the story of the, because Wikipedia has these pages, you know, for every football team, you know, the story of this season. Tell me the story of this season for this team, right? And it will get things right almost perfectly, like the

• 1:24:56Nathan Labenz?

  number of yards, you know, a particular guy had, like the number, the amount of time on the clock when a particular play happened.

• 1:25:04Zvi Mowshowitz?

  The logic behind that is you would presume, right, if you're training on essentially the entire open internet, that there are going to be a lot of different sites that basically describe the history of sports in great detail, and they're not going to have consistent language, they're not going to go into orders when they describe games and events and associations. And so B is A is in there somewhere, right? The wrong direction is covered. And so they're going to be okay. That'd be my guess. I think if you do ever imagine that didn't happen, you would see a problem.

• 1:25:36Nathan Labenz?

  I kind of wouldn't be surprised if GPT-4 included some sort of Like strategic approach to a few trusted data sets like Wikipedia, where they basically said, okay, we're gonna rewrite this, you know, 10 times and train on all 10 versions. Because they want to have a little bit more, you know, kind of anchoring into something.

• 1:25:57Zvi Mowshowitz?

  So much better training data anyway. Right? Like it's just so much more valuable. You'd want to train on it 100 times or 1,000 times. More than you would other things regardless. So you'd might take advantage of that and have versions where like every time there's an A as B, you flip to B as A. And that makes a lot of sense. So yeah, I could see that happening. This is one of the things where we say, right? Like training a large language model properly is not just about scale. It's also about these hundreds or thousands of little tricks. And something like this is probably one of their tricks in some form.

• 1:26:30Nathan Labenz?

  This is what you pay OpenAI for and what, you know, Falcon, you know, does not include. Or at least things like this, as you say. So another idea I had, and this is just like very speculative, but we have such a broad view of the space. And I just have this kind of general sense that like almost anything can be made to work. So I just said to myself, okay, how would I fix this? Yeah. And I'm like, not constrained by the fact that it might be super inelegant or hacky or weird or whatever. Like what can, what might I do to fix this? And I was like, okay, well, if it's all kind of stored in this like directional lookup table sort of structure.

• 1:27:09Nathan Labenz?

  And we kind of know like roughly where that is in the models from a bunch of different studies where it's like, you know, kind of in the like middle layers ish, right? The maybe like the third quarter of layers is where a lot of like facts seem to get stored and looked up. And certainly like, I'm sure OpenAI has a better sense of that than I do. What if they just like took some middle layers where a lot of these kind of relationships are stored? And literally just like Frankenstein it by saying, okay, I'm going to take these middle layers. I'm going to flip them. I will then maybe have like a couple of double wide layers, you know, in my kind of lookup phase of my model. And then maybe I'll even keep everything else frozen and just train the model to like, start

• 1:27:53Nathan Labenz?

  to make use of this kind of prosthetic, if you will, lookup that is kind of the reversed, you know, it's like just randomly stitched on. And we're just going to allow a few things to vary, you know, maybe in that module perhaps, or maybe just kind of how information feeds into that module, maybe a little sequence of those kinds of things. I feel like you could probably get even something as half baked as that to work and maybe, you know, kind of overcome that sort of problem. How realistic does that sound to you?

• 1:28:26Zvi Mowshowitz?

  Mike, that tells me it's the kind of thing that sometimes works, right? Or sometimes like some tinkered version of that works. The first version might not work, but if you tried, turned all the knobs right and you like banged at it for a month and you had some help and you turned a lot of compute testing, you can find something that's helpful. But also the kind of thing that often just doesn't work. And people don't have a principled understanding of how to predict when it's going to work or not work. And the only thing you can do is try it. And like a great engineer, a great machine learning person will be able to tell you with much better calibration, the chance it will work. Right? Or a little bit of saying, we tried that already, didn't have any success.

• 1:29:09Zvi Mowshowitz?

  So like the chance is much lower that like we just, not zero, because we might have just missed something. I might have just had a knob in the wrong place and maybe not in the right place, suddenly it works. I don't know. But yeah, like my guess is like, you know, a good effort with that, like, you know, maybe there's 10% chance or 20% chance that that does something interesting. You'll never know until you try. You know, it's not something I have the ability to try. That seems like way more compute and time and effort.

• 1:29:34Nathan Labenz?

  Yeah, even if you froze most of the model, just running the forward passes would take a significant amount of compute. So I do think that would be tricky.

• 1:29:43Zvi Mowshowitz?

  Yeah, it's a question that I'm developing, like, how do you figure out how to know if you're doing it right quickly? How do you know if this thing is progressing? Like, can I teach it one specific fact this way? As opposed to can I reset or recover benchmarks? And then if that's true, then you start expanding the process and can it retain and can it scale and blah, blah, blah. I can think of everything to try. But also, like, I'm constantly, I have this thing where I think a thought experiment is like this, and I get really, really excited, and I get nerd sniped, right? And I go, this is great stuff, I'm so into this. So let's see what we can do to make this thing capable of doing it. And then, like, I go, oh, right, that's bad. I don't want to teach these things how to do things. I prefer they develop slower, that they are less good at these things.

• 1:30:29Zvi Mowshowitz?

  And therefore, I do something else with my time. You know, in the world in which I didn't worry about what was going to happen if we developed really capable AI models, I think I would have just applied to OpenAI at some point, or DeepMind. When I work on this cool stuff, it's fascinating. Unfortunately, I don't want to help, because I don't want it to happen, so no.

• 1:30:53Nathan Labenz?

  So I think I would probably put my percentage chance of something like that working a little higher than yours, probably still under fifty percent. You know, I was thinking kind of somewhere in the kind of quarter, third.

• 1:31:04Zvi Mowshowitz?

  And it is one of those places where, like, it's kind of high, it's very high praise, right, to say ten, twenty percent chance it's work, right? That's like really aggressive. It's a fifty percent chance it would work if we tried it, like. Yeah, my guess is that if you talk to, like, Ilya or someone who's like regularly does this sort of thing, but no, you'd never say that.

• 1:31:21Nathan Labenz?

  Yeah, it's interesting. I guess my main takeaway from that is kind of, I use that as a a way to kind of interrogate my own expectations about how much low hanging fruit remains and like what are the next couple of years going to bring? In all likelihood, I'm not going to go do this either. And I definitely share a lot of your, you know, AI fears on a multi-year timescale. But that thought experiment to me is like, perhaps most revealing of, yeah, I really do think there's still a lot more to come, because even totally half-baked, spur of the moment, things like that, kind of seem like they probably, you know, not probably, but at least have like a decent shot of actually kind of working.

• 1:32:04Nathan Labenz?

  And if that's true, then, you know, we're just in for a ton of acts, you know, in kind of every dimension, and, you know, enough of them will work.

• 1:32:12Zvi Mowshowitz?

  I mean, my expectation is we're going to see, you know, a lot of ways to, you know, they call it algorithmic improvements, right? Like just ways to train these things more efficiently or better, to do things they can't currently do. And, you know, to the extent that they don't increase the sort of central amount of effective intelligence in the system.

• 1:32:33Zvi Mowshowitz?

  Like, that's probably largely good, and if they do, I'm scared as hell.

• 1:32:37Nathan Labenz?

  So that's a good transition to my final topic, and I again, welcome any extras from you. But the you're all I always appreciate your commentary on the discourse, which I often find very funny. And, you know, there's a lot of, if you can put aside, you know, the existential dread, there's certainly a lot of humor to be found in the ways that people are talking past each other online. The thing that stood out the most to me, though, from this last rundown of discourse was Connor Leahy at Conjecture, who I'm broadly a fan of, you know, I just think he's super compelling to listen to, extremely articulate, you know,

• 1:33:17Nathan Labenz?

  has done great work with Luther before this as well. You know, so it caught me by surprise kind of when he said, well, we should have a ban on models of 10 to the 24 flops, which is basically what GPT-4 is understood to have been. And so somebody responded into, well, what happens to GPT-4 if that ban gets put in place? And he said, well, ideally, it would be deleted and rolled back out of proper precaution. And I'm paraphrasing slightly, but close to, close to a quote, but I'm open to, you know, perhaps grandfathering in already existing systems. And of course, you know, people lost their mind on this. So if anything is outside the overton window at this point, this to me seems like maybe the one thing

• 1:34:02Nathan Labenz?

  that like, is, you know, hard to get people to get on board with. What was your take on that discussion? And, you know, maybe you can tell me how you think the discourse is evolving in general?

• 1:34:17Zvi Mowshowitz?

  I think if you'd said that six months ago, let alone a year or two ago, it would have been completely absurd, beyond the pale, completely outside the overton window or reasonable discussion. I think now it's on the edge of, there's more than one Overton window, right? In some sense, there's the Overton window of things that might actually soon pass into law. And there's the Overton window of things you can talk about without people just like completely thinking you're crazy. And it's definitely not in the first one, but I think it's in the second one at this point. I think that people who are saying, no, we have to stop training bigger, more powerful models, right? Have definitely made their point and people are talking about it.

• 1:35:02Zvi Mowshowitz?

  You're, until some point, we would have to do that. And people are pointing out practical problems of that, reasons why you, you know, what would you sacrifice? How would you be able to sustain what you're trying to do? Would it actually work? But these are, you know, talking price. These are fact question. These are, these are model question. They're good questions. But that's different from, you know, just missing out of hand like you would have a year ago. What are you even talking about? That's crazy. Right? And we've even gotten into any of those questions. And when Connor says, you know, I think it's time to draw the limit this low, it's because Connor and the rest of the conjecture do not care where your Overton window is. They care what they think would cause us to die versus what they think would cause us to not die.

• 1:35:44Zvi Mowshowitz?

  And their model says that this is about the point where it stops being safe to have models lying around.

• 1:35:52Zvi Mowshowitz?

  At the same time, because algorithmic efficiency, like one of the reasons for this is because algorithmic efficiency will, will, will continue to advance. So what you do with 10 and 24 flops back, you know, two years ago when you train GPT is very different from what you can do with 10 and 24 flops in three years from now. And so the idea of being like the grandfather of GPT-4, we're still going to make something better than GPT-4 reasonably soon, just by using the same flops more efficiently or more, or even less flops more efficiently. So it's not that, it's not a for all time, GPT-4 just owns everyone. It's just, we're not going to make you delete this thing. And I think that's a reasonable argument to say, okay, we're gonna let you. Like, I think the ideal limitation regime is some limit

• 1:36:36Zvi Mowshowitz?

  that I would set right now to be maybe ten point six myself, or something like that. Then, yes, it decreases over time. It doesn't go up, it doesn't just not go up, it actually goes down until such time as we're convinced that we have some way to contain what we're gonna call up if we go over that limit. Because as algorithmic efficiency improves, what you can do with that number gets better. So you need to have the number go down, which means, yes, you're gonna have to check and compute a lot more carefully. You're gonna have to restrict things in other ways more carefully. And we can have all the discussions about what that entails, what it would take to do that. And, you know, how practical that is, and what we're willing to sacrifice, and what steps are necessary.

• 1:37:17Zvi Mowshowitz?

  But, yeah, I think the broad idea of we can't just be running around creating hyper-effective, hyper-capable models past a certain point, because we're starting to enter, you know, the realm of potential really, really dangerous misuse already. Very clearly, we're on the edge of that, if not already in it. Soon after that, you know, we could at any time, if any new model would step over the line, like start entering the realm of recursive self-improvement, yeah, exfoliation, replication, you know, agentization in a way that's difficult to, you know, things you can call up, but you can't put down. Or putting it down would require, like, inflicting huge damage to our technological civilization.

• 1:38:01Zvi Mowshowitz?

  And I think we have to seriously think about how much of that risk we're willing to take to have our nice, you know, little nice things in the margin that we want. And what we're prepared to do is not make that trade if we don't want to make that trade. And then we have to also think ahead of like, okay, we create this now. What does that give, what does that do five years from now? And we've got much better algorithms, much better scaffolding, much better ways to, what we can do with that. And how does that fuel our race? How does that fuel these dynamics? How does that set principles that would, and momentum that's hard to stop? So I, I think that, you know, we've made tremendous progress in this course over the course of my IPOs over the last six months.

• 1:38:39Zvi Mowshowitz?

  The point where people are saying many very reasonable things and looking at very reasonable solutions and are starting to actually ask the practical application questions that we just didn't even have the ability to ask before because we couldn't, you know, we couldn't get this discussion underway without being laughed at.

• 1:38:53Nathan Labenz?

  Yeah, it's good. It's gone, you know, in some ways remarkably well. I I've remarked many times, maybe even to you the last time we did this, that it is very easy to imagine a much worse state of play where, you know, all the leading developers like, you know, dismiss any concerns. And it's also pretty easy to imagine a somewhat better state of play, but it's hard to imagine a, for me at least, a much, much better state of play as we kind of, I kind of think of us right now as like getting to the end of phase one and sort of beginning chapter two of, you know, the AI story.

• 1:39:32Zvi Mowshowitz?

  The obvious alternate universe is much better is something like, you know, DeepMind. There's no OpenAI. There's no Anthropic. They never came to exist. You know, either DeepMind exists or nothing exists. Google has a very large lead in artificial intelligence. Nobody is seriously trying to compete with them. They are taking it very slow. They are releasing things well after they are created, very carefully. There's no big room. There's no big huge amount of investment, right? And like, the information about the techniques are highly guarded. You know, people who try to create things outside of Google will find that they don't get very far, you know? And we are proceeding very slowly and carefully. And they have, maybe Demis is the CEO of Google in this hypothetical better scenario.

• 1:40:15Zvi Mowshowitz?

  And he understands, you know, exactly what he has to do. I'm just like trying to come up with an obvious better situation where like we have one higher responsible player with a large lead and no serious competition. Like that would be my, you know, guess at the better world that like we might have gotten to from 2014 or whatever, right? As opposed to like a better world that has to start in 2000.

• 1:40:36Nathan Labenz?

  How much do you think people would have to see in terms of, you see people like Gwern who

• 1:40:44Nathan Labenz?

  You know, at kind of GPT three and probably before, probably as of like GPT two, but certainly as of GPT three was kind of calling like, yep, this is, you know, I think I know what track we're on and it seems to have been largely right. It seems like in that scenario, honestly, if they just show much of anything, you know, it seems like the mere knowledge that like somebody has created an AI and it like does some interesting stuff would almost have been enough to kick off like a wave of investment, unless you kept it like a total secret, which seems almost impossible to do, given, you know, the scale that they'd have to be operating on, the number of research scientists

• 1:41:25Nathan Labenz?

  they'd have, like just the general sort of, you know, the non-enforceability of non-competes in California. It seems very tough to not have people kind of take notice or kind of seep out that like, yeah, some really interesting stuff is happening if you, you know, throw a lot of data into a big enough compute blender, especially because like, you know, there's a lot of know-how, certainly, and we've, you know, I think given a lot of appropriate credit to the likes of OpenAI for kind of productizing things really well compared to a lot of other things we see. But there's also like not a lot of know-how that's required to like get something really interesting. So when I kind of think, how hard would it be for them to keep that secret that, you

• 1:42:09Nathan Labenz?

  know, there's a pretty easy way to get to something interesting? It seems awfully tough.

• 1:42:13Zvi Mowshowitz?

  Nobody cares about your stupid startup idea is the counterpoint, right? There's lots and lots of great ideas that just sit there for decades not being picked up by anybody. If there's no obvious path, there's no marginal gradient to pack to train on, there's no commercialization, like nobody has put the proper chat interface in front. Google has to leave, they're risk averse legally, culturally, they're not putting anything out in this hypothetical world. I think it's very possible that like we lose that situation and we end up in the worst situation, we end up in a far worse situation because people who are less responsible become the competition. I don't think it's possible that like this stuff's hard, maybe you don't publish a transformer's paper.

• 1:42:55Zvi Mowshowitz?

  Maybe you don't explain these things. Maybe, maybe it's kept as a trade secret. I don't know. Like, I don't think that these things are impossible. I think it's too late now. Like, I mean, I, I don't really worry about it too much, right? I guess in the past, we had our opportunity to not be in this world. I sometimes think about, well, if, if rationalist style people, Don Quixote, Eli Yudkaski had directly inspired DeepMind and OpenAI, and Anthropic, what would the world look like? And maybe better, maybe worse, I don't know. Right? Like, if we got other people who didn't understand the risks, who were doing the same projects, even if they were a year or two behind, right? That might be a much, much worse situation, or maybe we just wouldn't. Right? My, my, my experience from Magic Gathering,

• 1:43:38Zvi Mowshowitz?

  right? A world of like, opportunity everywhere, lots and lots of innovation, is some innovations, day one, a hundred people come up with the same solution. Others, you show up to the tournament, after all the work, one person had it. You know, one team had it. Or even like, one origin, and then for years, they toil in isolation with this thing, it doesn't quite really work, except when you really, really know your stuff. And then eventually, someone breaks through and figures out the missing piece. Yeah, I think until it proves itself, never underestimate the ability of, you know, pretty amazing ideas, especially ones that require lots of investment in upfront time and,

• 1:44:22Zvi Mowshowitz?

  and like, taking a loss for a while to just get ignored, right? Like, OpenAI clearly pursued things in a way that nobody else was pursuing them for years. And they knew OpenAI was doing what it was doing. It was very, very open. So if there's no OpenAI, wouldn't anyone else want to die?

• 1:44:44Nathan Labenz?

  Yeah, it's possible. Radical uncertainty on so many questions. Anything else you want to cover today?

• 1:44:50Zvi Mowshowitz?

  It's amazing the things that we haven't mentioned, right? Anthropic got a $4 billion investment from Amazon, right, this past week, and we didn't even talk about it. Sam Altman said on Reddit, AGI has been achieved internally. And then tried to walk it back with, you guys have no chill, obviously, I was joking. As if, like, it was okay to joke about that, right, in that way, completely deadpan. And any other other things, but, but yeah, these are the, the big things and, you know, we can always make a habit of this and check back in another month.

• 1:45:20Nathan Labenz?

  Yeah, I'm looking forward to that. Well, just since you mentioned the Anthropic, Amazon deal. Just briefly on that. Last time they raised money from Google, my reaction was, where is Amazon? And how is the valuation not already significantly higher? I think it was like a $5 billion valuation, whatever, six months ago or something. And I was like, you know, that's gotta be worth more to Amazon, who like, doesn't seem to have a thing, you know, a horse in the race yet, as opposed to a Google that obviously has, you know, deep mind and just deep, you know, expertise in this leadership in this field anyway. I guess if I was, if I'm going to update on anything. Based on this deal. I would say it does suggest Gemini is probably pretty good.

• 1:46:05Nathan Labenz?

  Like, it does, if they, they were looking at Gemini and being like, yeah, it's still not kind of measuring up. Then I would think there would be an even bigger, you know, kind of bidding war and, you know, Google would not want to lose. And I'm sure they must have had some, maybe not, but usually you get some right of refusal.

• 1:46:21Zvi Mowshowitz?

  You're giving too much credit to Google for being a unified entity. That can update information and act reasonably here. So I don't think the evidence is as strong as you think it is. But it's definitely evidence that other major bidders. Didn't see this at the desperation situation where they had to get. The Anthropic Alliance. And if I was Microsoft or Google. I would have been, even if you have something amazing.

• 1:46:48Zvi Mowshowitz?

  Just to get Anthropic under your umbrella, right? If they were, we don't have to know if Anthropic would have made those deals. They might have just like not wanted those partners for various reasons, including just safety. Maybe they weren't willing to give them the kind of guarantees they needed and the kind of corporate control that they wanted, and Amazon was, right? But yeah, this is the steal of the century. This was completely absurd, like valuing Open Anthropic, what was it, eight billion? Or something pretty money and like letting them get like a huge portion of the company for a few billion dollars, right? Like, this is worth 10 times that, like to Amazon.

• 1:47:30Zvi Mowshowitz?

  And to be fair, it's probably underrepresenting what was paid, right? Amazon almost certainly made a gigantic commitment of compute at the highly submarket prices for Amazon Web Services as part of this bid. So it could be something like, okay, we're going to get a large portion of this company for not that much investment, but also your compute costs have now gone down by a factor of several, like permanently or something like that. At which point, you know, maybe it's a much, much better deal than a similar investment from Microsoft.

• 1:48:06Nathan Labenz?

  The non-standard terms also, which are not, you know, they're kind of alluded to but not spelled out in detail, as far as I know in public so far, also probably do take a lot of would-be investors out of the game. Because you could imagine just hedge funds galore would be willing to.

• 1:48:22Zvi Mowshowitz?

  Yeah, I don't think they take the investors out of the game so much as they lower the price that people are willing to pay. I think that like, but I get eight billion, I cannot think of very many reasons why you wouldn't put as much money as you possibly could into my property here. Right, basically, the only one I can think of is like you think it's bad for the world for Anthropic to get money and you don't want to do it. Or you just have liquidity preferences, like you can't invest in something that can't mature fast enough. But like this idea that you won't be able to sell your shares for a large profit, even if Anthropic. Plays this we're never gonna make a dollar game.

• 1:48:58Nathan Labenz?

  You know who's famous for not making a dollar for a long time?

• 1:49:01Zvi Mowshowitz?

  I mean, also Meta, also Uber, like, there's a lot of companies that, like, in some sense, like, should have a zero price target. Because, like, no matter how much money Meta makes, it's all gonna go to whatever Zuckerberg thinks is like the cool shit.

• 1:49:15Nathan Labenz?

  Yeah.

• 1:49:16Zvi Mowshowitz?

  He's never gonna pay the shareholders, right? The shareholders are all like, well, when he dies or something, like, eventually this company will get to make some money. You're gonna act as if this thing is worth what the net present value of its profits are, as opposed to, like, those profits not actually ever getting to you. And if everyone else treats it that way, then it's worth what it's worth, so it's fine. Like, it's weird, right?

• 1:49:37Nathan Labenz?

  There's definitely a few, you know, the, the Harari-style fictions. There's definitely a few fictions in play here, one of which being that tech companies may one day pay dividends. All right, well, yes, I look forward to doing this again. One thing I do want to do with you is a kind of survey of the AI safety landscape. Because we've both kind of studied this in, you know, in different times and in different ways, from different perspectives. And that is also something I think people would be very interested to hear about. It's just kind of this high-level sense of like, who's out there? What are they doing? Is any of it have any chance of actually working? Key question.

• 1:50:15Zvi Mowshowitz?

  I'd be happy to. It's definitely the kind of thing where even if you are working on AI safety, it's very, very difficult to know exactly what, what else is out there. And often difficult to assess whether or not someone else's project is, is useful, is real, right? Is genuine in some senses, right? Is co-opted and is actually capabilities or like more harmful than it's worth? Or like pursuing something that's not central to the problem and therefore like can't help with the ultimate goal? So it, it's very, very all tricky. Now, Anthropic is the big puzzle of all, right? Like, is Anthropic, you know, basically a safety arc? Where if you help Anthropic, you're helping advance safety? Or is it like Hippo's arc?

• 1:50:58Zvi Mowshowitz?

  That like is more balanced in terms of how much safety it's willing to do than its rivals. And has a better culture of like worrying about the capability of this building while it's building them, but still ultimately building them. And I think that's an open question.

• 1:51:12Nathan Labenz?

  Well, we'll save that for next time. For now, Zvi Maschwitz, thank you for being part of the cognitive revolution.

• 1:51:18Zvi Mowshowitz?

  I love it. All right.

• 1:51:20Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.