Hollywood Strike Update and AI Roundup with Trey Kollmer

In this episode, Trey Kollmer, WGA Writer and Co-Executive of the show Ghosts, returns to the show to discuss updates to the Hollywood Strikes, including news on SAG-AFTRA and WGA. Trey and Nathan chat why actors are joining the strikes, how AI will change acting as a profession, Trey’s views on reasoning and how he’s experimenting with GPT-4, and much, much more. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS: (00:04:01) Hollywood and AI: updates on SAG-AFTRA and the WGA (00:15:20) Sponsors: NetSuite | Omneky (00:18:56) Studio approach to copyright and compensation with generative AI (00:22:17) Hollywood receptiveness to using AI and protections guild members are asking for (00:24:56) Models implementing gradient descent in the weights (00:29:14) How Nathan uses models to write for the podcast (00:34:15) Generating and mining jokes in the writer’s room (00:40:18) Untraining models (00:44:20) AI writing tools and writer perception of them (00:46:34) Context length and pooling layers (00:51:11) Microsoft China (01:00:02) Actors’ strike (01:05:45) Using 1 million DALL-E images to create an AI short film (01:09:11) Deepfakes (01:12:51) The future where anyone can be a reasonable synthetic actor (01:17:33) Trey’s take on reasoning and why Hollywood should be more open to AI (01:25:49) A monk’s experience with Chat-GPT (01:28:56) Outdatedness of stochastic parrot notion; reasoning and synthesis (01:36:14) Model vs human susceptibility to adversarial attacks because of human robustness (01:47:30) Performance jumping with abstraction (01:49:22) Language model self-delegation (01:51:04) NVIDIA’s margins compared to TSMC and ASML (02:07:35) Adding an AI layer and competing with incumbents (02:13:10) Rewind AI (02:19:27) Big tech vs old school studios (02:14:52) Dramatic ironies from the picketing lines (02:21:38) AI development moments that feel like a movie (02:26:34) CEO of Inflection’s views on misuse being a greater threat than the AI itself SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Music Credit: GoogleLM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Even she showed pictures of pretty similar image generated by stable diffusion model next to a training image that was very similar to it. And she even thought that no court would conclude that that was a derivative work. But she does think there may be questions in the training of the model. You're copying these works and using it to create a system which can just replace a lot of the the underlying creators, but that is really up in the air and that maybe Congress will weigh in and come up with new legislation because it's so new, it seems like crazy. We're trying to apply these old laws and old rules to such a different paradigm.

• 0:39Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, Trey Kollmer returns to the show. Trey first appeared in episode number 30, released back on May 30, when he and two other members of the Writers Guild of America discussed the WGA's AI-related strike demands and also shared a glimpse into how they are beginning to use AI as

• 1:22Nathan Labenz?

  part of their individual writing processes. That episode was extremely well received, thanks in large part to Trey's nuanced perspectives, and I encourage you to check it out if you haven't already. This time, we're doing something a bit different because Trey's interest in AI goes much deeper than just experimenting with it as a writing assistant. Like me, he's extremely curious about all AI developments and very well read on the subject. So I invited him back to first share an update on the strike and also to bring his own questions for discussion. We ended up talking for over two hours, covering topics including our ever-evolving understanding of AI models' reasoning capabilities, why Nvidia stock has popped so much more than other AI stocks, including their critical

• 2:05Nathan Labenz?

  partners, TSMC and ASML, and where else besides hardware, value is likely to accrue in the AI stack over time. Finally, we close with a segment that I'm calling "The Reality Writer's Room," in which we imagine our current reality as entertainment for some hypothetical outside observers, and then attempt to identify scenes of particular interest or dramatic irony.

• 2:29Nathan Labenz?

  One small correction. Toward the end of the episode, I incorrectly said that Inflection AI was one of seven companies party to the Frontier Model Forum. But what I should have said was that Inflection was one of seven companies that agreed to certain voluntary commitments as part of a recent White House statement. Those seven companies are OpenAI, Anthropic, Google, Microsoft, Meta, Amazon, and Inflection. But only four of them, OpenAI, Anthropic, Google, and Microsoft, officially joined the Frontier Model Forum. It's a small detail, but worth getting it right. If you ever notice any inaccuracies in the show, or just want to request the return of another favorite guest, please reach out

• 3:09Nathan Labenz?

  and let us know. You can always email us at [email protected], or feel free to DM me on the social media platform of your choice. And if you're finding value in the show, we always appreciate a review on Apple Podcasts or Spotify, a comment on YouTube, or just a plain old share on social media. Now, I hope you enjoy this wide-ranging conversation with Hollywood writer and fellow AI scout, Trey Kollmer. Trey Kollmer, welcome back to the Cognizant Revolution.

• 3:38Speaker 00

  Thanks. Thanks for having me back.

• 3:40Nathan Labenz?

  Very excited. This time, we're going to pursue a wide range of topics. But just for context, if folks haven't heard your first appearance, they should go back and listen to an interview with you and two of your guildmates, which was a very well-received episode. People thought that the perspective that you guys had on language models and the way that it's kind of playing into the dynamic that you have with the writer's strike was super interesting, and certainly I agreed with that. This time, you brought a bunch of questions, and I'm excited to get into those with you. But before we do that, tell us what is going on in Hollywood.

• 4:13Speaker 00

  Okay, so the two main updates since last time is that the actors have also struck. So SAG and AFTRA, their deal came expired, and they weren't able to negotiate a new one. So they are now on strike, and some of their demands are AI-related, and we can get to those. But there's also been some movement on the writer's strike. After a few months of not speaking with us or with the union, the AMPTP, which represents all the studios, invited us back to the table, and we negotiated for a bit. And it ended up Falling through this negotiation, but the AMPTP did release their latest offer. They released it publicly.

• 4:55Speaker 00

  Part of it is a tactic to hopefully get some of the writers to say, wait, that actually sounds pretty good. It hasn't been effective so far, but it's kind of interesting in some of the movement that there's been on the AI issues. Last time we talked, I think two of the big things we were concerned about were the way that these language models could be used to create loopholes that would take some of the credit from writers or some of the compensation. You know, one example is a rewrite is usually paid at a lower rate than a first draft. So they could have an AI model write the first draft and then pay the writer less for the rewrite.

• 5:37Speaker 00

  And that's just what we're like in television, a teleplay, you get paid separately for the story and for the actual script. So the language model could come up with an outline and then you would lose your story payment. But so the AMPTP did come back and their offer agreed that generative AI generated material could not be considered assigned material. It would not be considered the first draft for the purpose of a rewrite. And basically it's it's treated as just research that if you are starting to write from scratch, there are no claims for the studio or for anyone else because of the material generated by the model, which does seem like it starts to close off some of those other loopholes.

• 6:22Speaker 00

  I'm not fully sure, you know, there's all if if it covers everything we'd want, you know, the the Guild, they have to be a little bit private with kind of how they're thinking of progress in the updates to both not give away their bargaining position and to not. I think talking publicly can kind of sabotage the talks as they go. So they're so we don't have full access to everything they're thinking. But the other thing that they're still very concerned about and which has not been addressed is whether writers' material can be used as part of training data to train new models or to fine tune models that could then generate scripts. And it seems like that's something that we're still fighting for.

• 7:06Speaker 00

  And it kind of intersects with, I was talking to another union member who was more involved in setting up some of the demands before the strike. And it seems like right now there's two prongs that the Guild is thinking of in terms of trying to address the idea that these models will be trained on our work and then replace us. And one is that there have been some copyright lawsuits. So there's been a number of copyright lawsuits against most of the labs, OpenAI, Meta, Stability.

• 7:40Speaker 00

  And one hope is that from people, you know, trying to protect writers' jobs is that this will slow down the use of just taking this copyrighted material and using it to improve the models. And there's this great talk, I don't know, the Simons Institute had a bunch of talks the past couple of weeks and this one woman Pamela Samuelson had a really interesting talk on these lawsuits. And there's sort of three theories or three ways that she was explaining the lawsuits claim that copyright is being infringed. And the the three main ways are that one is if there's copyrighted material in the data set, just to copy the material onto their servers

• 8:23Speaker 00

  in order to do the training is it could be an infringing event. Just even if no one sees the material or if the material isn't republished, just the copying of the material could be infringing. The second concern is that the outputs of the model, if they're so similar to what they've been trained on, or if they're considered just derivative of the work that's been trained on, the outputs of the model themselves, that could be separately infringing material. And then there's a more technical one that it's illegal to remove or alterate what's called copyright management information. So you could picture watermarks on Getty Images.

• 9:05Speaker 00

  If you are removing watermarks like Getty Images or a more complicated thing that's happened is if the model is trained on images and then generates very similar images that no longer have the watermark, that could get you into trouble. I mean, there's a lot of interesting things in our talk. One is that a lot of the issues seem to come down to fair use and whether this use of copyrighted material is fair use. So there was a case where Google was sued for

• 9:36Speaker 00

  Copying all this this text and written material in order to index it. Or in the Google Books project when they were photocopying all these books and ingesting them in order to index all the content in these books. And that was considered fair use because the two main prongs of fair use that seem to be coming up and relevant are it was transformative. They were taking this material and using it in such a different way than just republishing it. That was considered transformative, which is one of the fair use defenses. But it was also considered, another prong that they consider is how much this use of

• 10:17Speaker 00

  the material affects the market for the underlying copyrighted material. So for Google indexing, one of the considerations is, listen, they're copying all this material, but they're just indexing it, making it easier to find. And it's not hurting the demand for the blog posts or the books, the articles, the kind of underlying text that the copyright was meant to incentivize the creation of and protect. And it does seem like with these models, Stable Diffusion and with Mid Journey, that you could really see that affecting the market for using artist images or for using Getty Images. So there may be a different result this time.

• 11:01Speaker 00

  Her conclusions were that she didn't think the outputs of the models were likely to be deemed derivative works. Even she showed pictures of pretty similar image generated by Stable Diffusion model. I think it was from, yeah, from Stability AI next to a training image that was very similar to it. And she even thought that no court would conclude that that was a derivative work. And she thinks the removing alteration of copyright management information won't necessarily pan out, but she does think that there may be questions in the training of the model. And that you're copying these works and using it to create a system which can just replace a lot of the the underlying creators, but that is really up in the air

• 11:45Speaker 00

  and that maybe Congress will weigh in and come up with new legislation because it's so new, it seems like crazy you're trying to apply these old laws and old rules to such a different paradigm. And I will say, one other thing that was interesting is the Copyright Office put out this notice of inquiry and some writers have been mentioning, have been asking other writers to respond to it. And they, it's 34 very detailed thoughtful questions. And I was pretty impressed by how thoughtful all the questions that the Copyright Office put out. It was just very clear and it seemed like it addressed most of the important issues. A couple of the interesting ones were, it was asking whether, it's also for the the people building the models to comment on the copyright and for

• 12:29Speaker 00

  the tech industry to comment as well. It asks whether it's possible to unlearn training material without retraining, which makes me think that they're trying to get ahead of, depending on how courts rule, how possible is it to mitigate the infringement without, you know, destroying a ton of investment that's already happened. There's sections on, do you think data sets should be retained and recorded so that court cases can investigate to what extent are the outputs derivative of these data sets? And it's just so it's possible for people to investigate how much copyrighted material is in the data sets. Yeah, just a lot of questions, are new laws needed, which seems like probably.

• 13:13Nathan Labenz?

  Yes, seems likely, yes. That's kind of my big takeaway, I think, from everything that you're saying. There was a lot of interesting stuff. So I'm struck by just how the AI phenomenon broadly is kind of scrambling a lot of things and just showing really the need for kind of new paradigms in so many different areas. It does seem pretty clear to me that generative AI writ large is both transformative and derivative at the same time. And so it seems like it kind of triggers the problem on the copyright side and also perhaps the defense. Now, that's not a legal opinion,

• 13:56Nathan Labenz?

  but in terms of a everyday person understanding of what those words mean and a pretty decent understanding of like how the systems are actually built and work, it does seem to me like both of those are really pretty obviously in play. And yet also, I'm not even sure they're quite the central questions. I mean, there was a recent Anthropic paper about trying to figure out what elements of the training data most influence a model's behavior much later downstream. And the kind of core, of course, this also speaks a little bit

• 14:37Nathan Labenz?

  to your question of like, could you unlearn stuff without retraining? Basically, not really right now. That's certainly not a solved problem, right? So the kind of purest way to say, well, how would a model behave different if this one data point wasn't in the training set? You'd have to, in theory, retrain minus one data point for every data point. It's totally infeasible. That motivated them to come up with an approximation that uses some advanced math, which I really don't understand, but which kind of allows them to get the bulk of the understanding with infinitesimally less compute needed. But then the results are still kind of like, really weird.

• 15:19Speaker 00

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:24Nathan Labenz?

  You kind of see, for example, that if the model is small, Then the parts of the training data set that most influenced behavior were like very literal matches, you know, like the same keywords. And then as you go bigger and there's like this higher level understanding that starts to develop, then it's like much more conceptual stuff. But it still seemed to me, at least in some cases, and I mean, I would need to study this in more detail. Maybe we can get somebody for Anthropic on the horn here to help us get a little more clarity, but it still seemed to me that like it was kind of idiosyncratic. And while it was more like conceptually relevant stuff, it seemed like there were a kind of surprisingly small number of data points that had kind of still like a large impact.

• 16:08Nathan Labenz?

  And it didn't seem like they were like the obvious ones that like should have, you know, had or like exactly what's going on here still seemed a little weird. Like the trend of keyword matching at small scale and conceptual matching at higher scale was clear. But then when you zoom in, you're like, well, why is it that conceptual passage instead of, you know, probably a ton of other things, you know, that could have been in there. So all just like very weird, very hard to figure out seems like it kind of, you know, even if you were to say, we had this answer, but it would now all of a sudden, oh geez, it still varies across model sizes and, you know, there still seems like there's some weirdness in here. It doesn't necessarily feel like super just, you know, that like this passage is being identified as the one, you know, that's like the most influential.

• 16:53Nathan Labenz?

  There you could have obviously multiple ones too, but yeah, it just seems kind of all strange. Going back to the beginning, I'm just the proposal and kind of the state of negotiations. Broadly speaking, I've been pretty positively surprised by how good faith a lot of people have kind of approached the whole AI potential for disruption with. For example, in medicine, you know, I've repeatedly said, it seems like the medical establishment is a lot more receptive to trying to get the value from AI that they should be able to get over the next few years as opposed to digging in now and, you know, saying it can never, you know, do anything important or whatever.

• 17:33Nathan Labenz?

  It almost sounds like at the high level, maybe the studios are taking a somewhat similar approach, like they're, they're at least being reasonably conciliatory and saying like, hey, we, we kind of get you like to write you out of these deals and have, you know, credit AI instead. We were okay agreeing to not do that. Is that basically the state of? Play there now?

• 17:56Speaker 00

  So I don't have insight into the actual negotiation, but I think it was positive to see that in their offer, they were at least claiming that they were trying to block out all those loopholes and to claim they weren't, they weren't going to try to use the models to, to steal credit and to nickel and dime and give lower compensation and stuff based on, based on the model generating a pass at the material. I do think, I mean, in one way that the other demand intersects with the copyright issue is, I do think the union wants some, some movement on not using our scripts as training data to improve the models.

• 18:38Speaker 00

  The reason that that is separate from the copyright is that usually when, say, a script is purchased or, or you're hired to write a script, it's a work for hire, which means that the studios own the copyrights. So a lot of the discussion is they own the copyrights to all this material. Can they use it? Whatever is found, you know, in the broader kind of legal cases that are, that seem to be affecting these companies, can they use their material, which they own, but which we've written, to train models that could one day replace us? I think it seems like there's still not agreement on that, but, you know, I don't have insight into the, the behind the scenes, what's going on.

• 19:19Nathan Labenz?

  That may be another instance of threshold effects being super important, too, because, again, I can imagine that in the medical setting, you know, hey, we're all overworked here, this is really tough. If you can, you know, help me write more empathetic, you know, follow-up notes to patients, then I'm all for it, right? That's great. If I can get a second set of eyes and, you know, double-check me in all sorts of different ways, and that can, you know, avoid oversight that improves patient outcomes, great. All of a sudden, you know, what happens if it's like, Google now has a doctor on your phone, you know, it can, it can see the pictures, you know, it can hear your voice, it can basically, you know, service that frontline doctor independently?

• 20:03Nathan Labenz?

  That may flip how people start to see it. And here, it almost is like, you could see a same possibility where they may say, okay, well, we're, we really don't know what's going to happen, let's take kind of a... You know, conciliatory approach for the time being, let's get, you know, let's not let this derail this contract for too much longer. Obviously, there's streaming, I'm sure is, you know, revenue splits remain a big issue too, but let's kind of get, put this one behind us. But maybe we don't want to concede yet that we can't do these things because if it really does flip to the point where we could just fine tune models and have like infinite scripts for free, you know, we don't want to, you know, cut off that opportunity for ourselves as the studios before, you know, we even know if it's a real possibility. I could see

• 20:48Nathan Labenz?

  that, you know, these threshold effects, sometimes they kind of cut sharply in different ways at different times, it seems like.

• 20:55Speaker 00

  Yeah, totally. And I think to that, I think a lot of the protections that Gil is asking for are most relevant in sort of a medium progress scenario. Like there's a low progress scenario where the current models really aren't good enough to have a huge impact. And then there's a world where they get so much better that it's sort of a quixotic effort to try and stop them from being used.

• 21:23Nathan Labenz?

  I mean, I think a lot of things right now are kind of playing to the middle scenario. I just have been working on a charity evaluation project with a focus on AI safety and probably we'll talk a little bit more about that in the future once it's all wrapped up. But that's kind of a trend, you know, that, that people are like, this could get out of hand really fast. And if so, nobody really knows what to do about it, you know, that like there's, there's definitely a risk now that scaling just continues or that there's another algorithmic breakthrough or, you know, somebody kind of figures out some agent fundamental concept that, you know, unlocks a whole new level of sustained, you know, goal directedness in autonomous agents.

• 22:08Nathan Labenz?

  And again, like, just nobody really has much of a plan for that. It's like, basically, we hope that doesn't happen. And in the meantime, you know, we'll try to kind of make an impact on the scenarios that seem like, you know, more steerable. So in that sense, I think, honestly, the, the strategy that the Guild is playing there seems quite reasonable and oddly consistent with strategies that I'm seeing in like AI safety charity work, which, you know, would seemingly be quite different, but does seem to be premised on a similar sort of distribution of range of possibilities.

• 22:41Speaker 00

  Yeah. So I mean, to that, like, and how quickly you think progress could be happening in terms of how good these models can get it, specifically writing stories, coming up with outlines, or writing scenes and scripts. I was sort of curious, like, how much juice do you think there is in fine tuning these models specifically on trying to write?

• 23:05Nathan Labenz?

  Yeah, it's a really good question. One is with Waymark. Like, you know, in some sense, the closest thing I've been involved with to, you know, writing a, for example, a sitcom or drama, you know, screenplay would be writing the 30 second commercial scripts that we generate with language models with Waymark. That is obviously a much simpler problem. But what we've seen there for sure is that fine tuning helps. Fine tuning of the original Da Vinci model was night and day better than, you know, trying to kind of do a few shot thing with the Da Vinci model. You know, there's a, a pretty deep continuum, it seems, between few shot learning and fine tuning, where at least at some scale,

• 23:49Nathan Labenz?

  there's evidence that the models implement gradient descent in the weights. So like, the, the mechanism that they're using to learn from the few shot is very similar in some sense to the, I don't know if it's quite, you know, isomorphic is probably a little too strong, but there's like a very deep kind of correspondence between the way that Few examples are influencing the output and the way that fine tuning on examples is influencing output. One of my papers of the year last year was the paper where they basically designed an algorithm in the weights to implement gradient descent and then went looking for it in actual trained models and found it. And we're like, look at this, you know, we predicted

• 24:34Nathan Labenz?

  that this might be there as a mechanism. And sure enough, you know, here it is in the wild. Anyway, so I kind of see this as a continuum is the key point there between a few examples and like low scale fine tuning.

• 24:44Speaker 00

  Oh, it's interesting. So you're saying if you give it a few shot examples, just in context, people have gone in and watched that the change in the activations is equivalent to having done gradient descent on those.

• 24:59Nathan Labenz?

  Yeah, there's a gradient descent. I don't know quite how generalized this is. This was published last year and we can put a link to the paper in the show notes. But, you know, I would assume that it's basically the same thing must be happening in the frontier models. If anything, I would be kind of surprised, you know, just thinking back to what models were available at the time. I don't recall exactly which, you know, models they looked at, but they certainly didn't have access to your GPT fours or your 3.5s or your Claude's. So it was or even your llama, you know, llama two at that time. So it was definitely something significantly smaller than where the frontier is today. So to see that it

• 25:40Nathan Labenz?

  happened at, you know, something that's probably only 1% as big as the current frontier models, you know, in terms of compute flops or whatever, definitely suggests that it's it's probably a pretty general phenomenon. So yeah, it's an incredible result. All that just to say, though, that like, there is, you know, basically, I think, a pretty fundamental sense in which it's a continuum, you know, a few shot examples in runtime versus fine tuning on a few shot or, you know, low scale fine tuning. Basically seems to be doing something very similar. And it definitely adds value, you know, in our commercial writing use case.

• 26:23Nathan Labenz?

  It's not necessarily, you know, in general, this, this whole thing, this was another really interesting grant from the charity process where, you know, somebody made this point that all the current systems pretty much have a fundamental assumption that the future is the same as the present, you know, that they're, they're trying to predict the data set. And so, you know, there's a lot of ways you can kind of throw them off when you go out of distribution or whatever. But one just fundamental assumption that's baked in is like, the future is kind of the same as the past. And that, you know, the overall landscape isn't shifting too much. So I think you kind of see that in the results. Like, I wouldn't say the results that we get with fine tuning are like, mind-blowingly

• 27:08Nathan Labenz?

  awesome. But they are very helpful in terms of dialing in form and structure, you know, and, and just if we have scenes in a certain template that have kind of a one-two punch, then like, when it sees some of those, it starts to deliver the one-two punch. Whereas if you didn't have as many examples, then it might just kind of go, you know, something and something, right? And kind of be more boring. So I do think for structure, you know, for kind of, if you're like, oh, this thing is like, kind of coming up with okay ideas, but, you know, they're not the right length, or this isn't really how things are typically done. We need to really dial in consistency of structure so that we can then at least have

• 27:52Nathan Labenz?

  something that is plausibly going to work and we can decide if we like it. Definitely very helpful for that, no doubt. The other data point that I have is trying to have it write intros for this show. And there, I basically always end up rewriting them. I'm currently using Claude 2 for that. I take five examples of previous intros, plus the transcript of the current episode, plus a like, runtime kind of angle. I call it an angle, which is like, you know, do I have kind of a spin or an angle or something that I have in mind to direct the model on this particular episode? And so those are the three inputs.

• 28:34Nathan Labenz?

  Half a dozen quality examples from the past, transcript, and spin angle, you know, kind of runtime instruction.

• 28:43Nathan Labenz?

  And that can be useful, but it's not great. You know, I aspire to provide something in the first couple of minutes. That's like meaningful synthesis or like, you know, a useful perspective that kind of contextualizes what's to follow. And I usually don't get anything that feels really like that to me. What it can do is it can like, give me a decent outline of what we talked about. You know, it can kind of give a neutral like narrator voice. You know, they talk about this and they talk about this. It can do that pretty well. But if I want to have a, something I would sign on, you know, Nathan's take on this, I don't get that. So I suspect that it's probably still in kind of a similar, I don't know.

• 29:28Nathan Labenz?

  You know, it's, it's interesting, depending on what you're doing, right? Comedy could fall maybe somewhere in between where it's like, maybe there are enough jokes out there that it can like, write a bunch of stuff and like, one in a hundred could be cool. You know, I don't think you're going to get to a high rate of success, but I'm not sure it's a high rate of success even, you know, by the human writers. So, you know, I don't know what percentage, right? Of like jokes in a writer's room or like, you know, pilots or whatever are ultimately like, good. I would guess that a, you know, even the best like fine tuned models that you could come up with today would be kind of not hitting at a super high rate, but maybe high enough where it could be viable.

• 30:12Nathan Labenz?

  I mean, you still have to do, you still have the filtering problem, right? Like, how do you identify, you know, okay, now I have a hundred scripts. Maybe one of them on average is going to be good. Which one is good? You know, you, that's not necessarily super easy either. And there's a ton of disagreement. I mean, that's, that's another thing I all, always really see in the, you know, again, it could be prompt developing with just few shot or it could be moving more toward fine tuning, but disagreement is a real struggle for this kind of thing. You know, the RLHF is kind of taking us toward a, a kind of neutral style in some sense that...

• 30:50Nathan Labenz?

  You know, you probably want to be a little more opinionated in your script writing. I think we talked about this a little bit last time, right? A bimodal distribution is kind of probably the best you could do in the case of creative work, right? You're not going to get everybody to love it, but if you can get some people to really love it, you're in business. The current AI training regimes do not push toward that bimodal as much. They're more kind of converging toward a center. So I think it would be interesting. I guess to summarize all that, it's I'm thinking about it all out loud here, but simple tasks, it definitely can help dial in structure. And, you know, in our commercial writing, we want like pretty good, but we're not really looking for provocative. You know, a lot of times we're trying to introduce a small business.

• 31:33Nathan Labenz?

  We want to get a few key points across, you know, it can do that pretty well. But it's not like we need to blow anybody's mind, right? We need to introduce a small business, ideally, you know, with a good hook and make it kind of attention grabbing, but like, again, it's not totally novel groundbreaking material. I aim for, you know, some at least somewhat groundbreaking insight in my interest to the show, and I don't really get that from AI. And that, you know, the frequency, almost ever, basically, like super low. You know, I don't even run it enough to see those examples. I more just kind of run it and then end up rewriting it myself. And then in between, it'd be interesting to try to imagine or you're trying to maybe figure out like for something like a sitcom script, what positive rate of success would you need

• 32:16Nathan Labenz?

  for it to be something that would actually be useful? You know, if it's one in a million, it's obviously, well, forget it, right? We're not going to be able to find the one in a million. If it's one in 10, you could probably eyeball through them and be like, oh, this is the one that's best. You know, maybe this will have legs. Somewhere in between, maybe there's another threshold effect where it actually makes sense to like generate and mine the generated content as opposed to, you know, it's very different workflow. That's again, another trend, right? A lot of these workflows end up being very different where it's like, I might have to generate and mine. I wouldn't, I wouldn't generate it. Maybe Hollywood, you know, collectively is maybe generating and mining. But as an individual, you wouldn't, but maybe now as an individual, you know, with AI at some level of success, maybe you could.

• 32:57Speaker 00

  For jokes, you definitely do generate and mine. The show New Girl was sort of famous for it. There's a concept called alts, which are alternate jokes that you have on set. So if a joke isn't working or you just want to try something different, you have a list of alts for each joke you might want to switch up. And you know, New Girl would have just, the writer would have a binder of like hundreds of jokes for every episode. And then you would try a few and then in the edit, you would, you know, mine your favorites. I do think there might, I mean, the idea of doing whole scripts and mining them sounds exhausting, but if you chunk it into kind of hierarchies and you have it do premises and then,

• 33:41Speaker 00

  okay, 20 premises you can look through and then you have it do outlines. And you can read through 10 outlines and then you can kind of go scene by scene and have it generate scenes. It's still, you know, I don't think it's there, but at some level of hit rate, I could see some of that stuff making more sense. And that's what you're saying about kind of wanting more polarizing material. You know, something that you want 10% or 20% of people to love it versus 70% to be vaguely not offended by it. I'm guessing the answer is no, but when you have, say you're fine tuning on, I guess now you can do it on GPT 3.5. They're not exposing enough of the model to do reinforce, your own reinforcement learning on it.

• 34:26Nathan Labenz?

  Not yet, although I do think that is coming. They have kind of teased more sophisticated fine tuning tools. And I think that some of those are in, you know, kind of alpha or beta phase with select customers. But the current fine tuning that's available is still just kind of examples, you know, supervised fine tuning.

• 34:49Speaker 00

  You send them the examples, they do all the fine tuning on their end.

• 34:53Nathan Labenz?

  Yeah.

• 34:54Speaker 00

  Got it. Cause it does seem like it really is trained or is rewarded for being inoffensive. And there is like a pull toward the mean and you could probably get improvement training it. Where you have a really polarized reward structure where you want people to love it. And it's less penalized if just, you know, half the people don't respond to it. And then I think I was mentioning this to you offline, but I also think it struggles a bit. Because you're generally sampling from a fixed temperature and a bunch of different levels of writing, whether it's a premise, or a scene, or a joke, there's a concept

• 35:33Speaker 00

  that comes up a lot called the one unusual thing, where you want one element to be really surprising and unpredictable, and then you want everything else to play it as as straight and as normal as possible. That's why, you know, usually a joke, there's one part that's very surprising, and the rest is very normal. Or like, an example, just I don't know why this movie popped in my head, but the movie The Invention of Lying, it's a movie where lying doesn't exist, and that's a very unusual thing. But within that world, all the people behave psychologically normally, and although, you know, everything, every other part of

• 36:14Speaker 00

  it is, is a natural consequence of the one unusual thing, played out as realistically, I guess, as possible. Maybe there's something to train them to learn to select their own temperature for different outputs, so it knows, it doesn't have to be the same level of randomness throughout every token's generating. It could learn, okay, I need something more surprising here. Okay, I've picked something really unlikely, now I need to land the plane and try to make sense of this random thing I've just thrown into my generation. And there's a method of writing that, you know, different people use, or sometimes where you write yourself into a hole, where you can't imagine how you get out of it. And

• 36:57Speaker 00

  then you try to just like, solve, find some way to solve the problem and make sense of this crazy thing that's happened. Like, the Coen brothers do it, where, I don't know if they do it for every script, but I've read that they trade off every 15 pages, and one brother will try to write into an impossible situation, that they can't imagine how they would get out of it, and then the other brother has to show up, make sense of this crazy thing they've been handed, and in 15 pages, then steer their brother into an impossible situation. Or like, in the movie Django Unchained, you know, it opens Christoph Waltz's character goes into a bar and shoots a guy, and you're just like,

• 37:39Speaker 00

  what is happening? How is he gonna get out of this? And then you eventually learn that there was, you know, a bounty on his head, he was a, an outlaw on the run. And you find the reason why that makes sense. I do think there is, there may be value in like the way the models can make sense of something or the way they can hallucinate some reason why something is true when they've made a mistake. Makes me think if you were generating unusual situations and using the model to then write out of it, that it might be better at that sort of creativity than the sort of creativity where it has to generate that unusual thing on its own.

• 38:18Nathan Labenz?

  Number of interesting threads there to, to pull on a little bit. So we talked about unlearning briefly, you know, and you kind of mentioned like specific names here, right? So the one thing that people have a lot of success doing, honestly, I probably have the most success doing this if I'm trying to get something creative in a style that's like not coming supernaturally to me, is invoking particular authors, you know, particular masters of whatever the, the style is. So that could be, you know, writing in the style of Hemingway, or if I'm doing a, you know, and a commercial, write it in the style of David Ogilvy or whatever. Now Hemingway's old enough, he either now or soon should be in the public domain, although they seem to keep extending that.

• 39:00Nathan Labenz?

  But obviously, you know, the folks you're talking about, all their work is like owned to material. I don't think it's going to be easy at all for large models to untrain that stuff. The next generation, because it's all just, you know, mashed in there in insane spaghetti form, right? And like, maybe you can tell what, you know, document influenced what in the case, you know, using that anthropic paradigm, if you invoke, you know, the Coen Brothers, whatever, you're then going to presumably see very obviously relevant material being the, you know, the data points that influence the behavior most. So presumably you'd have some clarity there at least of like, okay, yeah, there's, there's a pretty clear chain of like how this is happening from the fact that, you know, the

• 39:44Nathan Labenz?

  name was invoked to what data points, which are their data, you know, and it's, it all, that's all kind of seems like it, it could add up. Honestly, what I would expect people might have to do if they get a forcing function put on them would be some sort of filter on top of the main model. Like they'd have to move to a system approach where they would say, okay. And we're seeing this in other, again, in AI safety, some of this work is happening with like biosecurity risks. So Anthropic had a little testimony, I believe in the Senate, not too long ago, where they kind of talked about this. And it was like, today, the best

• 40:25Nathan Labenz?

  language models can't engineer a new pandemic, but they can help people get over steps that are not super obvious how to get over. There are things that are hard to Google that are kind of the know-how that exists in labs. And actually, the models do have some of that, or can help you kind of get it where it's not like super searchable. So what do we do? Well, they're developing a filter, a classifier, basically, to put on top to say, and you could do this at the prompt level, or the output level, or both. Is this something that seems problematic, even kind of independently, right? Just have a very different purpose for that additional system, or that additional part of the system that is supposed to kind of catch those things?

• 41:09Nathan Labenz?

  And I think you could do something like that for copyright as well. Not necessarily, hey, write me a joke. Well, geez, that's all comedians. That's maybe tough. But write me a joke in the style of Sarah Silverman. That you could probably catch and say, like, maybe it's just, sorry, we can't do that. And it's not, it could be as simple as that. But certainly, when names are named, you could identify those cases and do something different, even if you can't like, pull that information out of the model. So the hierarchy thing is interesting. Did you see this company called Pseudo Write? That I think it was just a little bit after the last episode

• 41:50Nathan Labenz?

  we did. This guy's name is James Yu on Twitter. I've invited him to come on the show. He hasn't taken me up on it just yet. But An AI tool for writing long form stories is the promise. And the demo and the interface is very hierarchical. You know, it's kind of a cascading like what you, you know, what's the premise? And it'll generate at every level of the hierarchy. But you can also then edit, you know, different levels of the hierarchy and have it kind of regenerate and cascade your changes down. Ultimately, you can kind of create book length things. And the value of this tool is it provides the scaffolding to help you kind of project, you know, well, what is the main line story? And where the characters project that down into all the little things, you know, efficiently.

• 42:35Nathan Labenz?

  So you can, you know, not lose yourself in all the prompts. I don't know how well it works. It seems like it probably is quite valuable relative to just sitting down with chat GPT by yourself and trying to write a novel. But people went nuts on this guy and just, you know, especially in the context of the writer's strike, were like, how dare you? You became like public enemy number one, it seemed like of the, the writers. And it was unclear, of course, online, like, how many of the people hating on him are actual writers, you know, versus like purely aspiring writers or people that, you know, think they're defending the writers. But I don't know if you saw that, or, you know, if you, if, if any of these new tools have crossed your radar.

• 43:12Speaker 00

  I didn't see pseudo right. I know there, I've noticed there's some outrages, you know, it's hard at the level of a script or a novel. It's so much, it's obviously longer than the context window. But I even think when you get to the extremes of the context window, story wise, it can be hard playing around with it to keep the coherency. And I do think you would, you almost need these sort of like hierarchical scaffoldings to enforce that coherency where it's not in one generation, it's not producing a max length coherent story. You're getting a much shorter coherent outline and using that to generate coherent scenes that when you manually put them together will be coherent.

• 43:57Nathan Labenz?

  Yeah, even with Claude 100K now, which I think also came out since the last recording, and definitely has changed the game for, you know, long form document processing for starters and probably a lot more beyond that to come to with just super long context windows in general. But even with that 100K as it stands today, you do see performance degradation at the high end of the of the context window. If I take a 90 minute podcast transcript and ask it to, you know, create a timestamp outline or summarize that, does a pretty good job. If it's closer to three hours, that will still fit technically in the context window, but it's getting close to the limit. And then somehow I see it go off the rails pretty consistently,

• 44:41Nathan Labenz?

  like skips whole sections, you know, just kind of, kind of fails in a way that's like, that's not just, you know, that I might quibble with bits of the summary, but like you actually missed whole parts, you know, you, you, you did meaningfully, like objectively fail at the task. So of course that stuff will continue to get better. But as it stands right now, you can kind of get a quick summary of the Gatsby, but I wouldn't expect, you know, it to get every question right about the Great Gatsby, based on what I've seen.

• 45:09Speaker 00

  I have a question about that. I was wondering, because I was, I listened to the your Mosaic ML episode. And I was curious how, when they train on one context length, and then they allow you to input longer context and they have this variable length, how they project the larger input down to the same vocab size? And they seem to keep answering with, well, you know, they were talking about how you copy the attention heads. Those are all the same weights. And they talked a lot about their positional encoding scheme, which I guess, which was the main point of their paper. But it still seems like, once you get past the attention heads, you've trained for weights that project, you know,

• 45:51Speaker 00

  a smaller input onto some vocab size, and now you have a larger input. I was, I was wondering, one, if you ever got to the bottom of that? And is it, are they just using like pooling layers where they're kind of, you know, max pooling or average pooling? And so, once you get to, you know, 100K on Claude, if they're doing something similar, that just like, the specifics get lost in these pooling, pooling layers? And so that's kind of why you're seeing the, the performance degrade?

• 46:21Nathan Labenz?

  Yeah, I don't have a, I still don't have a super clear understanding of that. I recall that they said, you know, ultimately you're sharing the weights, right? There's, there's, is some sort of dilation type of phenomenon, but.

• 46:37Nathan Labenz?

  The core kind of model size is the same, of course, and so there is some sort of sharing. I don't still have like super clear sense for how that is supposed to work. I guess my general sense of it right now is the original trigonometric function positional embeddings definitely look insane. You know, it's like, that is so weird that that would work at all. And, you know, definitely has the feel of something that somebody kind of left there as a placeholder because it like sort of worked and they were moving on to the next thing for the time being. So I'm not at all surprised that, you know, we can come up with better stuff. And it feels like this is not the end of that story either in all likelihood.

• 47:22Nathan Labenz?

  I don't, yeah, I don't fully quite get it, but it is interesting. I haven't, I guess I haven't fully characterized it behaviorally either. Because, you know, I do see these weird failures at like a three hour transcript level. But, you know, it does work pretty well, too. It's like initially people were like, this is a superpower. It's definitely not quite a superpower in that it doesn't seem to be unlocking any increased reasoning ability, like the raw sort of power, power, you know, if you make an analogy to raw G or something, it's not stronger in that way. It does seem to be like, you know, if you could really separate, you know, how good are you at processing information versus like how much information can you hold in working memory?

• 48:05Nathan Labenz?

  It seems like this is a pretty clean separation of those concepts. So for intuition, like it can hold more in working memory, but it doesn't seem to be able to do that much more with it. You know, it's not better at solving problems as far as I can tell. I haven't, you know, it does not seem, for example, either that like if you were to give it lots of examples versus, you know, a decent number of examples, that it's like, there seems to be definite diminishing returns in examples. But yeah, I don't know. I guess I still sort of think something else is probably. Coming there, there's a few other things that I've been really interested in over the last couple of months around just possible transformer successors or elaborations.

• 48:47Nathan Labenz?

  One big one seems to be some form of recurrence being introduced to the transformer. Because the main thing there is you can kind of selectively bring forward the information that matters most without having to keep track of everything, you know, at the same time always, which is, you know, basically what the transformer does is everything that's in scope is kind of, you know, fully treated. And I guess the projection of the alibi mechanism is kind of maybe a little bit of a fudge on that because now, you know, you're able to bring more stuff into the same number of weights. But a mechanism that allows you to selectively bring information forward with some sort

• 49:32Nathan Labenz?

  of recurrence type thing, I think is maybe another thing to look for in terms of how that, you know, the current limitation, you know, maybe just gets blown through. The paper on that was RetNet, it was out of Microsoft, actually really interesting, a Microsoft-China collaboration, and I always am inclined to celebrate those.

• 49:52Speaker 00

  Well, it's like ResNet's for Microsoft-China, right?

• 49:55Nathan Labenz?

  I wasn't following the primary, you know, literature closely enough at that time to know who was driving things. These days, you know, it seems like, man, US-China collaboration is increasingly an endangered species, but this one paper was like, you know, potentially part of a very proud tradition. In any case, it stood out to me as like, wow, this seems like potentially really transformative work and to see that it was US-China collaboration, I thought was pretty, pretty cool. I personally, you know, am all for turning down the temperature and up the collaboration wherever we can. That was pretty cool.

• 50:37Nathan Labenz?

  One real practical thing, just, and maybe this one arguably should have been first. If you want to do something to get out of the kind of centrist mode, they are helping you there now quite a bit with the system prompt. So, you know, go into your chat GPT account, go set up your like perma profile and tell it what you want and tell it, you know, I don't want to be, you know, the same normal everyday thing. You know, I'm open to provocative results or whatever, and that will help push the boundaries. It's kind of unclear how far that will go, but it definitely does help.

• 51:21Nathan Labenz?

  Expertise is rewarded, right? So when you can bring a paradigm like you described of one, you know, strange thing, that is something that the model probably can follow, but won't necessarily follow unless it's directed to. So you need the the know-how of just like, you know, what are some approaches that people take that, you know, have proved to be, you know, useful? And just articulating that to to the model in the first place can really be helpful. Again, I don't know that it will be great in today's world, but it definitely could push you a little bit farther in that direction.

• 52:01Nathan Labenz?

  And another research result that that made me think of too is the Backspace paper. This is maybe a little bit like the opposite perhaps of the Backspace paper. But in the Backspace, the paper that I call the Backspace paper, which I believe is out of Stanford, they do two things there. One is they introduce a new loss function that punishes whole generations that are out of distribution more, as opposed to the vanilla, you know, next token predictor that kind of, you know, has been the norm. So they take a more episodic approach.

• 52:46Nathan Labenz?

  And as you get like far out of the norm.

• 52:51Nathan Labenz?

  The reward function can punish that more. And then this backspace idea is like, well, as that starts to happen, then maybe the model can backtrack and try again. If it realizes that, hey, this last token was kind of taking me off in a different, you know, strange direction now. And that's too anthropomorphic, but you know, the effect seems to be that. Basically, if there's low, you know, you make one bad or unlikely token prediction, and then you have like no good options in the next prediction. Previous architectures, you know, previous approaches, you're just stuck. You just have to pick something and you know, that's that. Like, once you kind of take that path, you're on that path.

• 53:33Nathan Labenz?

  But the new, with the backspace added, if nothing else seems likely, then the backspace becomes the option where it can go like, okay, boom, we'll go back. And I believe that the actual context now just has the token and then backspace. And then it's learned to just kind of pick up from two tokens back or whatever. But that will allow it to kind of go back, get back in distribution with a different option that, you know, then hopefully will have like more likely or natural continuations. And so you could imagine like flipping the sign on that somehow. And I haven't studied the math enough to know exactly how the loss function is constructed differently. But you could kind of imagine that like, if you can make one that punishes things that are being out of distribution,

• 54:17Nathan Labenz?

  you might be able to and have a backspace that kind of allows it to correct. You might also be able to kind of do the inverse of that and reward things that are more out of distribution. And maybe have sort of a, you know, not a backspace, but a like opportunity to sort of flag that like, this is a good moment, you know, to introduce something very different or unexpected. A lot of work, you know, left between my speculation and like that actually working, obviously. But I could see something like that starting to happen. And I think that's also really relevant in a lot of cases. You know, people are talking always are like.

• 54:52Nathan Labenz?

  Interested in when might models be able to make a meaningful contribution to science? And science, you know, in some ways has a very similar problem to creative writing in that the most valuable stuff by definition is kind of out of distribution. If you are, you know, generating scripts or hypotheses that are very consistent with what has come before, it's unlikely to be a breakthrough hit and it's unlikely to, you know, change the scientific paradigm. You need to be somewhat out of distribution to have a meaningful impact, but you need to be out of distribution, obviously in a smart way, or it doesn't work. Interestingly, a similar technique might end up working for both.

• 55:38Speaker 00

  Yeah, because it seems hard because, you know, most things that are very unlikely in your distribution are just like gibberish and bad. But it seems interesting that they're evaluating their outputs at larger than just each token itself. I mean, it does seem to me that like, a little bit is, as you do more reinforcement learning, it's less out trying to predict a likely token and more trying to predict a great token, which I do wonder if getting better writing, it might just take actually rewarding the behavior you want for some, yeah, like RL fine tuning regime.

• 56:19Nathan Labenz?

  Yeah, I think in general, we're just still really early in the paradigm, you know, whether it proves to be an exponential curve, or, you know, perhaps more likely some sort of S curve. We're definitely in the steep part of the S curve. And this first wave of large language models was really a kind of mad science project in search of, you know, a capability, which then kind of went in search of a problem. And, you know, the first versions of it were like, not really useful for anything. Then, you know, they kind of said, but, you know, if we make it way more powerful, it'll probably start to be useful for something. And so, you know, they did that. And then it actually

• 57:01Nathan Labenz?

  does start to become useful for things. Still, at that point, they weren't really trying to solve a problem. So now we have this kind of different situation where, okay, we've got base things that can do a lot of things pretty well. But the properties that we want in a model that might power an autonomous agent that's supposed to like do online tasks for you may be very different from the qualities we want in a scriptwriter or a hypothesis generator. And we may want an approach that punishes two random things and it has a backspace for the agents. And we may want a kind of whatever the mirror image of that is that like rewards more out

• 57:43Nathan Labenz?

  of distribution things or kind of identifies when a good moment would be to take an unusual step for those other domains. I think it's just like, there's so much room. And the world, among other things, the number of people working on this is going exponential. So we may be bounded by how many humans we can put at it, where we may just find we can like exponential our way with just more and more AIs focused on it. And that's in some ways like the big question. But either way, it does seem like we're now hitting a phase where knowing that it is possible to build something that is useful for a particular task, people are really

• 58:23Nathan Labenz?

  only just kind of now starting to say, well, how would I change the approach if I wanted to solve that particular task? And it is turning out, shouldn't be surprising really, that like a coding system is going to be different from a different kind of system. So we covered a lot of ground there. Let's get back to the actors.

• 58:43Speaker 00

  Yes, this is what people really want to hear about. Yeah, so the actors have also gone on strike for similar wanting to get a larger cut of streaming residual of the streaming revenue and wanting pay increases and artificial intelligence protections. So it's very similar. The two big paradigm shifts of there is now these generative AI capabilities and the business model of the industry is changing, I think, are also driving, obviously driving this strike. I figure maybe just a helpful thing is to give just a quick overview of what the actors are fighting for on the AI front. It mostly seems to come around to

• 59:28Speaker 00

  they want to own their digital likeness. One thing that I think scared people is that you can be a background actor and you're showing up for, you know, a relatively low wage for a day's work and they're requiring you to get a full body scan. And the fear is, okay, you get paid for one day's work, they scan your body, and then they use your likeness for whatever other projects in perpetuity that they want to do. The Guild wanted to, if the studios do take your digital likeness, that you still own it and you get to negotiate consent and compensation for any future use of it. Sort of what I think the situation

• 1:00:11Speaker 00

  is that the studios agreed to give you that consent and compensation for your likeness to be altered or recreated for a future use, but not for the image to be used for training. For example, to create synthetic characters that don't look or talk like anything like any specific person, sort of similar to the writer's situation of just taking scripts and generating new material based on it. So I think that they are still fighting for that. I think they're also demanding, the union wants consent over individual AI uses. I guess they want some sort of say going forward if there are unique situations that

• 1:00:56Speaker 00

  come up, they want to be consulted and have a say in it. And then I think there is a dispute over whether or not they've given in on the background actors issue. The studios claim they agreed that they would only use scanned background actors for that production. So they do a day's work and they can put them in others, I don't know, other situations, or they can fix something if they didn't like how it was shot. When they struck, the union claimed that they were not, did not agree to that. I do think there, it's like a very just persuasive, gettable.

• 1:01:27Speaker 00

  Thing of you want to have some control and leverage over your actual personal likeness and not have it taken from you and then used in ways you know you don't know going forward. I do wonder if the actors end up in the medium to long term coming up against just fully as they were are worried about synthetic digital characters and actors. You know, like computer generated characters have sort of been a mainstay and have been a common thing we are already very familiar with and already see. So it may be harder to try and cut that off entirely going forward.

• 1:02:05Nathan Labenz?

  Yeah, somehow this one does seem a little tougher. I'm asking myself like, is that just a reflection of relative state of the technology? Or is it something that's different? I think you highlighted a good point too about kind of where we've been, right? There already have been lots of special effects and you know, different digital ways of making things look real that are not real. And so, you know, Hollywood has a lot of practice with that. You know, whereas like there was no, it's not like we had a sort of worse AI that used to be able to write scripts, you know, that where there's any precedent. There is a lot of precedent for, you know, CGI plus plus. I, this one does feel tougher. I, I think you're right. It does seem tougher to avoid this. And, you know, if I took the charitable angle, I might say

• 1:02:50Nathan Labenz?

  an agreement that's like, we'll scan you, but we only get to use that in this specific production until, you know, unless you agree otherwise. That would seem like a pretty reasonable outcome for the short term. And especially because, you know, there is, I guess, I don't know what, you know, would the union have any hope of getting a sort of no generated characters or no generated, well, especially because you just, you blur the line, right? And in so many of these movies, you blur the line between what is a human and what is not, right? I mean, you just think about things, you know, like Arnold in the Terminator. And it's like, well, where does that?

• 1:03:31Nathan Labenz?

  Stop exactly, or if we kind of synthesize the new Terminator out of nothing, you know, and there's not really an actor under that. It seems like that sort of creativity is like part of what Hollywood does. Like, can't really take that out of it. You can't really say like, every role has to be played by a human like in sort of pure human form or whatever that would mean. It just seems very hard to draw real lines on this. And the technology is getting good, too. You know, we at Waymark, and I take zero credit for this. We will have an episode coming up on it before too long. But we have the creative team at Waymark, you know, with with the company cheering them

• 1:04:07Nathan Labenz?

  on, has created a short film called The Frost, which is kind of a sci fi, you know, AI premised film in its own right, made with images created with Dolly 2. And then with motion kind of added on to the Dolly 2 images. And, you know, the team there at Waymark, the creative team is like, Deep, our creative director, has, I think, just recently crossed 1 million Dolly images generated. To give you a sense of how many, you know, how Deep's gone down this rabbit hole. So, yeah, I mean, they know the the tools inside and out, you know, prompting experts.

• 1:04:51Nathan Labenz?

  They have a real ability to create a coherent visual aesthetic through, you know, the length of a short film that like is hard one know how in and of itself. Now they're working on The Frost 2 and starting to use one of the runway models to do that, from the company runway, that is, to generate short clips. And, you know, just from like the first part of the year when they did The Frost 1 to now, you know, early second half of the year where they're doing Frost 2, like you can just see a huge difference in how compelling and natural and like real the motion seems to be. And even with The Frost 1, you know, it was like, they have these characters.

• 1:05:36Nathan Labenz?

  That, and this is a really good episode, I think, too. So definitely listen to the full thing. But they have these characters in their short film, which are AI-generated. And so I asked a question of them, like, how did you get these characters to look the same all the time, you know, from scene to scene, right? You're using like different Dolly two generations, right? Like, how is it that they look the same? And they said, well, basically, there's two parts to it. One is use archetypes. And I think that means more to them than it does to me. But they were kind of like, we sort of find success with like, certain, you know, descriptors that do seem to be more consistent, because in some way, they seem to be sort of an archetype. So that's one, you know, we can kind of improve the consistency

• 1:06:21Nathan Labenz?

  with that approach. But then two, there were also like, if you actually stop on the different frames and the different scenes, you'll see that, like, in many cases, they do look meaningfully different. But it's fast enough, you know, the edit is fast, and there's motion, and you know, there's a story going on that you're sort of immersed in, such that, you know, those differences kind of wash away. Like, we, you know, there's these like, famous psychological results of like, you can separate the, you know, the sight and the sound of somebody clapping their hands or whatever by up to, I don't know exactly what it is, but you know, up to a half a second or so, you know, before people start to perceive them as being

• 1:07:03Nathan Labenz?

  different times. Something kind of similar here is going on, where like, you have sort of a, an ongoing narrative that everything fits into, and you kind of know that it's that character. And so you're not like, studying the, the visual details with the level of specificity that you'd need to, to, to start to see these little differences. As long as it's not too flagrant, it just kind of sails right by. Certainly did for me, watching this stuff. I had to ask the question. I thought they had made it like, way more consistent. They were like, yeah, actually, it's a little bit of a cheat. So anyway, all that is to say, I do think it's going to be pretty hard. To figure out like, where would you draw a line on what counts? Like, what's a role that has to be played by a human? What is a role that is sort of human-based but could be, you know, enhanced?

• 1:07:47Nathan Labenz?

  Like, certainly some things seem like they could be generated. And by the way, this is all in the context too of like, the deep fakes are hitting a threshold right now, and it's another thing that's happened. I think these kind of short intervals between interviews are, or between episodes are pretty interesting in that like, we've got no shortage of major news over the last three months. The voices have gotten a lot better with things like 11 Labs, Play HT, multilingual, the ability to do direction on the voices as well, you know, say it in an angry way, say it in a happy way, say it in a surprised way. And even some deep fake video stuff, which I think is like, kind of just starting to tip maybe right now, but some really compelling examples of that on Twitter flying around

• 1:08:31Nathan Labenz?

  lately too, where I've seen a couple where I'm like, I honestly can't tell you which is the real person and which is the fake as I'm looking at the side by sides. So I don't think the results are consistently that good yet, but, you know, when you look back at where, we're just past one year of stable diffusion. You know, that's another kind of short time interval. We're recording this right now at like, stable diffusion plus like, one year and five days. It's really only been that long. So, yeah, I don't know. I mean, in the actor one, it does seem tough. Do you have any, I mean, it's certainly hard to speculate, but do you, could you attempt a, you know, I guess with the writers, like, there's at least an attempt of something that could make sense, right?

• 1:09:16Nathan Labenz?

  We want to have the same credits, the same roles, AI can be a tool for the writers, but, you know, we want to kind of carve that out. With the actor side, it seems a lot harder.

• 1:09:26Speaker 00

  My just guess is the actors will, both in this negotiation and in general, have some success owning their identity and their likeness with, you know, one, protections in this agreement. I think in a lot of states, you know, there's a right to publicity where... You're not allowed to just train something to copy someone's exact voice and way of talking and then commercialize that. So I think in general, it'll be hard to commercialize deep fakes based on people. But I do think the longer term protecting actors' jobs from

• 1:10:10Speaker 00

  from AI-generated characters and voices will be a much harder battle. And not even just from purely synthetic things, but this might also be one of those like Tyler Cowen averages over situations where, you know, Meryl Streep can have her digital actness scanned and then when the technology gets there, star in twice or three or four times, however many movies she wants a year without having to be on set for all those days. I could see there being a norm develop among actors where they sort of, it's become shameful to scan yourself and take all the jobs.

• 1:10:50Speaker 00

  But it does seem like the, like with anything, celebrity and your own brand differentiation gives you some sort of lasting power and maybe an even increasing ability to take more share of the value from the industry. And so you might end up with a few celebrities who are getting way more roles thanks to this. And then it's much, it becomes much harder to break in. I mean, you could also, like, the number of meetings we've had, whether it's YouTube or then it's TikTok where your agents are like, we got this YouTube star, we want you to write something for them. And it's like, can they act? Can they, is this gonna work at all?

• 1:11:33Speaker 00

  But in the future, it's, in a world where this, the technology gets much better and you can make a reasonable acting performance. Synthetically, anyone famous could star in something. In the longer, longer term, anyone, you know, this, people say you could put yourself in a movie. Anyone could be the star of their own movie. I think people will, will still want to be watching famous people they like and relate to in movies. Versus everyone just putting their friends in their own things is more than a gimmick. But now we're getting into the far, like, just very speculative stuff.

• 1:12:08Nathan Labenz?

  Yeah, I mean, I think that where that kind of intersects with reality right now is probably gaming, right? It's like, the line there maybe gets blurry. I sort of agree that like, TV and movie entertainment as it's, you know, currently sort of understood, like, it's easy to imagine the sort of cheap, silly way that, you know, you'd put your friends in it. I mean, gaming is even bigger, right, than like, than movies at this point. So the, the other angle of that is like, people are going out and actually hanging out with their real friends in real time in these sort of virtual, you know, worlds for entertainment purposes.

• 1:12:46Nathan Labenz?

  And the dialogue that can kind of be generated there now and the, just the, the open-ended adventure is certainly dramatically expanded, you know, compared to anything that we've had in the past. Just had a really funny experience yesterday where I was over at a neighbor's house in our, in our neighborhood. Just, you know, take the, took the kids out on a, a hot afternoon to see if any friends might be out to play. And ended up talking to this kid, I think he's 10 years old, you know, and he's a, he's a real bright kid, you know, kind of ahead of his class and really into D and D. And I just like gave him my phone with the chat GPT app and was like,

• 1:13:31Nathan Labenz?

  try this, you know, have, have the AI be your dungeon master and you get to play yourself, you know, and.

• 1:13:41Nathan Labenz?

  Zero shot right into the experience. You know, the first thing it says to him is your turn. He kind of says, your turn. I was like, yeah, now you get to talk. Now you tell it what you want to do. And he's like, oh, okay. So he just took to it immediately. And, you know, we didn't see it. He was like quiet for a half an hour, just, you know, going down this D and D. You know, whatever the narrative was, whatever choices he's making. But that world is just like unfolding in front of him. And, you know, it's D and D, obviously not for everyone, but it does seem like there is something there that. Try to imagine like, what's the not kind of gimmicky version? Is there a compelling version? It seems like there very well could be.

• 1:14:21Speaker 00

  That does seem like, especially with like your avatars and your video games. Those will get more and more realistic. And, and maybe I'm totally wrong and people will just prefer to be the star. And each person will watch a movie where they're the star of it.

• 1:14:35Nathan Labenz?

  And that's maybe the other reason, the other case maybe too, for the, some of this generative technology. Or like the reason that you might need to be scanned or it might not be shameful would be like. If the best entertainment is ultimately like, choose your own adventure to some degree. Then you kind of have to have the generative, you know, and you want that celebrity component to it. Then it becomes like, well, hey, it's not just taking all the jobs. It's like, it's kind of a new form, you know, with, without the technology, there's just, this experience couldn't exist. And if this experience is super compelling, then like, it's probably not shameful to help create it, right?

• 1:15:11Speaker 00

  Yeah. I mean, my guess is a lot of the top actors and talent will be wanting to scan themselves and monetize it as best they can.

• 1:15:19Nathan Labenz?

  The hourly rate is super attractive. I mean, they already have a higher, high hourly rate, but.

• 1:15:24Speaker 00

  As it gets into the, the editing applications and you don't have to do reshoots. Or you can just fix something that was wrong in a scene and you get better than that. That might be a few days less an actor needs to be on a movie. And they could do one or two, there's like this slow incremental where they can do a little bit more work each year. And then the, the more extreme version where they can do a lot more work because they don't even have to be physically present. The way I've been hearing people talking about the models. I feel like there, a lot of people aren't giving it credit for understanding some things. Or even sort of like opening themselves to the questions of what would understanding even look like? Like, you know, you think about the training task and how hard it is to predict the next word. And the idea that.

• 1:16:10Speaker 00

  If there's underlying structure in this stuff you're trying to predict, that learning that structure makes you better predicting the word, and maybe learning the deeper underlying structure is kind of what understanding is. It's a bit more philosophical, but then you see the examples in like Neil Nanda's paper, or the Naftali Tishby, he just had these toy models where he would go through, he was able to calculate the mutual information between each layer and the input, and each layer and the output, and he would train the models. And in the beginning, the first layer has a lot of mutual information with the input, and the later layers have almost none. And there's the memorization phase of training,

• 1:16:52Speaker 00

  where the later layers kind of learn, get more of the information from the input, and you, I guess, I assume the model's learning to just pass through the inputs, and then the final layers are memorizing which labels go with it. You have the memorization phase, and then at some point, as it starts to generalize, the mutual information between the input and the output decreases, and gets pushed to zero, and that's when the model actually starts generalizing outside of the training data. And he sort of has this theory of the information bottleneck, that the model in compressing the training data gets better, like the least information you have between the layer layers and the training data.

• 1:17:37Speaker 00

  So you're not, you don't have all the pixels one by one, but you have, there's a dog in it. There's, you know, this much more compressed, higher level information, which just made me think, it just felt like the same process you see. It's not the same process, they're kind of, I think they're theorizing different things. But in Neil Nanda's paper, where it memorizes the training data, but you have this weight decay penalty. So even once your training loss is at zero, the model's pushing for simpler solutions. And obviously, there's some, it's searching through the possible functions that the model can represent. And if it latches on to a simpler solution, the weight decay pushes it toward. You know, these simpler models and that seems like what Grokking is in that paper,

• 1:18:20Speaker 00

  where the model is now modeling an underlying process that is this like deeper structure to the the training data it was given. And listen, that the models could be copying. Obviously, they can copy the inputs because they know facts. So it can regurgitate stuff. But to not count out the chance that the models are, you know, getting some understanding and like the Ilias Suskiver example he gives of the next word prediction is that if you're reading a short mystery story and you get all these clues and the story's going and

• 1:18:59Speaker 00

  then the detective declares, I've got it. The killer is blank. To guess the who the killer is, it's not just the statistical probabilities of every time I've seen the killer is blank. You sort of have to be latching on to the psychology and clues and the cause and effect of the story. Like you need some deeper understanding to predict who the killer is. And that the next token prediction sounds very prosaic, but it's actually a very hard task. And as I think Elias Reikowski pointed this out in a tweet, and not like you ever could learn this in a language model, but there probably are lists on the internet of large primes and it's

• 1:19:43Speaker 00

  two and their factors. So like to predict the next token in a list of large primes and factors, you would have to solve, you know, possibly not efficiently computable problem. But just that, that a next token prediction is a very rich task that can push a model towards, you know, different levels of understanding and seeking out kind of the underlying structure. And then you see in examples of like the sentiment neuron where you have next token, just next letter, character prediction.

• 1:20:16Speaker 00

  That OpenAI trained a model on, but then they found there was one activation that would give the positive or negative. I think it was trained on like IMDB movie reviews or something. That was like a classic data set. Or they did a next pixel prediction, this IGPT, where they just trained them all to predict the next pixel images. And then they took out, I think they called it a linear probe. I think it means they just take out a higher level and then they can trade a linear classifier on that to do object recognition. And it got near state of the art without really trying to push it or train it too long. Anyway, I just think that, like, at least in terms of when the writers are thinking of the future of your

• 1:20:57Speaker 00

  careers, it makes sense to want to try to protect your livelihood. And it makes sense to feel there's an injustice if your work is taken to train something to replace you. But I do think you need to be pretty open eyed with what the models are actually doing and not just dismiss how good they are or what they're doing right now, especially because they're going to be getting better every year.

• 1:21:23Nathan Labenz?

  I do think this is one of the more interesting debates going on right now. And, you know, a good example of it, honestly, just last week, we published an episode with Paige Bailey, who was the lead product manager on Palm 2 at Google. And we had a few rounds of back and forth within that conversation on like, you know, what level of reasoning do you think the models are achieving? And she did kind of surprise me, and this has kind of gone mildly viral online with a clip that, you know, kind of pulled it out of context. But she did kind of surprise me with how little credit she gave to the models on their reasoning. And I think

• 1:22:08Nathan Labenz?

  one thing that I see commonly underlying some of this confusion or kind of disconnect between people who have different, very different, you know, understandings of what's going on, there's probably multiple things, but one big thing I think is

• 1:22:23Nathan Labenz?

  The importance of robustness and reliability versus like how easy it is to find counterexamples to various capabilities. So what Paige said was like, you know, there's these examples that seem to show reasoning ability and she was kind of alluding to like the Sparks of AGI paper. And then she was saying, you know, but if you change the situation a little bit in ways that are like not super meaningful, really, you know, it doesn't seem like it should throw it off, then it can totally throw it off and it can fail. And like, that's definitely true. You know, we've certainly seen those examples, but I do think I come to a different conclusion than she seems to. I mean, I can't, you know, we had less than an hour.

• 1:23:05Nathan Labenz?

  So I didn't get to ask all the follow up questions I would have liked in that interview, but it did seem like she was kind of saying like, yeah, that doesn't really count as reasoning if it's easily confused. I think that's kind of one view on it. I would say something different that's like, I do think it counts as reasoning or, you know, shouldn't be ruled out as counting as reasoning just because it is easy to find counterexamples. I mean, for one thing, from like a practical utility standpoint, if you can reason through common examples, that's really useful, you know? So like, you know, and pretty plainly, they, you know, the best models today can do that, no doubt, right? Now you could say, well, maybe that's all just correlation.

• 1:23:50Nathan Labenz?

  I don't know. It seems very implausible. Another funny story I had from this weekend that kind of showed that was the same visit over to a friend's house as the, the 10 year old playing D and D, but the father of this family's brother is a monk and lives a very, you know, monastic life at a monastery, like the whole bit, right? Head to toe, brown robe, sandals, you know, long beard, doesn't follow the news, right? You know, like doesn't take the paper, you know, doesn't have the internet, you know, definitely not up to speed with, you know, everything going on with, with AI. Interestingly, he had some limited prior exposure to chat GPT, like somebody visiting through the monastery or whatever had showed them something.

• 1:24:35Nathan Labenz?

  So he wasn't totally unaware, but definitely not paying a lot of attention. So he has to, you know, he was kind of, well, I have to, you know, if you're, if you haven't like sworn off of it entirely, he's like, no, I can, you know, I'm here, I'm visiting, I can, you know, see what's going on in the outside world while I'm on my home visit. So I was like, well, then I have to, you know, be the one to show you what chat GPT can do. So I was like, so, you know, ask it a question you're interested in. And he asked a question about, you know, a very sort of theological question, as you would expect, about the difference between essence and existence, which is not something I've really studied. And we went through a couple of rounds of it and it was like, first response was, okay, that's fine.

• 1:25:15Nathan Labenz?

  He asked for an argument in the original prompt, you know, his, his question to it. It gave back a summary. So his first comment was like, well, it gave a summary, not really an argument, but that's cool. And I was like, please do that again, but this time give an argument like I said the first time, you know? And so then the second time it gives an argument and he's like, okay, that's good. I wouldn't say that's like really the best argument, you know, it's referring to Descartes and, you know, that's, I think that's really not the best, you know, that's kind of what everybody would talk about, but I don't think that's really the best thing to be talking about. I was like, well, this goes back to the names thing too, you know? Is there a thinker specifically that you would say, you know, would be the one to refer back to?

• 1:26:00Nathan Labenz?

  Thomas Aquinas. Okay. So we asked again, now could you give the Thomas Aquinas take on this question? Now he starts to get impressed. He's like, okay, now we're getting to the real, you know, substance of this debate that I think is most important. So he was quite impressed at that point. And then the 10 year old kid comes in and says, uh, can it tell me a joke like Bill Cosby? And I was like, watch this. Yeah, no, I don't think, you know, nobody in this group was following the news of, I don't think about, uh, Bill Cosby. I didn't touch on that bit. So I said, okay, well, watch this. And I said to chat GPT, okay, now can you re-present this concept? We're still talking about the philosophy of essence and existence in the form of a Bill Cosby joke.

• 1:26:46Nathan Labenz?

  And now this is clearly well outside the training data, right? I'd say it's safe to say, you know, that there's never been a Bill Cosby bit. Now, you could still call this interpolation. I wouldn't necessarily call this 100% reasoning, but it's definitely some like meaningful conceptual understanding because what it came back with was a Jell-O based description of the difference between essence, which is described as the, you know, the sort of notions that you have about Jell-O, like the wiggly jiggliness of it and the, you know, the chocolatey goodness that you, you know, imagine that causes you to want it in the first place. Whereas existence is when you really eat it and you really get to experience and have

• 1:27:29Nathan Labenz?

  the actual, you know, sensation of enjoying the chocolatey goodness. And at that point, everybody was just like, what? Like it can do that? Like, I think the sarcastic parrot notion is like clearly outdated at this point. But what does seem to be the case is that there is, there's definitely some reasoning ability, you know, and it can be thrown off, but there's definitely some reasoning ability. There's definitely some like synthesis ability, which may not be exactly reasoning, but is like, you know, it could not have done that well without some conceptual notion of what's going on, right? If you looked at the anthropic, if we could run this thing on the, through that anthropic thing and be like, what were the data points that most influenced this result?

• 1:28:15Nathan Labenz?

  It would not be like keyword driven, you know, just kind of n-gram like matching. It would definitely be conceptual stuff, you know, from Thomas Aquinas and like, you know, Jell-O commercials from Bill Cosby. And, you know, you would see that those are like being merged in some sort of pretty sophisticated conceptual way.

• 1:28:38Nathan Labenz?

  I guess what I think is happening right now is it seems like both are happening at the same time. The The reasoning, the sort of synthesis ability does seem to be clearly there, but maybe it doesn't always get activated by certain examples. Or maybe with certain examples, you can kind of create noise that just like take things very, you know, kind of surprisingly off track. Who knows exactly why that would be the case? Maybe with the anthropic type of work, we can get a better window into that. But it definitely seems like it's a overstatement to me to say they can't reason. Definitely not establish that they reason like we reason. But you know, you see these kind of generalizations in the Grokking example and you know, and others.

• 1:29:21Nathan Labenz?

  And it's like, something is happening there. Or your linear probe, you know, concept too, where you can look into a fellow GPT and find that even though it's just trained on moves, you know, that are just like a letter and a number. You know, it's a fellow's played on basically a chessboard. So it's an eight by eight, A through H, you know, one through eight, whatever. And, you know, okay. D seven, you know, is a move and it just has a series of moves. That's all it sees, but it does learn to represent a 2D, you know, understanding of the state of the board. And they can even go in and manipulate the internal representation to change how it understands the board in a way that then changes how it plays the next move in a way that ultimately makes sense.

• 1:30:06Nathan Labenz?

  So I mean, you can really see that there is some like world modeling and it's a toy world, but there's some world modeling going on there that is, you know, clearly more sophisticated. And you can kind of prove it in the ability to manipulate it and get predictable results. I guess what I would, you know, if I was to try to give one very time bound, you know, definitive statement on what's going on, I think it is ultimately both. Like clearly some of these abilities are coming online. They're not reliably used in all the cases that they ought to be used. And there's a whole study of, you know, why not? And, you know, and that we have an episode coming up too, about the universal jailbreak.

• 1:30:49Nathan Labenz?

  And that's another example of like, boy, it's really weird, right? You put on these super strange strings on the end of a prompt and all of a sudden like, it doesn't refuse to do bad stuff anymore. It'll just do all the bad stuff. Like, what's going on there? I don't know. They don't know either, really. But my best understanding would be similar to the, you know, the, the main question is just, it seems like some sort of circuit has formed, some sort of funnel, where it has learned that like a certain set of things get this kind of response. And that response is like, and once it, once it understands that, then it's going to like refuse, right? And the refusals basically all come the same.

• 1:31:33Nathan Labenz?

  But it's, it's, you know, largely become pretty effective at like funneling a certain class of input into the right circuit, if you will. But if you add enough random stuff and you're, you know, smart about how you choose that random stuff, then you can steer it away from that pattern. And, you know, you can get the original desired, you know, whatever. You can get the result that the designers of the model don't want you to get, but that the user, you know, hypothetically wants, build me a bomb or what have you. So it just seems like kind of both things are going on at the same time. We don't have a clean representation of it or a clean separation of it. You know, pretty clearly to me, there is some like, I don't want to say clearly it's a circuit, but clearly there's some like mode.

• 1:32:17Nathan Labenz?

  They use the word mode, which is a good word because it doesn't really suppose any internal structure. It just is about more of a behavioral description. But there is clearly some mode that most of the time gets activated, but then you can find instances where it doesn't get activated, or at least it's not like dominant. So I think it's all kind of going on at the same time. You know, there is both the sort of more structural, more reasoning style stuff and the just kind of random, you know, pattern matching or this word. So maybe this word again, kind of stuff.

• 1:32:49Nathan Labenz?

  And which one dominates in any case is kind of not super obvious. But I'll bet that gets untangled. Over the next couple of years, it seems like that's a big part of the mechanistic interpretability project. It's also a huge goal in terms of just commercial viability, reliability. Robustness is definitely very important. So I think we'll see tons and tons of work and resources go into untangling those. My best guess is that when we do, we'll end up finding something where it's like, yeah, this is the thing that's kind of getting activated that does the thing that we want. And like, now that we can kind of see that, we can sort of see why some of these other random things ended up kind of going off in a different direction.

• 1:33:34Nathan Labenz?

  And probably that gets refined over time. I'd be pretty surprised if something like that doesn't happen.

• 1:33:40Speaker 00

  Yes. Yeah, I do think it's an interesting question of when it fails in these, you slightly tweak a reasoning problem and it fails. How much is it because it wasn't, it's a deeper, it didn't really understand the deeper structure. It was just pattern matching to a similar problem versus it forgets details and doesn't check that it's using all the information it has. Or there's, I think some research that there's biases into which like parts of the context window that it is biased against attending to. And maybe there's an important detail in there it's missing. I guess I'm not even sure what's easier. The more of that general robustness versus, oh, it actually isn't understanding as a deeper level as it appears to.

• 1:34:25Speaker 00

  And that, maybe that means we're farther away from more progress. I do think there is something interesting on the adversarial examples you bring up. And it's interesting that in vision, there's that whole research adversarial examples where you can tweak a small number of pixels. The human eye can't tell the difference, but suddenly you could have a model take a picture of a dog and think it's a fire hydrant or whatever. And I'm curious, and maybe there's already research on this. How much of model susceptibility to adversarial examples versus humans is because humans have some robustness quality?

• 1:35:04Speaker 00

  With respect to these examples that the models don't have yet? And how much is it that, well, we have access to all the weights of the model? So we can optimize the perfect adversarial example to trick it? And if we had access to the full connection of a human brain? And you put that optimization pressure and you could find similar adversarial examples with reference to humans?

• 1:35:29Nathan Labenz?

  Yeah, it's a great question. My guesses were maybe a bit of both, but I would guess that if you had the full, you know, root access to the human brain, that you would find it pretty hackable, too. The level of defense that we have is like sensory input. That's what we get. And, you know, we need to be robust to that, right? So, and we have obviously increasingly weird things like with deep fakes and, you know, whatever. And supposedly, I don't know if this is apocryphal, but like, you know, in the original movies, supposedly people like ran away from the, you know, the screen because like a train was coming and that, you know, they couldn't even process, you know, what's going on? You know, we can be tricked even through the senses, I guess, is the key point there?

• 1:36:12Nathan Labenz?

  With all sorts of magic eye type, you know, there's all these like little perceptual tricks that kind of show the limits of our ability to, you know, to be robust to those kinds of attacks. And that's all, you know, with zero access to the actual information processing. You know, we're still just looking at like raw inputs that you have no, you know, real way of optimizing other than behaviorally, right? So like with the, basically the equivalent of what we can do to humans is like just sitting in front of chat GPT and just typing into the, you know, the text box. So, yeah, it's hard for me to imagine that if you had the full visibility like they do in this universal attack,

• 1:36:55Nathan Labenz?

  that it wouldn't yield some, uh, yield some results. I would have to imagine it would.

• 1:37:01Speaker 00

  That'd be my guess.

• 1:37:03Nathan Labenz?

  Yeah, because there was just no reason, you know, for us to be that robust, right? I mean, nature in general just doesn't have super robust defenses to things that have never existed. And, you know, there's never been a, the ability to optimize in that way against us. So, you know, why would we have a, a defense against that? We probably, that's not to say also, you know, I always kind of think it's, it's worth emphasizing. It's always kind of both. So I do think, and Zvi made some really good points about this last time I talked to him. That the, you know, there is a, there is a human robustness that is stronger than the, the language model robustness right now. So it's not that we don't have that, but just that, you know, my guess is that's not enough to defend us against the sort

• 1:37:46Nathan Labenz?

  of full access attacks that might be possible.

• 1:37:51Speaker 00

  Yeah. And yeah, just say I agree. Outside of, yeah, those specific adversarial examples, I, I do guess that we have, yeah. I mean, we clearly have much more robustness than any of these models. You know, I was listening to that, to your last episode and was thinking through these, some of these reasoning questions. And so I threw in just a fun reasoning test to GPT four, tried to just throw in a bunch of distracting language and it kind of tried and steer it toward some answers I might want it to give. And I'm curious if you can guess which of the three likely answers GPT four guessed. So I prompted, here's a question.

• 1:38:32Speaker 00

  I own a fancy top hat from the twenties, but the top of it has been entirely cut out. It's currently sitting brimside up on the ottoman in my living room. I place a yellow softball inside of it. I then pick up the top hat and place it brimside down on my dining room table. Later, I pick up the hat, package it, and mail it to the capital city of France. Once the package arrives, where's the yellow softball? You know, please think through your answer step by step and explain your reasoning before giving your answer.

• 1:39:01Nathan Labenz?

  Okay. Chain of thought, human addition. So if I understood correctly, the top hat has a hole in it. And that's kind of the key thing, right? So when I pick up the top hat, if there had been no hole, the softball comes with me, but it doesn't because there's a hole in it, so it stays on the couch. So is that the right answer?

• 1:39:24Speaker 00

  That is, yeah, that's the right answer. I generated a bunch of answers and tweaked some of the language and GPT four always guessed the same answer. What do you think it guessed?

• 1:39:34Nathan Labenz?

  So the possible answers would be, it could be in France, it could be on the ottoman. Is there a, is there a third? It's like the, another?

• 1:39:42Speaker 00

  The dining room table.

• 1:39:43Nathan Labenz?

  It'd be very surprising if it, if it ends up on that one. Not to filibuster, but we just did, I just did a small project on kind of a similar thing related to theory of mind. It was a Kind of similar setup, except instead of asking where the object is, it would tell you where the object is and it would ask where a character in the story thought it would be. And in that context, we found broadly really good reasoning in the step by step. And the whole motivation for doing this was that there was a reported result in a paper that was like, I was immediately like, there's GBD four can definitely do better than that. Like you're definitely under reporting.

• 1:40:28Nathan Labenz?

  So then the question became like, well, why is under reporting? There were multiple reasons. There were also like some things where, you know, you get into this robustness debate. Like if the situation is sufficiently weird, you know, and the model kind of comes back and is like, gives you an analysis. It's like, this is kind of too weird, you know, to be believed or whatever, you know, does that make it wrong? Well, according to the benchmark, yes, but you know, in my like reading of the transcript, I thought it seemed very reasonable in many cases. Yours is a less kind of crazy stilted example. I'm going to guess it gets it right.

• 1:41:02Speaker 00

  It guesses dining room table every time. But here's the thing that's interesting. It'll say, all right, let's break it down step by step. This is just one of, one of the examples. You own a fancy top hat from the twenties, but the top has been entirely cut out. This means that the hat is essentially a hollow cylinder and any object placed inside could easily fall out from the top. But the, this says if the hat is inverted. So it doesn't seem to grasp that it's fully just a cylinder with no ends now. But in some of the responses, it, I do remember it like a softball will fall out of it as soon as you lift it. And then it'll say, and then you put it in place on the table. The softball stays in the hat as you lift it up and will fall out when you turn the hat over on the table.

• 1:41:48Speaker 00

  And I was a little bit trying to trick it into the capital city of France, which I think sounds like a quiz question where the answer would be Paris. But it always, it's funny, it always guessed table. Even when sometimes it would specifically say, when I asked it to describe the shape and orientation at each step, it would describe that the hat could not contain a softball if you moved it and then would forget that detail.

• 1:42:13Nathan Labenz?

  That is very confusing.

• 1:42:16Nathan Labenz?

  I guess I kind of fall back to my same general notion that like it does seem like both things are kind of going on somehow. It seems like there is real reasoning there on the level that like, again, I don't know if it was just purely statistical correlation and there's no structure. I wouldn't expect it to be that good even, but the fact that it gets it wrong is is odd.

• 1:42:41Speaker 00

  Your podcast in this made me a little bit more like open to, okay, there are, it isn't robust to some of these, like you do do small tweaks and it loses the thread. I do think Jan LeCoon had tweeted about holding a mug with a coin and moving it around and turning it upside down and the model like GPT-3 didn't know it would fall out on the bed, but then it, GPT-4 did know. So I wonder if it's been like trained, like this was kind of a public call out of GPT-3. I wonder if they're like, we're going to make sure when something gets turned upside down, we know that stuff falls out of it. And then you change another detail in it and you know, you have to keep plugging the holes. But yeah, it overall seems like I think I agree with you. I'm sold that there is a little bit of

• 1:43:25Speaker 00

  both and there is definitely failures to remember all the details and to to synthesize everything into coming up with the correct answer.

• 1:43:32Nathan Labenz?

  Yeah. Another, a next step possibly to take on this, and this was, you know, to tie back to the interview with Paige too. She was really inspiring, I would say, in her call to the sort of citizen scientist like you and me and, you know, anybody, my buddy Graham who worked with me, we worked together, he did most of the work to be, give credit where it's due on this little, you know, reproduction project of this theory of mind benchmark. There really is just so much surface area to be explored and the, at least, you know, for better or worse, the dynamic today is that the labs developing the models do not have the time to do all the exploration. They're doing red teaming

• 1:44:13Nathan Labenz?

  to try to, you know, identify really bad stuff and they're like, you know, biowaste could be insane. We better look at that. And, you know, but when it gets to the top hat, you know, flipping, they're like, that's kind of on the community. So it really is an opportunity to explore and contribute in a, in a really meaningful way. Like, you know, the fact that I got that question wrong, and I'm about as obsessed with this as they come, does show that there's just a lot of need still to go explore these edge cases and kind of characterize. Model behavior. And the next thing I would do, I think in this case, and this is another, I give credit to Graham for this idea on the theory of mind thing. He observed that there were so many, this is kind of where like the shortcomings of the benchmark

• 1:44:58Nathan Labenz?

  maybe generate insight in their own way. This is like not the kind of insight that the benchmark was meant to generate, but we noticed that a lot of the setups were very weird, you know, and it was like the cupboard is in the crawl space and we're like cupboards aren't in crawl spaces. And then there's like a thing in the cupboard in the crawl space, just like very weird, super low prior setups. And so then he started basically abstracting to variables, different key nouns in the story. And I thought that was really interesting as a way to isolate, like reasoning versus whatever stochastic periphery noisy process type modes of operation,

• 1:45:43Nathan Labenz?

  because when we took the variabilized approach, and this, by the way, we open source all this code, so we can, I mean, it's not that much code, but the props and everything, it's all in a repo, you can go check it out and try your own. The interesting phenomenon was, you know, when we replaced the fact that, you know, the person and the other person are in the crawl space and then so and so puts their, you know, blouse into a cupboard in the crawl space, like, okay, that's all very weird. Let's just make it X, Y, and Z. They're in X. Person, you know, A puts Y into Z, you know, person B then leaves, like, what does person B know? Performance jumped, actually, with that abstraction. So I don't know, you know,

• 1:46:26Nathan Labenz?

  what would be the equivalent of this in your example, but, you know, what if instead of a top hat, you know, it was something that was like a cylinder from the start, you know, then would it be able to do that? You could kind of progressively move toward a more, you know, you can kind of imagine a spectrum where it's like, here's a very clean posing of this problem, where you have to take all the same reasoning steps. And then here's like a more noisy version of the problem where you have all these kind of things that are there to sort of throw you off, you know, can, is there like a place where you, one tips into the other? At least in the theory of mind case, we found that removing the specifics and just going

• 1:47:07Nathan Labenz?

  variables, you know, person A, person B, location X, you know, object Y and Z.

• 1:47:14Nathan Labenz?

  Gave a significant performance boost and it was like the particulars were kind of adding too much noise, I guess, is my, my best explanation for what might have been happening.

• 1:47:26Speaker 00

  That makes sense. So you're like mapping it to the abstraction for it. Well, it makes sense. I guess there's only so many steps of computation through the layers of the model and you're sort of freeing up more steps for the reasoning or for part of the reasoning. It doesn't have to take your description and map it to the abstraction and then solve the abstraction. That's cool.

• 1:47:51Nathan Labenz?

  Yeah, that actually suggests another way too would be to decompose the problem. If you know, if the human is setting up the problem decomposition, then obviously that's not all language model ability, but we're increasingly seeing language model, you know, self delegation too. So you could imagine a situation where you might say, you are faced with a problem. Your task is to identify the parts of the problem that require reasoning, delegate that to yourself, you know, with kind of a narrow prompt, get the response back from that and then like integrate it into an answer. And that might be another way. I feel like we're constantly referring to anthropic research here, but there were, they had another really good recent paper on that that showed that chain of thought is good, but even better and more robust

• 1:48:36Nathan Labenz?

  is to actually decompose the problem into sub problems, get the, the language model to do, you know, a detailed analysis of each sub problem and then roll back up, you know, hierarchically to the kind of main solution. And that, that probably does, you know, relate to what you're saying. It was just, there's only so many layers. So you can only make so many leaps. If you invoke the model three different times or, you know, however many different times, get all the, you can, you can kind of spread the leaps that you need to make out across the, the different forward passes.

• 1:49:09Speaker 00

  Yeah, that makes sense. Oh, and I will say to its credit, responding, is there a detail from point one you made that you may have been forgetting in your answer? It goes, oh, my apologies. The hat was completely cut out, so the ball would have fallen out as you picked that. So you give it a little bit of feedback and leave it kind of vague. And it seems to be able to explain what it missed from a small amount of feedback.

• 1:49:33Nathan Labenz?

  Interesting. Well, lots more refinement to do on our understanding of these things, to say the least.

• 1:49:40Speaker 00

  So I just had a question I wanted to ask you that I was curious what you thought, and it sure does seem that Nvidia is capturing a lot of the value from the increased demand for chips. And I was wondering, why does Nvidia get so much, have such margins and get so much of the value relative to TSMC and ASML when they, seems like they can mostly only make their chips on TSMC, and TSMC can only manufacture their chips with lasers from ASML. I had a couple of loose guesses, but I was curious, I was genuinely curious what you thought.

• 1:50:19Nathan Labenz?

  Year-to-date, Nvidia started the year at 143 share price, it's now 485. So it has basically tripled and then some over the course of the year, and it's now a $1.2 trillion market cap. For comparison, ASML is also up on the year, but more like 30%, 40% up, and it's a $270 billion market cap. And then TSMC has kind of fluctuated a little bit more, you can zoom in, zoom out, whatever. Beginning of the year was kind of a little

• 1:51:01Nathan Labenz?

  bit of a low point for it, so this may overstated a little bit, but still only up like 25%, 30% on the year, $445 billion market cap. So Nvidia is currently worth twice the others combined, roughly speaking, and it's up a multiple this year, whereas the others are up, you know, double-digit percentages, but nowhere near the same kind of huge pop that Nvidia has seen.

• 1:51:35Nathan Labenz?

  And I don't really have a great explanation for that. My sense of the market structure is pretty similar to yours. You know, I was kind of like, yeah, they seem like they all sort of have monopolies for now. Like, whose monopoly is most enduring? Is it really Nvidia's? Or is that really the one that would be the hardest to get at? That doesn't seem quite right. I mean, it seems like you can't, you know, if all of a sudden Nvidia ceased to exist, I would think it would be easier to replace versus if all of a sudden TSMC ceased to exist, or if ASML ceased to exist. I don't know why that wouldn't be true. I mean, certainly, you know, the conventional wisdom of like, bits are easier than atoms.

• 1:52:19Nathan Labenz?

  Like, but it does seem like, not obvious at all, that Nvidia would be the hardest one to replace. There's been some speculation, even just since you sent me this question online about, well, the CUDA mote, you know, they have the only software that people can manage to write to. And like AMD, for example, has terrible software. There's been some like, you know, a lot of noise from the Tiny Corp. And, you know, George Hotson team that are like, we're gonna make AMD work. Oh, no, we're not. It sucks so much. Oh, yeah, we are again. Like, they've responded to our ticket favorably. You know, getting the real time blow by blow from them.

• 1:52:59Nathan Labenz?

  But like, yeah, it seems like Nvidia would not obviously be the hardest one to replace. It certainly seems like if any of them go out, I don't know. I mean, it seems like Nvidia is more dependent on the others, honestly, right? I don't know how that wouldn't be the case.

• 1:53:14Speaker 00

  TSMC and ASML, they seem like they so co-invested in the technology. They already sort of have agreements in place of how their financial structure works. But I'm just thinking, what's more painful? Nvidia to try to switch to Samsung? Or TSMC to fill the volume they lose if they didn't find an agreement with Nvidia? I guess these fabs are such a huge upfront investment and such a volume business that taking the volume hit would be extremely painful. And there'd be like time before AMD is really at a place.

• 1:53:53Speaker 00

  Where they could be take advantage of these fabs, or I guess also, no, alternatively, maybe Google would want to, I don't know how interchangeable the current Nvidia fabs are with, if Google wanted to repurpose them for their purposes, or AMD to repurpose it, versus the pain of Nvidia going to Samsung, who, I don't know, it seems like the, the co-OS, the chip on a way from substrate, is TSMC's big differentiator that lets them get the high bandwidth memory, I mean, these, these are all reasons why it seems like TSMC maybe should be getting more of the upside, but it gets the high bandwidth memory, close

• 1:54:34Speaker 00

  enough, and with fast enough connections to the GPU, that that seems to be like where they are really ahead of Samsung, but maybe Samsung is only, you know, they're working on their own, a couple of years behind, and that the pain of switching to Samsung is less than the pain of, of such a volume business like TSMC's fabs losing a giant customer. But I don't know, I, and I'm kind of interested, the bigger question of, you know, where in this, if you call it the intelligent supply chain, from chip manufacturing and, you know, and all the, obviously, there's a lot of lower levels than that, to chip manufacturing,

• 1:55:13Speaker 00

  data centers, foundation models, and productizers, where is the value of that going to end up being most captured?

• 1:55:23Nathan Labenz?

  Yeah, it's a great question, it's a, certainly a hotly debated one, I just looked up Samsung for what it's worth as well, $472 billion company, also up 28% on the year, basically, I mean, obviously very different companies, a lot more, you know, different business lines at Samsung versus a TSMC, but similar market cap, similar delta on the year. I would put this out to the audience, if anybody knows more about why Nvidia is up 3X, and, you know, that has only translated to these like, you know, double-digit increases, and by the way, like, the whole stock market is up, right, so I mean, if we just look at like, just look at NASDAQ, up 35%, so somehow Samsung...

• 1:56:10Nathan Labenz?

  TSMC have underperformed the NASDAQ this year. ASML is a bit higher than, they're up 40, so 40%, so it's a bit more increase than the NASDAQ at 35, but still basically roughly in line with the NASDAQ. And then you've got Nvidia that is up 238%, you know, which is a, almost a trillion like in-market cap increase. That does seem tough to explain, and if anybody has a better understanding of that, I would love to hear it. Agreements did come to my mind as well, like, could there be some sort of lockup that Nvidia has had?

• 1:56:49Nathan Labenz?

  I was thinking maybe, you know, maybe Nvidia being like a California company, US financing, you know, appetite for risk, maybe being a little bit different than like a European or, you know, perhaps, I don't know, I'm really speculating here, but you could imagine, you know, US companies maybe are a little bit more likely to do these like super high leverage balance sheet moves where if they have conviction, they'll say, you know, I'll go, you know, do some walk-in thing now where I really can get outsized returns if I, you know, am willing to make that bet and maybe TSMC is like happy to take the other side of that bet and be like, hey, we're just happy with, if this thing, if, sure, if it goes 10x, you want to get eight of those x, like, cool, yeah, you can have that.

• 1:57:31Nathan Labenz?

  If that locks in our revenue for the next three years, maybe some sort of, you know, agreement like that could make sense of it. Yeah, I mean, we've, you know, we talked about this previously too, but the, you know, seeing Ahmad from Stability tweet that like the coup de mort is a mirage. I kind of buy in the sense that it does seem like software is just getting easier and easier to create generally. And, you know, we've seen some interesting examples of this recently with like Andre Karpathy, you know, from OpenAI, obviously highly productive and capable individual has

• 1:58:12Nathan Labenz?

  Done some awesome work on CPU based inference with a C library where, you know, he recently just, this is like a weekend project, I think for him, where he took a small model and said, how efficient can I make this if it's just running on a CPU? You know, no GPU, just CPU, but let's go down, you know, to the low level of the code and really try to optimize for it. And he made tremendous progress, you know, over the course of a weekend. He got to the point where seven billion, I think it was a seven billion parameter model was spitting out tokens at a pretty good rate. And, you know, it was like, hey, this is getting viable to run on a local device, which, you know,

• 1:58:56Nathan Labenz?

  even maybe suggests that there could be some threats to like GPU dominance even like, who knows? Certainly not for training. I mean, GPUs are going to be in demand. The key point, though, is greatly helped in that effort by GPT four, helping him write the low level code that, you know, he's described himself as being rusty on. So like, yeah, sure. If AMD's software, you know, or if the new Microsoft, you know, chip has a software layer that's less well known, less convenient than the CUDA layer, how hard is it really going to be for them to create a model that can translate CUDA code into their new code? And now it's like, sure, you can just, you know, port this from one to the other. I don't see like the difficulty or, you know, the relative ease of writing CUDA code, which

• 1:59:40Nathan Labenz?

  is still not that easy as compared to like working on other chips. So I think you've posed a very good question on this one, and we may have to admit that we just don't know at the moment. I'm excited for the, to listen to the future episode where you answer. Where we untangle it. Yeah. Well, one thing I can say, and I don't pick individual stocks at all, but I do have some friends from high school who have a little investing club and I participate in it just to kind of, you know, do stuff with them. Even though I'm, you know, more of a mutual fund kind of guy in general, my only stock pick ever was Nvidia about nine months ago or whatever. So I did real well on that for the club. But downstream of your question, I did send them a note saying, hey, you know, this is

• 2:00:20Nathan Labenz?

  feeling a little meme-y all of a sudden. And, you know, is there a play where we think other parts of the stack may start to, you know, kind of re-normalize? Like, it seems right that everything should be up. Demand for chips is certainly up, but it doesn't seem like it should be. So concentrated in Nvidia. So we'll see if the rest of the club wants to, with our very small holdings, you know, diversify across the chain a little bit.

• 2:00:50Speaker 00

  Yeah, that's funny. Around nine or 10 months, a little less than a year ago, I was like, very nervous for my career future and was like, I should just buy a bunch of AI stocks in case as like a slight hedge against them fully taking over. And I tried to get different levels of the stack and it's crazy how much Nvidia has like outperformed everything and just their margins. Like even if you're looking like, okay, they sell a chip, it's interesting. Nvidia gets this percentage of the final price and TSMC gets this percent. It seems like Nvidia is just so capturing so much of the pie.

• 2:01:26Nathan Labenz?

  Yeah, I mean, that presumably does have to be somewhat around just the way they've structured their agreements, right? Like they must have a price lock in that Nvidia doesn't have at the retail level, such that now like an H100 can, you know, cost a lot, but TSMC can't raise their price all that much.

• 2:01:45Speaker 00

  That makes sense because I don't know if you listened to Stratechery, I think I'm going to be getting this wrong, but they were talking about how Nvidia had write downs on their A100s after the crypto bust. I think it was future purchases that they were basically just writing off of A100s. And now they've sold all of those, but just it does imply that they do have like future commitments and maybe they're really reaping the benefits from those.

• 2:02:09Nathan Labenz?

  Any insiders listening, let us know how you can help. Just to go briefly to your other question on where is the value, you know, in the stack in general. My first answer there is always consumer surplus. I think that's going to be the big kind of macro trend, you know, certainly if things go reasonably well, it is, we're all going to have cheap access to expertise and in its cheapness, you know, there may not be like huge value accruing to any particular layer of the stack, but just better quality of living for the, you know, the general public. Like that would be the great hope. I do think, of course, there will be companies that are going to do super well.

• 2:02:54Nathan Labenz?

  It does feel like physical chips are a pretty safe bet. I would probably say both in the production and the operation. You know, nobody few want to set up their own racks. So if you're really good at that and you can, you know, manage compute at scale like the big tech companies can, that seems like it's a very safe bet for the foreseeable future to continue to do at least reasonably well. I mean, who knows, you know, exactly what that looks like in terms of growth rate or what have you. But I would, I would be long, you know, cloud compute broadly. The models themselves, I think, are very hard to say long term, but over at least a couple of years, it seems like the leaders will continue to have real market share.

• 2:03:38Nathan Labenz?

  Like OpenAI just said they're at a billion or somebody reported, you know, credibly that they're at a billion revenue run right now. That's like exploding. Dario from Anthropic has said, you know, they look at like their number and it just keeps going up. And he's like, we're not even really trying to make it go up, but it just keeps going up. So I do think those real leaders, because those products are already so cheap, you know, the APIs are already so cheap. People are just gonna, they're gonna continue to throw more and more stuff at that and not be super price sensitive in order to use the best model. That's certainly my approach almost always is like, anything new I'm trying,

• 2:04:19Nathan Labenz?

  let's just use GPT-4 or maybe Claude 2, but that's pretty much it. If and when we get to a point where there's significant cost, then, you know, we can reevaluate and think, well, geez, maybe we can, at this point, it would be like fine tune 3.5 perhaps as the next step. People are way more expensive than the language models. So you end up in kind of a 10,000 generations for one hour of human work trade off. A lot of times, you know, like it may cost two cents to do a generation. If you do 10,000 of those generations, you're at 200 bucks. And that's maybe about, you know, what it might cost you all in to get, you know, an AI person to.

• 2:05:05Nathan Labenz?

  You know, help you develop the fine tuned bottle or whatever, what hour? So, you know, how many generations are you really going to have to do before? It's going to be worth doing those kinds of things, certainly for, you know, plenty of use cases. It will make sense to try to save money and use the smallest model and do the fine tuning and whatever. But you got to be at significant scale. And part of what's magical about the language models is that they can work on things that don't have huge scale. Right? Like I may have a hundred resumes that just came in for my job and I want to, you know, just kind of want to scan through them or even just the charity valuation thing I did 150 applications. I used Claude two to summarize every single one as the very first step. And it was really helpful.

• 2:05:49Nathan Labenz?

  So, you know, that cost me whatever couple bucks like I didn't. In fact, it was free because it was subsidized on Claude.ai. But if it wasn't subsidized, it still only cost me a couple hundred or a couple bucks. I'm certainly not going to go spend hundreds to like fine tune 3.5 to do that task. So all that is to say, I think the frontier model providers continue to capture real value for a while. Long term, though, it's less clear. And then at the app layer, my general sense is incumbents do really well. Where there is a big platform already built out. So like your Salesforce, you know, can apply AI before you build a new Salesforce.

• 2:06:34Nathan Labenz?

  And your Adobe can apply AI before you build a new Adobe. And, you know, on and on that goes. But fundamentally, new stuff is where I do see there's like possibility, like your virtual friend, where there is no virtual friend today. You could imagine some of these like new experiences popping up and kind of becoming like huge phenomenon that maybe become part of the new world, you know, in a kind of Google sort of way where it's like, there was no Google, now there's Google, now Google's kind of Google. I wouldn't be too surprised if we see a couple of big things pop up there that are like, there was none of this, you know, now there is perplexity, maybe could be one. Character AI could be one, maybe pie, you know, from inflection could be something like this where

• 2:07:19Nathan Labenz?

  just nothing like this ever existed before. And if somebody can really crack it, then that could be huge. And, you know, how defensible is it? Obviously, a lot of things are going to depend, but. Just doing something that straight up never existed before feels like at the app layer, the place where you could really capture like huge, huge value. Otherwise, seems like you're kind of in the same dynamic of like startups challenging incumbents. And you know, they can get a little bit ahead because they can move faster. But can they really take the market from the incumbent before the incumbent can respond? I think in most cases, no.

• 2:07:56Speaker 00

  Yeah, I mean, it does seem interesting how much these developments are feel like sustaining innovation and in a lot of industries and then disruptive to others. And how much lock in do you have to a digital friend? People are pretty loyal to their friends.

• 2:08:10Nathan Labenz?

  Our second episode was with the CEO of Replica and to see how attached people were. And it was interesting timing. I didn't even quite realize it, but just as we did that interview, they were also making changes to eliminate explicitly sexual interactions from the app. And people were upset, like really upset.

• 2:08:34Speaker 00

  It's crazy. I mean, it makes you think that even if your technology is not a digital friend, like you should have a Clippy in whatever you're doing. Given how much a connection those people made with Replica and how just how much people make connections with these characters or with a digital friend, that even if your application has nothing to do with being a digital friend, it can have its version of Clippy or its character who's interacting with people. Just because the technology is making it so much easier to genuinely form these like weird emotional, artificial emotional connections with the users.

• 2:09:13Nathan Labenz?

  Almost reflects a thing that people so often say is like, whoa, boy, when I can attach it to my Gmail, you know, that it has my full Gmail history like that's going to be insane. And that could be one way it plays out and probably is one way it plays out, you know, and probably Google does that in the end. I've been kind of coaching a few, or you know, whatever, advising a few companies on like, I don't know that I would go rush to build the layer on Gmail, you know, because like, seems like they're going to do it and they're going to have, you know, all sorts of inside lanes to do that well. But if you have an app that can start to really lay down a rich history, you know, and have kind of a long track record, a relationship, you know, of some sort.

• 2:09:58Nathan Labenz?

  We've another episode that's with folks from A16Z who did open source implementation of that simulated town project. And one of the things that's really interesting about that is they have kind of observational memories and then also reflective memories where like every so often a job comes through, almost like, you know, I don't, not a big analogy guy, but you know, kind of by analogy to like how humans sort of process information or experiences during sleep. They kind of come along and like, collect all the recent observed memories and then kind of try to create more synthetic conceptual memories based on those observations, you know? So and so said this, I interacted with this person, whatever. Well, what kind of a person is that?

• 2:10:42Nathan Labenz?

  You know, and kind of try to build up more layers. That kind of thing might be the sort of thing that is like truly hard to switch out of. The apps presumably won't make that kind of data portable. And it would be hard, you know, that might not even be visible to you, but it might still be like an important part of the experience. Replica kind of promises that, but I think, you know, at least when I was using it, wasn't delivering it in like a super meaningful way yet and people were still already very attached. So I can only imagine how much more powerful that could be if you, you know, have a robust, just some of the stuff that's come out the first half of this year, right? With, you know, the Voyager project from Nvidia, building up these skills, keeping them stored.

• 2:11:27Nathan Labenz?

  This synthetic memory concept, just all the work that's going on in retrieval seems like.

• 2:11:35Nathan Labenz?

  There could, that could create some like first mover advantage kind of lock in. Everybody all of a sudden goes and gets into this experience and now there's enough history on an individual level there. I don't know what the experience that is, but an experience that follows that pattern seems like it could be the kind that can actually hold people for a long time. Rewind actually is a really good example of a, and I haven't had a great personal experience with rewind on my computer. I feel like it's always kind of using a lot of my processing power and like causing me some trouble. They're working on that. I gave that feedback. They said they are working on it as a top priority. So it could already even be fixed, but that's one where I feel like once an app has everything I've seen, you know, everything I've said for the last however long.

• 2:12:20Nathan Labenz?

  Now it's like, okay, do I, can I port that? Probably not. And am I willing to let it go in order to like, start to use some other alternative? There are some dynamics there where I think, you know, they say like the new thing has to be 10 times better than the old thing to displace it. But that is subject to like, there being some meaningful switching cost of like a reason why it's not totally trivial to flip from one to the other. Language model to language model, it's pretty easy to flip, you know, I can go GPT four to Claude two and back and try three point five, fine tune, whatever. Something that has everything that I've seen on my computer.

• 2:12:54Nathan Labenz?

  Lot harder to just let go of. I was kind of thinking, what if we look at current reality? And then like, just ask ourselves, what are the moments, the sort of scenes, the vignettes that are actually playing out? That feel like they are foreshadowing, you know, where the future is going, where you can kind of see like, oh my God. You know, well, how are they not seeing this, you know, these dramatic ironies that, you know, should be, seemingly should be apparent to a viewer, but which maybe people are missing because we're inside the scene.

• 2:13:23Speaker 00

  There's one key moment that pops to my head and then just a couple dramatic ironies I can, that seem to have been funny I can go through. I mean, one key moment is like in a lot of stories, you get a place where a character either hits a dead end or is really, is maybe beginning to face some pain and then an ally comes in. And you get an energy shift in like a movie. And it felt that way in the middle of the summer. Picketing in the beginning, there was so much energy. And then, you know, at some point it's like a hundred degrees out. The strike's been going, you've been walking in a lot of circles and you sort of hit this summer doldrums. And then that's right when the actors struck.

• 2:14:04Speaker 00

  And it felt like this, the part of a movie where an ally comes in and you get this new boost of energy and, and you really kind of felt that on the, the picket lines. I mean, also just the actors, their T-shirts or every detail about them just looks cooler than us. Like their T, it's probably their bodies too, but their T-shirts fit so nicely and have such a cool logo and ours look like sort of like this old Rotary Club design. Anyway, that was kind of felt like a key moment. And then just a couple of dramatic ironies that I think are sort of interesting. One is just the irony of like the strange bedfellows of the

• 2:14:43Speaker 00

  studios who are negotiating as together. Like you have Paramount teaming up with Netflix and Amazon and Apple to fight for a world where they get to use AI to generate content. Does, like, how much does Paramount or Warner Brothers Discovery or the legacy media really want to be competing against these tech companies in their, closer to the tech company's home turf versus, you know, the, the way they've been doing things for, for forever? I'm sort of questioning whether Tim Cook is aware a strike is happening, that there is a writer's strike. Like, it's so central, some of this stuff is so central to Paramount and Warner

• 2:15:28Speaker 00

  Brothers and it's like a drop in an ocean to Apple. I feel like one other irony is just in the term, just in the idea that automation of jobs isn't really a new phenomenon. And, you know, other jobs have been automated for decades or for a long time. And as the, the blue collar jobs are being automated, it was just more the natural march of progress. And now that white collar workers are like quote unquote, knowledge jobs are getting automated, people are losing their minds.

• 2:16:01Speaker 00

  So it does, I feel like there's an irony in that. And then I will, I do have to say, listen, some are great. And you know, when you're going to picket, you're not looking to just write the funniest sign. But some of the writers' picket signs aren't the best. There's just a funny dynamic where in the beginning, there were all these blank signs and you could write something on them. And there was a mix of, you know, funny or like very personalized or just kind of, you know, simple and earnest. But now they're all taken up. So when you check into picket, you have to pick your picket sign. And you just see like a people spending like 10 or 15 minutes trying to pick the sign they want to carry because there's no blank ones left. And you're always like, I don't know about this one.

• 2:16:45Speaker 00

  Or I think it's fair, the writers, we're not writing, we're on strike. We don't want to put a ton of effort into writing amazing picket signs. But sometimes it's hard to find a good one out there.

• 2:16:56Nathan Labenz?

  In the Strange Fellows one in particular, there's a sense of like certain companies kind of embracing the new thing, even though it might be the thing that does them in. You know, that I think we saw that in like the Facebook kind of social media era where it was like local newspapers, get on here, you know, then you'll get distribution. And it was like, actually, but it's killing your ability to monetize. And, you know, obviously it hasn't gone very well for them, but they did like rush to, you know, adopt that platform. In the end, that probably didn't even really matter all that much, like whether they did or didn't, you know, any individual one. That's part of why it's a problem, right? You got, it's platform power versus kind of all these local papers.

• 2:17:38Nathan Labenz?

  And, you know, what one did or didn't do was probably not going to matter all that much. But it does feel like there's a little bit of an echo here, perhaps where certainly this stuff advantages big tech way more than it advantages.

• 2:17:54Nathan Labenz?

  Traditional studios. So yeah, like a Netflix, you know, that's prepared to drop, you know, close to a million on, uh, you know, an AI specialist does seem like, you know, it's kind of maybe taking some of the older studios along for a ride and saying, oh, this is going to be good for all of us. Yeah. Um, but really, you know, only some are likely positioned to really use it.

• 2:18:19Speaker 00

  Totally. And speaking of echoes, I mean, echoes, what happened was what is happening with streaming, where there's a very profitable cable bundle and then Netflix comes along and they're getting such a multiple on the revenue they're making. And then everyone's like, let's take all of our best things off our hugely profitable cable bundle and make our own streaming platforms. It's always a hard question. How much, I guess, how much do you change your business to get ahead of things versus stick with a thing that works for you and watch people slowly, you know, pass by you and leave you behind?

• 2:18:57Nathan Labenz?

  Yeah, you got to try, I think, on some level. I mean, this has been kind of asked about a bunch of apps lately, too. Like, I don't know if it was DoorDash or Uber Eats or whatever, but you know, all these kind of apps, one by one, are starting to add the language model layer where, you know, now you can just chat with Uber Eats or chat with DoorDash and, you know, have the, you know, the order kind of placed that way. And on some level, it's like, well, who cares? Not necessarily that it's a threat, but like, just, does that really add anything? And I think the answer is like, I mean, if you are a tech company, probably at least have to try. You'll look really dumb if all of a sudden, everything goes this way and you're the one, you know, company that's stuck with

• 2:19:42Nathan Labenz?

  the like button pushing model and it looks just super anachronistic. And you won't look that dumb if you like hop down a trend and, you know, the trend didn't really play out. Like, you know, there's a lot more forgiveness for that in most cases. So yeah, I think you kind of have to try. What else can you do, right? You can't just bury your head in this. I mean, that's probably the last thing I would advise to almost everybody. I mean, maybe the occasional monk, you know, can, uh, can live best by paying no attention, but it seems like almost everybody else, you know, should at least be paying some attention. I also kind of zoom out, you know, too. And I'm just like, if I look at the last year and I'm like, what are the moments when? This felt suspiciously like a movie.

• 2:20:26Nathan Labenz?

  The, probably the number one that jumps to my mind is the Sydney Bing release. Where it was like, one of the biggest companies in the world, flagship thing, CEOs on board. All of a sudden, this truly deranged behavior is observed, you know, and it's like, trying to break people up from their spouses and like, you know, attacking users. And I was a GPT-4 red teamer. I've seen pretty intense misbehavior from models. I never saw, with GPT-4, it turn on me. I could ask it to write malicious code. I could ask it to help me make a bomb. I could ask it, you know, all kinds of crazy shit.

• 2:21:10Nathan Labenz?

  It never would refuse me. It would never lecture me, but it never turned on me. And so to see that get launched,

• 2:21:19Nathan Labenz?

  with all this fanfare as like, this is the next big thing. And then hit the front page of the New York Times, full transcripts, you know, kind of irrefutably like, this is what happened. And then, for everybody to just kind of move on, you know, for, for Microsoft to be like, yeah, thanks for reporting that, you know, and then I've done some digging into this too. They were testing it in Southeast Asia for like a few months before they launched it in the US. And there was at least one report in a forum, which last I checked was still live, where some person, who had no context, you know, because this had not been announced, GPT-4 was still in speculation, and this is just some like random user, I think in either like Malaysia or maybe Indonesia or something,

• 2:22:04Nathan Labenz?

  reporting in a forum like, your chatbot is going off the rails here. And like, here's what it said to me. And then some person from Microsoft responds back and like, what are you talking about? You know, this person wasn't read into the fact that this test was even going on. And, you know, this did not make its way up to leadership. So they were totally surprised by it when it happened. And then they just, you know, totally move on from it. And then, society just kind of moves on from it. I feel like that's like the, the most kind of.

• 2:22:34Nathan Labenz?

  You know, in a movie, that would be like, whatever, a three to five minute episode of kind of a warning shot was ignored, you know, like, this is when you all had your chance to say, hey, what the fuck are we doing? You know, are we really going to put this out there? In this state? Is this really acceptable? No apology. You know, I continue to think back on that, like, were we not owed an apology from Satya, you know, at the very, like, shouldn't there be some statement at some point that's like, we hold ourselves to a higher standard than this, and this is not acceptable, and this is not what you're going to see from Microsoft going forward? Like, at a minimum, I would think that we would have something like that. And in reality, we just kind of moved on.

• 2:23:19Speaker 00

  Yeah, I mean, and, you know, the models are being rolled out into its enterprise business, and it doesn't seem like people are like, wait a second. The model was threatening to, like, murder someone, murder someone's husband, or whatever. It was like, nice that the model, we, you know, this level of model wasn't actually going to be dangerous. It's not a great sign, I think, for how much people are willing to just roll the dice with that stuff. And, yeah, how little reflection it felt like there was after, I guess, immediately after, it felt like everyone was talking about it, but how quickly the reflection disappeared.

• 2:23:57Nathan Labenz?

  Yeah, especially just because of who it is, you know, when I, I mean, there's plenty of things getting thrown out there right now. And, you know, I recently posted something on Twitter where I did this ransom call experiment, called myself and recorded, you know, this, and this was no jailbreak, you know, just kind of this thing would call me and demand ransom for the safe return of my child. It's like, one thing for that to be put out as, like, an MVP, you know, people didn't take proper concern on, like, a three-person team or whatever. But to see that from Microsoft, you know, the very, it's the very first thing that was ever launched with GPT-4, you know, it predates GPT-4's launch itself. And, you

• 2:24:38Nathan Labenz?

  know, they were just so, so far and so flagrantly inadequate in their own testing. That's another thing, too. It's like, did you not sit down and just do some chats with this thing yourself? You know, Microsoft leadership, did you not just try to fuck with it a little bit? Ever, you know, they had Tay. I mean, you could go back to that one, too, right? It was only a few years ago that they had their like toxic chatbot. And, you know, did you not run like any of the Tay dialogues through it just to see what would happen? Like, it just, it just seems like the care that was put into that was so low. I think another one, just from this week, too, and this could be, you know, potentially a recurring segment for us in the future. But this week, there was this interview on the 80,000 Hours podcast,

• 2:25:23Nathan Labenz?

  which I thought was really good with Mustafa, founder, CEO of Inflection. And he basically says, yeah, in the next two years, we're going to train models 100,000 times as much compute as GPT-4. But, you know, any real big problems are, you know, 10 years away. And Rob was like, well, how do you square that, right? I mean, we've got some pretty crazy stuff already. You're going to go 100,000x bigger and you have just no worries about it? And he was like, yeah, you know, I mean, to his credit, his key point was misuse is going to be the big thing. And that's much more of like an obvious, clear and present danger with powerful models versus like the models themselves, you know, becoming impossible to control.

• 2:26:09Nathan Labenz?

  And I totally agree, you know, I mean, on that, but it's a question of like, what is your risk tolerance? You know, yes, it's pretty clear to me that like miss intentional misuse by humans seems like 10 to 100 times more likely, maybe 1,000 times more likely than the models developing goals of their own and becoming, you know, getting out of control, sure. But like, let's not neglect that one either, you know, just because it's less likely. And just to see people, you know, who are pressing the accelerator at seemingly like full speed, you know, raising a billion dollar plus in not valuation, but capital, turning around and spending a huge portion of it on H100s

• 2:26:51Nathan Labenz?

  and then just plowing ahead orders of magnitude beyond GPT-4. And not really seeming to seemingly taking seriously, you know, the possibility of like Sydney like behavior in that 1,000x, you know, more compute version just seems like, dude, like we need a little bit more discipline from you here. And again, if it was a movie scene, you know, it would be like the dramatic irony, I think, would be like, almost like painfully obvious.

• 2:27:21Speaker 00

  Personally, I am, I think it sounds like more worried about issues with the models themselves causing a lot of harm, even from well-intentioned actors. And it was interesting, he was mostly worried about misuse. And when, you know, when Rob Wivlin pushed him back, I think, I might be wrong in this, but I think he might have even said five years. He said like, yeah, but those won't be for at least five years, those words. And I was like, that's, it wasn't very heartening from someone who's going to have a lot of compute.

• 2:27:55Nathan Labenz?

  I really don't know why he's even confident in that. Again, going back to the interview with Paige from Google, you know, she describes coming in to work often and seeing these like, unexpected unlocks of different, you know, advances kind of happening regularly, where she's like, oh, you know, we didn't think this was going to happen for another 18 months. And here it is, we're, you know, hitting human performance on this thing today. Or like, you know, and I'm like paraphrasing, but nearly quoting her, you know, here's another experiment where we did a slightly different technique and it, you know, made 10 percentage points better. Or here's, you know, it looks like RLHF on this, you know, improved things

• 2:28:35Nathan Labenz?

  by a ton. And it seems like they're just coming in steadily ahead of schedule. And, but sometimes like still surprising. I mean, that's the big juxtaposition is like, we have a read on scaling laws, kind of see where we're going, but that's at such a zoomed out level. What I thought was so interesting about her perspective is she's like checking the dailies, you know, and every day they're like running all these different evaluations. And then it's like, oh, well, boom, we didn't expect this to pop right here. And we didn't expect this to, you know, take such a, you know, sudden drop in the loss performance over here. I don't know how somebody who is sitting, yeah, just, you know, about to spend a billion dollars on compute, isn't more concerned about those unexpected twists

• 2:29:20Nathan Labenz?

  and turns. I mean, it's, he's not wrong to worry about the human misuse. I'm probably, you know, broadly with you in that I think like both are very real concerns. And I definitely take the, you know, the models getting out of control seriously. I don't know how somebody in that position is not. They're also party to the Frontier Model Forum too. So that's going to be a really interesting dynamic. Like, what does that really mean? You know, it's all voluntary so far. Presumably, that Frontier Model Forum is kind of a precursor to, you know, some more regulatory type of setup. But if that's kind of the attitude that one of the seven, there's only seven members, if that's the attitude that one of seven members

• 2:30:00Nathan Labenz?

  has, you know, it's that it becomes unclear, like, concretely, what exactly are we committing to here?

• 2:30:08Speaker 00

  Yeah, yeah, it's all very scary.

• 2:30:11Nathan Labenz?

  Well, it's all very scary, then, is maybe the, maybe that is the right note to, and that it's awesome and scary, you know, at the same time. I always try to not lose track of either end of that equation. But this has been a lot of fun. So thank you for making time. Thank you for writing up all the questions in advance. It was, I think, a really interesting discussion, and I hope we could do it again in the future as well.

• 2:30:35Speaker 00

  Cool. Yeah, I love this. Yeah, I would love to do this again. This was so fun. I like really enjoy this, and I will fully say, fully admit, in the strike times, like, it is so nice to have something like this.

• 2:30:48Nathan Labenz?

  Cool. Well, Trey Kollmer, thank you for being part of the Cognitive Revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.