AI-Powered Filmmaking with Waymark's Stephen Parker and Josh Rubin

In this episode, Nathan sits down with Stephen Parker and Josh Rubin of Waymark, and creators of The Frost, an AI-powered 12 minute short film. In this episode, we get a behind the scenes look at their creative process, the prompting and creative techniques they used to generate and animate the DALL-E results, and an overview of the current state of AI art. If you're looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. TIMESTAMPS (00:00) Episode Preview (00:01:00) Nathan’s introduction for Stephen Parker and Josh Rubin (00:05:01) - The Frost is a 12-minute short film created using DALL-E 2 images. (00:07:06) - The Frost started as an experiment to see if a narrative film could be created completely from AI imagery. (00:10:38) - Parker started generating images with DALL-E 2 when he got access to the early preview. (00:12:26) - Prompt technique to get consistent images by providing context about a hypothetical film. (00:15:57) Sponsors: Netsuite | Omneky (00:19:37) - Compositional continuity, like shot-reverse shot, was hard to achieve through prompting. (00:22:13)- Rubin would request specific shots and the team would prompt DALL-E 2 to create them (00:32:25) - Getting consistent facial features for characters was very difficult. (00:39:03) - The storytelling helped cover inconsistencies that viewers might not notice. (00:40:15) - Working with the images DALL-E provides (00:44:53) Inpainting and compositing to refine DALL-E Images (00:50:43) - The AI art is limited by what exists in the training data. (01:02:05)- Animating the human characters was challenging because of missing or incorrect appendages. (01:07:36) - The team had to find creative ways to convey emotion through the limited animation. 01:02:24 - Animating subtle human movement and emotion is still very difficult. (01:06:35) - A romantic comedy would be much harder to produce with current AI capabilities. (01:12:17) - For Frost 2 they are using text-to-video models like RunwayML. (01:15:43) - AI voicing advancements applied to filmmaking (01:19:27) - The future of AI in Hollywood and filmmaking: quality narratives still require human vision SPONSORS: Shopify | NetSuite | Omneky Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive NetSuite has 25 years of providing financial software for all your business needs. More than 36,000 businesses have already upgraded to NetSuite by Oracle, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you're looking for an ERP platform ✅ head to NetSuite: http://netsuite.com/cognitive and download your own customized KPI checklist. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. Music Credit: GoogleLM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  This stuff, it was exceptionally hard, maybe even harder than a traditional animation. You're working with an unknown artist in the room who can give you exactly what you want, or who can give you some random wonderfulness, or some, as other people say, grotesque image. So it's a huge challenge. In order for it to be good, there needs to be like a big 500-foot human vision.

• 0:30Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm thrilled to be speaking with my longtime friends and teammates, Stephen Parker and Josh Rubin, creative leads at Waymark and creators of The Frost, a groundbreaking short film made entirely with Dolly 2-generated

• 1:11Nathan Labenz?

  imagery. For a bit of context on the AI journey that Stephen, Josh, and I have been on together, from 2017 to 2021, Waymark had built the easiest-to-use video creation app on the market. And the quality of the video templates that Stephen, Josh, and the creative team produced was our standout feature. However, feedback showed that users wanted more than an easy-to-use DIY solution. What they really wanted was an app that could create content for them. Now, I had been interested in AI forever and was always looking for AI tools to enhance our product, but I'd only done a tiny bit of hands-on development because,

• 1:51Nathan Labenz?

  frankly, none of the AI technology available at the time really worked for our purposes. That first started to change with OpenAI's release of GPT-3. And in September 2021, when we successfully fine-tuned the Q-E model for the first time, I became convinced that generative AI was the solution. Some on the team, I think, thought that I'd lost it when I used my prerogative as CEO to pause just about everything else we were doing, up to and including board meetings, to organize a generative AI 101 crash course for the team, and reorient our product roadmap entirely around generative AI.

• 2:29Nathan Labenz?

  Steven and Josh, to their credit, came along for the ride and have since caught the AI wave in their own unique way. While I've got my 10,000 hours of AI usage with a mix of language, computer vision, and text-to-speech models, Steven and Josh have gone super-deep on AI art. We got first-wave access to Dolly 2 as an OpenAI Innovation Partner customer in early 2022, and since then, Steven has personally generated over 1 million images with Dolly 2. The result? The Frost is not just a proof-of-concept, but a legitimate 12-minute film with a coherent narrative and a consistent aesthetic. In this episode, we get a behind-the-scenes look at their creative process, a sense for

• 3:13Nathan Labenz?

  the challenges they face and the strategies they use to overcome them, and overall, one of the most sophisticated accounts of the current state of AI art that you'll hear anywhere from a team using these tools with the highest level of taste, vision, and skill. Creating the Frost was not easy, but this project does show how transformative generative AI is likely to be as it continues to mature. Only seven people are credited on this film: Josh and Steven, our Waymark team members, Tommy Herman, Zach Polley, and Lexi Dietz, and collaborators Matt Session and Robert McFalls. As always, we appreciate your reviews and your online shares, but this time, I really

• 3:54Nathan Labenz?

  want to encourage you to watch the film itself and also the trailer they've recently released for The Frost 2, which they are already making with an entirely new generation of AI tools. We'll have links to these in the show notes and also to some visual behind-the-scenes content and an MIT tech review write-up of the film as well. Now, please enjoy this fascinating conversation with Stephen Parker and Josh Rubin, AI creative pioneers and makers of The Frost. Josh Rubin, Stephen Parker, welcome to the Cognitive Revolution.

• 4:27Stephen Parker?

  Thanks for having us.

• 4:28Nathan Labenz?

  Very excited to have you both. Obviously, we've worked together for a long time at Waymark, where you guys have led the creative department and brought a level of creative quality to the work that we do that was certainly previously inaccessible to the likes of me. So, you know, very much appreciate that over the years. But today, we're here to talk about your recent project, which you've done that at Waymark, but you know, kind of for broader exploratory and creative purposes. And that is The Frost. So I guess for starters, tell us, what is The Frost?

• 5:07Stephen Parker?

  So The Frost is a short film, 12 minutes, part one. It is a film that we created using Dolly images, essentially, which are still images that we prompt for, curate, and then take into After Effects, cut up, use puppetry, various styles of animation, run the images through DID, do a whole bunch of things to essentially create video from still imagery, and then assemble a short film out of that new quasi-video. I would say that Josh probably has more to say about what The Frost is than me. He is the director on the project. But you

• 5:50Stephen Parker?

  know, it's just fundamentally about the exploration of what, you know, new AI can enable for creators.

• 5:58Speaker 00

  Yeah, I mean, it was, The Frost was kind of a happy, wonderful accident. It was kind of born out of, out of the curiosity to see if we can make a film generated completely out of AI imagery. And what I mean by film, it's more like, not just like a

• 6:19Speaker 00

  You know, a montage of images set to music and to see if we could animate them and to get the images looking as best as possible. I think there's a lot of people out there in the world doing that, doing that well and doing that probably better than us. What I mean, my film is like to create a narrative out of AI-generated imagery. Could it be possible? Is it possible? We saw the text-to-image generations as like the beginning of AI cinema, which is the beginning of this whole new revolution in entertainment, really. And like, just this is like, we just saw the beginning of it.

• 7:03Speaker 00

  And, um, and the frost was kind of like an experiment put into action. And, um, yeah, and then three and a half months later and thirteen, thirteen minutes later, we had a, um, a narrative. We kind of, you know, for better or worse, like we kind of, uh, we achieved our goal. So we're, we're quite happy with it.

• 7:27Nathan Labenz?

  Yeah, it's been very well received. I think it's, it's been really interesting to see how the, you know, the, the sort of media has kind of taken an interest in the project. And what seems to capture people's attention most is this, you know, kind of standard that you're speaking about, right? Where you're, you guys really set out to do something that is not a proof of concept, but instead saying, you know, given these new tools, can we create something that, you know, we as creators and filmmakers would be proud of and that people would actually want to watch, not as a pure AI curiosity, but as something that, you know, hopefully.

• 8:13Nathan Labenz?

  Stands up against other entertainment on its merits, right? I think that's a key difference in the way that you guys approached this project relative to, you know, so many things where people are like, look what I, you know, kind of spit out, you know, look what AI spit out for me. There's a lot more, I guess I don't really know a lot about your process and I don't really, to be honest, know a lot about filmmaking in general. So maybe you could take us through the process and maybe highlight kind of to the degree that it differs from a traditional filmmaking process, obviously in many ways, especially at the kind of technical execution level where it does. But even maybe just starting at kind of the conceptualization, how different did this play out, how differently did this play out relative to a, you know, a traditional project

• 8:57Nathan Labenz?

  when it came to, you know, just the first questions of like, what are we trying to make? What story are we trying to tell? Did you feel like you could start with the same questions or did you have to approach it in a fundamentally different way, given the different tools that you were going to be using?

• 9:11Speaker 00

  I thought the genesis for the project was, I mean, it's a very different, it's a very different process from when you're kind of ideating a completely original piece without any kind of images to go off of really, you know? Normally you start with a script or an idea. You start with an idea, then it kind of evolves to the script phase and then you kind of storyboard things out and then if you're lucky, if you have enough money, then you, then you get to shoot the thing, right? And you go location scouting, you choose your, you choose your environments and your, and your sets and scenes, and then you kind of go out and film it. And with Dali, it kind of gives you a great starting point.

• 9:56Speaker 00

  Like it gives you some, some place to start. And whether you're, you take to it or not, that's up to the creator. But, um, you know, what happened with, with our project was, you know, Stephen was extremely excited about the, this new technology and just went gung ho in terms of generating all these fantastic, extremely photorealistic cinematic images that really lent themselves to. You know, just fantasizing about like about, hey, this could become a movie. Um, and so it was easy, at least for me, to see, you know, Stephen's initial Frost

• 10:41Speaker 00

  series, whereas I think it was a series of, you know, 20 or 30 images. And from there, you know, these, basically, these images were faces and mountains and close-ups of gear and things like that. It's like, it was the beginning of a world. And all it kind of needed was a story to kind of tie it into. So like, we had this amazing starting point for this, which normally you don't get unless you're kind of drawing inspiration from a bunch of different things, and then you ultimately have to make it your own. But with, you know, a Dolly image generation, it gave us this incredible starting point that we could just hit the ground running with.

• 11:26Nathan Labenz?

  So let's dig in on each of those a little bit more. I think, you know, people have seen a lot of cool stuff, I would say, you know, safe to say, if they're listening to this podcast, they've seen some cool AI art. Probably, though, you know, it's often the case with different AI systems that you kind of, you know, to a first approximation, kind of get out of it what you put into it in the sense that if you don't have any vocabulary or expertise in an area, then you kind of get amateurish stuff out. And that seems to be true, both on the image, you know, creative side, and also on the language models as well, right? You know, some of the most creative and interesting projects in both realms

• 12:11Nathan Labenz?

  are predicated on the person, you know, the creator really knowing deeply what they're doing. So I'd love to help people leave this conversation with a little bit better sense of like, how can they? Prompt for consistency or kind of create a world. You know, when I go in there myself, I feel like, okay, I, you know, I want an image of this. It doesn't really come out how I'm looking. I kind of mess around if it's still not how it's looking. You know, I'm wanting it to look in my head, like often just give up or try something else. But to make, you know, I don't know how many cuts there are in this 12 minute film, but it seems like, you know, typical whatever, every two seconds or something. You've got a lot of shots over the course of 12 minutes.

• 12:52Nathan Labenz?

  And so to sustain an aesthetic for a full 12 minutes to make something that ultimately feels coherent. It seems like you really pushed on this frontier of consistency, predictability. So I'd love to hear kind of more about how you conceptualize that and the techniques that you and the team developed.

• 13:12Stephen Parker?

  Right. So I think it's important to kind of imagine in your mind, how you think these images were captioned originally. Because those captions are critical in the training of the data set and are kind of a fundamental first base for how to think about getting images. So before everybody had access, I was taking a lot of prompts from people and people would just kind of give me random prompts and I would help to improve those prompts. Right. And they would say things like, you know, I really want to get

• 13:47Stephen Parker?

  something that looks like a photo, a real photograph of a gray alien skull. You know, so they would just put gray alien skull into an AI image generator and they would get back an illustration or a painting or something that, you know, didn't look like a skull at all, or maybe it wasn't a gray alien. So I would say things to them like, hey, we should, let's imagine this were real. And there were a gray alien skull, you know, where would it be? It would probably be somewhere like Museum of Natural History. Right. It would be in New York and like, I want you to see it in your mind as that thing. And like, now we're going to prompt and we're going to say like, the skull of a gray alien

• 14:29Stephen Parker?

  circa, you know, 1946 or whatever in the Museum of Natural History, something like that really sets up a kind of contextual framing for DALL-E or an AI image generator in a way that helps it to understand. What are we going for? What are we looking for? And where might we find that thing? And so, you know, if you were to imagine gray alien skulls being captioned also inside of something like National Geographic. Right? Like, let's imagine there's a photograph of a gray alien skull held in a museum collection, um, taken in the 60s. Now we want a prompt for that image. That is really the way to think about achieving images for me inside of these generators.

• 15:14Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:18Stephen Parker?

  So if we kind of take the pre-context and then apply it to a production like The Frost, the first thing we really want to do is just kind of set up a contextual prompt structure that looks like, "Hey, what kind of image do you want? Portraits. I want portraits from the film." Then let's give it an imaginary film name. For The Frost, we called it Tundar, just because it's kind of a tweak on the word tundra and felt like something that wouldn't be in there, but you know, was like pointed enough, um, to kind of set the mood. So we want portraits from the film Tundar. Then we're going to add a comma.

• 16:01Stephen Parker?

  And extra commas are a huge part of, you know, crafting the prompt, if you will. But the next section is kind of like, okay, what are we shooting? Right? So we're shooting people climbing a snowy mountainside, right? That's kind of the subject matter of the prompt. And then we put another comma and we say something like, "Filmed by, you know, various, uh, famous like Hollywood cinematographer or director." And now it's kind of like, okay, if you were to imagine the location of that image in a data set,

• 16:38Stephen Parker?

  you know, maybe it would be in a blog post about a film, or maybe it would be like, uh, in an IMDb kind of still or caption, or maybe it would be, uh, there's tons of film sites, right? And so sort of like, Imagine the imaginary caption for this this film image that is really kind of pointing to where that image might be hypothetically and also kind of an art directed structure. We have a structure that's like, hey, I want this type of image from this imaginary project. Here's the subject I want in this particular instance.

• 17:21Stephen Parker?

  And then we can add additional kind of consistency information to the end. So maybe it's like always directed by the same director, or maybe it's not, but we always want like a consistent element to be in our shot, like the color yellow or, you know, the colors blue and gray in the case of the frost. And by setting up what I keep referring to as a contextual wrapper like this, we kind of have a frame around the subject matter. And now what we do is we go in and we play with that subject matter to kind of achieve different shots. So if you go look at the frost and you think of it not as kind of like moving video, but

• 18:02Stephen Parker?

  just like one representative image from each of those video moments, that is, you could basically substitute that out for the subject matter portion of the product.

• 18:12Speaker 00

  That's what helped us get the look of the frost. There's a couple of extra things that I think would help in continuity to answer that question. Like that definitely helped us in terms of attaining that at times seamless look. The whole thing took place outside in the snow. Right? So when we're cutting from shot to shot, that's an easy transition for your brain to make. Like, okay, well, in this shot, it might not be exactly the same background. Those tents might not be the same as it was in that reverse shot, but I'm buying it because there's snow and there's mountains. So that, yeah, that was huge.

• 18:53Speaker 00

  Like kind of creating an environment that, um, that was outside. Cause once we got inside. That's a whole different ball of wax, too. Also, you know, another challenge with this stuff was that um was to try and create like the, you know, cinematic continuity. And, you know, by that, I mean, just like, okay, just the fundamentals of of a conversation. We have a shot and a reverse shot. And that that kind of stuff was the most challenging. So, once we had that look, then we kind of had to go back in and refine. Okay, that's when we had to kind of use our knowledge and apply it.

• 19:38Speaker 00

  Because it's like, okay, you can only have a certain amount of wide shots within a scene. You need to, you need to punch in or you need, you need to, um, you, you need, you need more wide shots. So you need a different angle. You need, you need a profile, you need this. And so like, having that knowledge like of filmmaking essentially was a great help. Because that's what we, you know, once we had our foundational shots, our big master shots that the teams created, then it's like, all right, how do we enter this world? How do we explore this world um in a cinematic way, in a way that audiences are used to experiencing?

• 20:20Speaker 00

  And that's where like the real specific prompting came into play. And sometimes, and you know, and we got mixed results out of that. You know, sometimes uh we get exactly what we want. And then other times, we kind of let Dolly take, take the reins and, and, and got some great stuff there. So.

• 20:39Stephen Parker?

  Right, I think that's a good way to think about it, maybe for the audiences, like, not so much me, but my team, they're like, they're, they're really trying to get out there and dig up the raw materials. And by raw material, like, think of cinematography, think of these shots. That's kind of my side of it. Then Josh's side of it is like making that into a film, like focusing on the story, figuring out like how to build a story out of those raw materials. And then we kind of get into an iterative feedback loop where, you know, he's like, okay, this scene is working or this scene isn't working. I need more reverse shots here, you know, how do we, let's go take a crack at that. And then we try to go mine for some more of that type of shot and bring it back to him. And then, yeah, maybe it is working or maybe it isn't working.

• 21:24Stephen Parker?

  We need to make a tweak. Like definitely the biggest thing in all of this, I mean, that's kind of like a standard flow or could be considered a standard flow in terms of iteration around a project like this, but it Dolly or any AI gen is like kind of an artist on its own. It's almost like a, you know, a cinematographer that you don't, you don't fully control. I, I, I think what's um, interesting about all this is that, you know, we talk a lot about these things or we refer to them quite often as tools. But there's also sort of an artist's component there. There's definitely an unknown with the AI that it's just not fully within your control. And so we have to treat

• 22:04Stephen Parker?

  those AIs as if they were themselves artists and sort of like be willing to be receptive to what they give us in the moment on the day in this shot or any other shot. And then, you know, see what we like, see what new ideas come out of that. That's definitely kind of the most new aspect of this, I would say, because in a typical production, you control everything, you know, you go there, you art direct the scene, you set it up the way you want it, you have the talent you want, you've got a script, exactly what you want them to say, like, you may deviate here and there, you know, as um, encouraged by the director, but fundamentally like you're in control.

• 22:44Stephen Parker?

  In the case of AI image gen, it's kind of like, it's much more like an active substrate that you're trying to sort of manage and mess around with in order to. Create something that fits into your project and works for you.

• 22:58Nathan Labenz?

  If this were real, let's imagine it were real. Where would it exist? Who would have created it? How would it be described? So that you're not just starting off with, you know, a kind of naive, you know, purely descriptive thing, but you're trying to bring in all these other associations and kind of marshaling the, you know, the vast knowledge base that the system can draw on. On the language model side too, and we've had a couple moments like this in the past where, you know, I've been trying to use adjectives to get a language model to do something for me, like write better, you know, copy, right? For our videos at Waymark. And then occasionally you'll say something to me like, why don't you tell it to make it like David Ogilvy wrote it?

• 23:40Nathan Labenz?

  And then I'll be like, oh yeah, that's quite a different approach. But in some ways, way more powerful, right? Cause I'm, I'm usually very like literal and kind of sitting there. Much like your kind of, you know, gray alien skull example, I'll be like, I want concise copy and I want vivid copy and I want memorable copy. And I want, you know, you know, all these sort of, you know, action verbs. And I'm kind of just giving it like a checklist. And you are instead invoking a master, which is, I think, something probably a lot of people could stand to benefit from in really any AI project that they're doing.

• 24:19Nathan Labenz?

  So that's a bottom line that I think, you know, people should definitely take away and incorporate into their own usage. You know, there's this kind of iterative cycle in not knowing a lot about filmmaking. To me, that sounds like pretty profoundly different, right? I mean, I imagine in some cases, especially with all sorts of technology that is used, you can kind of patch over things that you didn't actually film when you were doing the filming, you know, later at the editing stage. But still, I also would imagine that like, you're pretty limited in most cases by.

• 24:53Nathan Labenz?

  What did you actually capture when you were in the capture phase? And it's not easy, right, to go back and like, reset up the set and bring the actors back together. So this fundamentally does kind of change the flow from one that is much more planned out in advance and kind of, you know, waterfall, to use like a software analogy, to now one that is actually much more kind of agile and allows you to, you know, to spin back to the raw materials anytime you want. Am I overstating how different that is, or is that really like a very different reality for making a film?

• 25:30Stephen Parker?

  No, it's, it's, it's super different. I mean, you, you know, you can punch in as sort of as you were referring to on various projects, whether it's, you know, audio or something else with video. But yeah, it's very expensive and time consuming to go back and shoot this stuff. How I would set it up, though, for the audience is like, it's really a trade. Because, yes, it's very iterative. You can go back anytime you want. You can revisit shots, get new material, stuff that, you know, would otherwise kind of cost you a fortune to go back and reshoot and rethink about. The trade, however, is like, you're not fully in control. You still have to take those ideas again from Dolly, from Mid Journey, from whoever. And

• 26:13Stephen Parker?

  yes, you're getting more, but you're also getting more, again, that is not within your control. And this is really where Josh, I think, has stuff to say about, you know, what that process actually feels like coming from, you know, the world of a traditional director to a newer project like this.

• 26:31Speaker 00

  I mean, that was probably the most revolutionary, most eye-opening oh shit moment that I had during this project was when we were actually, we were, we were in the cutting room. We had our, we had our stills. They were still, you know, yet to be animated. And you know, it's like, just like in any kind of creative process, you're kind of searching for more. You want to make it as best as possible, you know, the best it could possibly be. Like, yeah, this is cool. And this shot is kind of cutting with that. And the music is hitting right. And like, we're, you know, these two people are talking and it's a scene. But like, wouldn't it be great if

• 27:12Speaker 00

  this could happen? And then, you know, in it, right? Like you said, in a, in a, in a normal traditional production, even in animation, to go back to shoot is, is a luxury that, you know, is that a lot of people can't afford to do. It's like, you got to work with what you have a lot of the time, especially if it's already in the can. And with Dali, you can, in real time, say, all right, let's get this hand clutching this rope. That's gonna, you know, that's gonna heighten the drama. We want this scene to be super tense, like when the person is, you know, clinging for their life and has slipped off the mountain. It's like, okay, how do we

• 27:56Speaker 00

  heighten those moments? And, and we could literally engage the team for a prompt. They can get us back an image within 20 minutes. And then we could put it in the timeline, watch it down. And no, that's not working. The fingers are messed up. Or, you know, let's, let's pull back even more, have them outpaint it like two times or something. Ah, that's it. Beautiful. And then you have your shot. And you're, and, and that's how, that's how the the polish phase happens, really. It was just like, that's how this thing became polished. And like, I think as, as

• 28:37Speaker 00

  these models get, you know, start to develop even more, we're just going to see the polish get even that, that phase just be faster and, and like be more exhilarating.

• 28:49Stephen Parker?

  Yeah. In terms of practicality there, I think Josh mentioned the storyboard earlier, which we do use. But I think in our case, something that's different is ours is really an active storyboard, right? So we use just a whiteboard website that, you know, all the team members can go and see simultaneously and kind of work on collectively. But You know, we kind of have a row for each scene and images laid out in sequence there. But the, you know, the images can be taken down and new image gets put up or we, you know, make space and put a new image in. And those images are not like a hypothetical or concepted or like an idealized version of what we want.

• 29:33Stephen Parker?

  Those images are actually what we're working with. So we go through this active storyboard phase where without worrying about the animation or anything else, we're just kind of worried about a sequence of stills. And I do think that's not totally different from like an animatic process or something, but it's nice to just be able to do that continually throughout the project. And so as Josh was saying, like, okay, he's decided in the scene, he really wants a tight close up of somebody gripping a rope. All right, seems straightforward enough, but now like the variable part is, is that hand like masculine or feminine, young or old? What does that rope look like? Is it like, you know, a sleek, uh, black, uh, mountain climbing rope of some sort?

• 30:18Stephen Parker?

  Or is it like an old nautical rope from a ship, you know, from the 1600s or whatever? And you're really like getting all of it back at the same time. And you, then you kind of have to decide like, you know, what fits here. And obviously you, you hope to like hone it and get more specific than that, but hopefully that kind of like helps to contextualize or provide an example for people of, even though you've ID'd this very simple thing, like all the variability that comes or like explodes out of that one maybe individual request.

• 30:49Nathan Labenz?

  When it comes to faces, you know, the, the continuity of the characters through the movie, you had mentioned the tents, you know, in the background can kind of change and you can kind of get away with it because, you know, it seems coherent enough and people kind of, you know, don't latch on to those kinds of details. But for things like the characters' faces, I would assume first of all, you know, a lot lower, uh, ability to slip past, you know, the viewer when those things change or are kind of inconsistent from scene to scene. And that seems like it would be really hard to.

• 31:26Nathan Labenz?

  Get consistent across, you know, a bunch of different shots, a bunch of different contexts. So how did you approach that problem of getting these characters to feel coherent across all these different images when in fact, like presumably they're being kind of generated from scratch each time, right?

• 31:46Stephen Parker?

  Yeah, I mean, it was stupid hard in the context of our project, to be clear, because we're working with DALL-E, because it's, you know, back in time in terms of the tech. It like, another like caveat for now is like, this is much easier now within Midjourney, which you can do either by blending images together, which will kind of give you a consistent character, or feeding it, like uploading an original image and then asking for like subtle variation on that image. It's still not perfect, but it's a lot easier today than it was then. In terms of back then, I think we, yes, we really spent a lot of time banging our head against the wall, Josh in particular, just, yeah, I remember, I was all the time trying to ask, you know, like, couldn't we just kind of follow a group and like, let's make

• 32:31Stephen Parker?

  the story about the group so that we don't have to care too much about any particular character, but Josh was like, and I'm kind of twisting his arm the whole time in that direction, but you know, there was this one character, Dr. Ulrich, that he was like, "Look, we like, we really need time with a few characters, you know, you've got to find some way to give me a consistent character." Like, you know, and he was very amenable to many different possibilities for how that might manifest, but ultimately, the thing we kept coming back to is like this idea of an archetype, right? So I think, not to be understated, is just the prevalence and significance of archetypes within these image generators, right?

• 33:13Stephen Parker?

  You can think about a common archetype, like a statue of the Virgin Mary that we've all seen 100 different times all over the world. DALL-E or any AI image gen has certainly like seen, you know, a ton of those statues. You could also think of an archetype in terms of something like the Mona Lisa that's, you know, essentially the same image, but, you know, a million different times in the dataset. There's a lot of different ways to think about this, but we, for the character of Ulrich, the archetype we set up is like...

• 33:43Stephen Parker?

  You know, a gray haired Wiley-esque mad scientist in a white lab coat, you know, with, you know, long gray hair and glasses, something archetypical enough that, you know, it's not the same person, but like if you study the faces, obviously the faces are changing, but the idea is really to kind of just establish continuity so that that character feels like the same character. Josh uses a lot of tricks there, but like at the gen level, we're really working with archetypes and thinking about it in that way, knowing that, again, we're not going to get the same thing. We're just kind of like in an address field of possibility for what the generator may return.

• 34:29Speaker 00

  That was a huge thing with Dr. Ulrich, you know, to try and get that character. There's a scene in the movie where, you know, this Dr. Ulrich character is, you know, making a speech at the, at the United Nations, trying to save the world, and it's action-packed. And when you're hoping to get action and movement and, you know, and kind of meet the bar that's been set by, you know, modern day editing techniques, it's like you kind of have to go there. You have to cut. You have to make cuts. You gotta, you gotta move your camera. You gotta, you gotta play the game. It's already been, that's been set

• 35:09Speaker 00

  in motion. So we were just, we found this, this trophy archetype of like the mad scientist. And then, you know, at first I was like, eh, yeah, it's not the same guy. And how do, guys, how do we get the face, you know, transplanted onto this shot? And it's just like, it's not going to happen, right? And it's like we, but after a while, you know, you take a step back from the project and it's like you look at the, you look at the scene and when the voices, you know, and the voice of Dr. Ulrich was read and that's the, you know, that's a consistent voice and the music is put into play and. I bought it, you know, it's like, oh, it's the same guy. But if you look at it from shot to shot,

• 35:51Speaker 00

  it's not. It's some of them look very different. One guy looks like Santa Claus and he's, you know, probably 30 pounds heavier and the other guy is, is a little more, is a little on the more trim side and, and he's just got a couple of different facial features. But, you know, we're relying on that like suspension of disbelief that's just like very prevalent in, in a lot of movie making anyways. So we're just really leaning on that with, with these characters, especially with that character. Um, cause he probably makes the most appearances. So that was a huge struggle and I'm kind of, I'm, I'm really proud of how that turned out.

• 36:28Nathan Labenz?

  Yeah, that's super interesting. I honestly had not studied it that closely. And, you know, I've watched it a few times, but in retrospect, you know, hearing what you're saying now, it just kind of, you know, it just kind of worked for me. Like, I didn't really notice those variations or think twice about it, frankly. Human perception is actually kind of accommodates a lot of things. Like, we're not really trained to, to be on the lookout for things like this. I mean, it, there's a really interesting, you know, potential lesson here for kind of human AI interaction in general, right? That like, certain things that we're not primed for are really easy

• 37:06Nathan Labenz?

  to slip past us. We've, we've got no prior context or, you know, reason to be on the lookout for a person to be like, not the same person, you know, from scene to scene. It just doesn't happen. So in reality, then you can actually slip a lot past somebody because they're, you know, they're kind of prior, so to speak, on that happening is just so low that, you know, it doesn't, it has to be pretty egregious, I guess, or maybe not pretty egregious might be too strong, but it has to be significant to get over a threshold where people would see it if they didn't specifically come in looking for that.

• 37:44Speaker 00

  I think it speaks to the power of the story, too. You know, not our story, but just the power of storytelling. And when people are engaged in a story.

• 37:56Speaker 00

  You could forgive a lot of things. You know what I mean? It's like, there's a whole, there's countless Reddit boards dedicated to like, you know, gaffes in movies, you know, where you could see the, did you spot the boom in the shot? Did you spot like, this actor wasn't the same actor as in this scene? It's just like, a lot of that stuff, we don't catch just because like, if you're watching a story, you want to be engaged, you want to feel, and so like, that's, that did nothing but help our cause with this. It was a cool thing to see put into practice.

• 38:30Nathan Labenz?

  So, going back to your comments about the, you know, the shot with the rope, right? What kind of hand is it? What kind of rope is it? All these little details. How much of that are you accomplishing through iterating on a single image? Like, people have, you know, have seen these kind of mask and fill techniques, or outpaint techniques. When an image actually gets to the point where you're going to use it, you know, how often is that something that like, Dolly spit out and you're like, great, we'll use it, versus, you know, it spit something out, but then you went and, you know, redrew the rope five times, and, you know, the hand five times? How much of that kind of image level, you know, partial editing and reworking are you doing?

• 39:14Stephen Parker?

  Quite a bit, quite a bit. I mean, most of the images have some, particularly with Dolly, because, you know, those images are starting out so small, and then kind of scaling up from there in terms of the way the images are generated, you know, you're often like going back and correcting an eye, or, you know, painting over hands, or something in order to try to get a better result. The rope example is a quality example, in terms of adding a new shot in that you want, but I think in many, many, many of the shots, there's just some aspect of it that you want to touch up or make different, or, you know, sometimes, Dolly just does crazy things, like, it'll just throw a random object into the background, you know?

• 39:57Stephen Parker?

  It's kind of a fun, like, creative occurrence, but also when you're...

• 40:04Stephen Parker?

  Looking for consistency and there's like a steel pallet or something in the background that just like has no reason for being there. Then Josh is making notes like get rid of this, get rid of this, please change this. Another example is like, okay, these outside shots are great, but I'd need a fire. Like, can I get some plumes of smoke? So, things like that are happening all the time. And then, in terms of like, the real cheating Photoshop work that we're doing in this process, like, the one thing we really needed was this sort of MacGuffin object. I think of it that way, anyway. And it was this idea of a transponder that is sort of moving up the mountain with these characters over the course of the story. And so, in order

• 40:48Stephen Parker?

  to achieve that, we went into Dolly to first create that object, you know, went through many, many, many sort of prop iterations on what that thing might look like. Once we had it, though, in a few different angles, then that is an example of an object that literally gets composited, or you can think of it as stitching it into various shots over the course of the film, so that it's there. So that we have that kind of one little consistent object that can follow you through the film and sort of help add another layer of continuity.

• 41:23Nathan Labenz?

  Can you unpack that notion of a MacGuffin for folks that aren't familiar with it?

• 41:28Stephen Parker?

  The MacGuffin is kind of the idea of an answer, an object, or as something that isn't really there. It's more the idea of it that's there. So, I'm not exactly using it in the correct, like, in the context that it would, you know, most often be used, in terms of like, a film where people are searching for a thing, that's the MacGuffin, it's the answer, it's the object. They may find it or not find it. It kind of gives meaning to the story without providing like substantive meaning itself. The transponder is similar in that it's not a real transponder. It's not really there. It is something we're just kind of, you know, stitching throughout the film.

• 42:12Stephen Parker?

  I think of it that way because like it's not really coming from the generator in place. Right. Like we're putting it in in order to add this kind of idea of consistency to the project. So maybe I'm equivocating a bit there, but it's how I think of the construction of the film.

• 42:31Nathan Labenz?

  So another kind of challenge, Josh had mentioned this kind of, you know, shot and counter shot, or, you know, was it reverse shot, you know, characters in dialogue, you're going to kind of see over each one's shoulder, right? People can recall how that often happens in the, in the movies and TV that they watch. It strikes me that, you know, if you want to go back and like replace a rope, there's pretty good tools to do that, right? Where you can mask it out and try it again and, you know, iterate on that. But for broader composition, that's obviously a lot tougher, right? You can't just like mask, you know, simply you're masking out the whole thing is like, yeah, the layout, you know, the whole kind of

• 43:14Nathan Labenz?

  composition of this, you know, if it's wrong, it's kind of wrong. You can't just kind of locally change that. So how did you work on that? And how would you describe kind of what the tools can and can't do there, right? I mean, there's, there's actually like benchmarks around this where people are like, you know, can you, you know, create an image where there's like, you know, one blue circle and a red square and the red, you know, square has to be, you know, below the blue circle, whatever, these kind of compositional, you know, sort of somewhat stilted, but, you know, clearly defined compositional tests. Yours are obviously much more kind of in the eye of the beholder in terms of whether they're going to pass or not. But I mentioned that must have been a real challenge and you know, what do you do to

• 43:58Nathan Labenz?

  try to dial that in and, and you know, get the hit rate up to an acceptable level?

• 44:02Stephen Parker?

  A couple of things to think about there. One, it's always getting better, you know, like the ability to prompt for unique things and unique situations, which I think is ultimately what you're getting at there is all the time improving. So I would just put that upfront. Two, Josh is asking for any number of things. He's not just asking for small like tweaks to a hand or a rope. Like he's very often saying like, I want a different background here or you know, I like, I like this character from the image. I want them combined with another AI image. And so that, you know, there is kind of like some compositing happening at the level of like putting images together. If you want to think of them almost as collage, like from the Dolly generated images, that's another way you can think of it. But then I guess like

• 44:48Stephen Parker?

  at an artistic level, there's multiple things happening as well. One thing is like sort of knowing what form looks like to begin with is going to give you a much better idea of where you should be, you know, erasing, if you will, in order to sort of force a better image to arrive based on the amount of space you've taken away. And that is like also contributing to the available space to paint back in. So

• 45:17Stephen Parker?

  if you want to refine a form, you know, thinking about it as an artist, there's like definitely a structure or a method of attack in terms of how you are in painting and out painting within an image. But then also you have to go back to the data set and think again to the data set. This is like, I remember we had a number of shots that were daytime shots that Josh wanted to see fires in. And they were like overhead shots of the camp. He wanted fires to be there. And it's like, you know, if you just go do a survey of film or go do

• 45:57Stephen Parker?

  a survey of this type of image, like there just won't be that many examples of images that are like daytime with a brightly burning flame as well. Like you might get that at night, you might get that in other contexts. And so a lot of times it feels like you should be able to just ask for, you know, object X, a fire right here in this place, but All of the informational context surrounding the place where that object goes is contributing to how the machine is going to paint into that space. And then that's also coupled with like the sort of contextual information that lives in the data set. There are other times like the color grade of an image.

• 46:41Stephen Parker?

  I remember we were trying, we had shots that were very, very, very blue. And we were trying to paint in something like a red scarf or a purple scarf or a yellow scarf. And like all of those colors, there is no way that after a film has been graded, that that color is going to show up bright in the final image. So there are many, many instances in film where like someone is wearing, you know, a bright piece of costume or whatever. But after the grade happens, the color grade, those colors become much more faded. And so Dali is never going to paint in like an electric red or an electric purple into that scenario.

• 47:21Stephen Parker?

  Because it's never seen any examples historically in terms of the way lighting works in these images. That is, you know, going to give it any indication that that should happen. That's an interesting mystery in terms of where we are with the training right now is kind of like trying to twist the arm of these machines to like reach into new or novel spaces to create things where there really isn't great kind of pre-context. Around what those things should look like in the images. That also is a bit of a tangent, but like it's caching out in many ways when you're interacting with these generators.

• 48:00Stephen Parker?

  If an idea is complex, well-understood, but maybe like without pre-context, you're going to get a fuzzier image. You know, it's almost like the machine is having to think about a lot of different things at once. And so like, you know, you're really kind of getting something that's maybe a little fuzzy or maybe a bit more painterly. When there's a really clear idea of what that thing is, it's almost like. You know, the image dials in to something more specific, much more detailed. All that's a long way of saying all these things are kind of happening at the same time.

• 48:39Stephen Parker?

  And at an interval level, we're kind of just listening to what Josh wants and doing our best to get that thing. But we also have to be conscious of what is possible in this image, given what it sort of already looks like, such that we don't like step wildly out of bounds and make a novel request that's just kind of impossible for the machine.

• 49:05Nathan Labenz?

  I mean, I think this is another really fundamental point for different kinds of AI systems as well. Right. It's like the notion of can AIs have like breakthrough insights? You know, can they have eureka moments? Can they suggest like science experiments that are actually, you know, worth running? Because there's enough insight behind it, you know, that makes it like a not previously kind of well trod, you know, hypothesis. In this case, you know, can it create stuff that like, doesn't look like anything in the set? In general, my read is like, that's very tough and, you know, almost vanishingly rare.

• 49:48Nathan Labenz?

  Certainly, you know, when I've looked for things like, can it come up with good science experiments ideas? You know, with current systems, I would say not really, you know, it can take a good science experiment idea and start to map out the steps, but I haven't seen any examples where it would actually give you a really good, you know, science experiment idea upfront. But I guess, you know, so in the context of this image stuff, like we have seen things like the avocado chair. How do you, how do you make sense of something like the avocado chair? Which, you know, maybe that exists somewhere out there, but you know, that's pretty vanishingly rare. I don't know, I'm just kind of trying to get a little bit better understanding of

• 50:27Nathan Labenz?

  what are the kinds of like never before seen things that it can do versus the kinds of, you know, never before seen things that you're still feeling like are impossible?

• 50:38Stephen Parker?

  This is definitely at the forefront of a lot of different machine learning research and insights being worked on, yet to be developed, etc. I don't have the definitive answer on this. You're going to get lots of different answers from lots of different people, but I think there is important thinking like within this realm of questioning. So the way I think about it and the way I appreciate these image generators the most is really celebrating the fact that there was a data set that contributed to them. You know, I don't like to interact with these machines kind of apart from that thought and that appreciation. At a high level, this is really like

• 51:20Stephen Parker?

  an exploration of human history that I feel like I'm taking part in. The way I think about that sort of metaphorically or hypothetically is by thinking about that latent space and thinking about kind of a 3D space where a lot of points exist. And I think of those points as sort of all the things previously created, all the things that exist within that data set. Another way to think about it is maybe like a library of Babel analogy or a space where all possible things exist. But if we were to conceive of a thing, like a space where all things exist, then the data set is like only the things we've found within that space of possibility.

• 52:04Stephen Parker?

  And then what we're actually interacting with, with this generative approach is really the space between that set of coordinates. So.

• 52:17Stephen Parker?

  To reframe it, within a space of all possible things, there are all the things we've found. And now we're interacting with the space between all the objects we've found. So in the case of the avocado chair, like, no, there weren't a lot of avocado chairs, but like, there are a lot of avocados, there are a lot of chairs. There are also a lot of avocado pillows. There's a lot of like, there is a ton of different chair designs that look like many, many things. Certainly, many of them have like, been close to the shape of an avocado. And if I were like, thinking about the, the space between those points where like, an avocado comes close to a chair, like, probably that is like, occurring many, many, many times. The fact that a machine is able to,

• 53:02Stephen Parker?

  you know, create an inference point between those two and manifest it as like, an image of a chair is not that surprising to me. And there are a lot of cool examples where that happens in ways that just like, we haven't thought of yet. And that's kind of like where the novel happens here, right? It's like, what happens when we blend two, three, five points that we haven't really blended before. That's the sort of everything is a remix philosophy. That's the idea that, you know, we're getting to new places by existing places where we've previously been. And there is great opportunity for that to happen. There are a ton of like, just purely abstract kind of

• 53:46Stephen Parker?

  artists who are producing novel, I would say, just by kind of like, putting unique concepts together. But I think that is very, very different from asking the machine to like, fundamentally travel outside of the space of points that we know about and get to a new point, a new location, a novel location. That really doesn't happen. And it's my understanding that that is not really like,

• 54:16Stephen Parker?

  The space we're working with when we interact with a ChatGPT, a Dolly, or a Midjourney.

• 54:23Nathan Labenz?

  I still a little bit struggle with like, why can't the scarf be super vivid red? Or, you know, just things like that, that seem like, that doesn't seem too conceptually crazy. Maybe it's more of just a, you only have so much like a bandwidth, right? You only, you only have so much data you can communicate to the model in your prompt. And so it kind of has to have, I don't know, I'm struggling though, because it's like, you can get avocado chair. Seems like you should be able to get that bright red scarf if you want it, you know? Even if it's like, not super coherent with the rest of the shot.

• 54:58Stephen Parker?

  Yeah, I mean, I'll just jump in for a second, like the pre-context though of the prompt is important there, right? Like in the case of an avocado chair and really with any prompt, the less kind of specific it is, the more opportunity there is for like, what I would call open blending, just, you know, intersection between points or like force intersection. The more we contextualize it, the more we say like, oh, I'm looking for cinematic 8K portraiture from a hypothetical film, you know, by this director or lit this way, like within this context, like, we are triangulating into a tighter and tighter realm of the space, right? The avocado chair is like a great example of, if I could pull from any image in this data set and put it together with any other image in this data set,

• 55:44Stephen Parker?

  in this like, you know, um, mathematical mind of the AI, then a lot more freedom is available to me. And also to be certain, like, the mid-journeys of the world are all the time doing a better and better job of being able to make ad hoc demands even within context for things like the bright red scarf in a very, very blue cast image. So that's kind of happening, it's happening through RLHF, it's happening through a bunch of other techniques for the type of images that get generated and we are still developing and figuring out like arm twisty ways to like have these machines generate the sort of images that we want to see them generate.

• 56:26Stephen Parker?

  But I also think it's worth thinking about like, being in the space, vectoring into more and more specific places in that space and then like contextually what that means for. You know, what the AI sees in terms of like its its local area of what it might grab from. It still does cool stuff all the time, stuff that's really creative. I mean, Dolly trolls with the Mona Lisa continuously. And I don't know that people know that if they haven't, you know, generated a hundred thousand images with it or something like that, or they're not looking for it. And maybe, you know, maybe the developers toss that in from time to time as, you know,

• 57:09Stephen Parker?

  a pre-pen or an append to the the prompt. I don't know. Maybe Dolly just does it because it's seen the Mona Lisa so many times that like, you know, it will throw the Mona Lisa in here and there. But, you know, like, in the case of Dolly, Mona Lisa is really kind of a character, a ghost in the machine, if you will, that just like shows up in all kinds of different places and contexts. Maybe as a background character, as a photo real version, as a like photo bomb sort of character. In another instance, as a drawing, as an illustration, as a sculpture. In many, many different scenarios, Mona Lisa just kind of pops up in the most creative ways. And that's just one example, but, you know,

• 57:53Stephen Parker?

  it is really, really fun and funny at times to see that level of creativity is sort of coming from the machine. That's happening all over the place. The suggestions that the AI is making through its lack of specific knowledge are also really, really cool. Like, I see a lot of clothing that the machine conceives of that's kind of like a mix of two different ideas, you know, it's a button down shirt like yours, but maybe it ends, you know, right where that top button ends. And, you know, it's a poncho from there on out, you know, and it's kind of like, you know, the machine has no reason to care. You know, even though we talk about kind of bias in these machines, at the art level, it's really willing to kind of

• 58:38Stephen Parker?

  mix and match and pull from so many different things that like.

• 58:43Stephen Parker?

  It's really creative. It's really interesting and it's really unique at times.

• 58:49Nathan Labenz?

  Fascinating. Okay, so just being somewhat mindful of time, I want to talk a little bit about the motion techniques that you guys used in this project. Then I kind of want to get a little bit of an update on like where you're going next. I understand there's a Frost 2 in the works and you know, as you've mentioned several times, the tools are improving and the you know, what's available is changing and you're probably going to use significantly different tools and techniques for the second installment. So really interested to hear about how that's going. And then I maybe just to conclude, we could kind of zoom out for a second and like, talk about the big picture of you know, what all this means for content creation in general, you know, content consumption, you know, it seems like it's going to have

• 59:34Nathan Labenz?

  a significant impact. So I'd love to hear your speculations about that. But with that roadmap, tell me about the motion. You know, one comment that we've heard from a reviewer is that the vibe of the film is, I'm quoting, grotesque and unsettling. And I think, you know, that obviously plays nicely with the story that you guys are telling. I wonder to what degree that is kind of a, you know, is that was that like an early, late 2022, early 2023 constraint where you were kind of like, hey, this technology is still kind of in the uncanny valley. Let's just make it, you know, uncanny valley and grotesque

• 1:00:17Nathan Labenz?

  and unsettling because it plays that way. Could you have made something that like, you know, wasn't grotesque and unsettling? And I'm understanding that that is in the motion layer. Correct me if I'm wrong with that assumption, but it seems like Dali 2, you know, is spitting out pretty realistic, not uncanny valley images. And I'm guessing it's the motion layer that was kind of introducing this sort of this vibe. But you know, correct my misconceptions there.

• 1:00:45Speaker 00

  Yeah, I the motion was.

• 1:00:50Speaker 00

  Was was a challenge, right? Because we have these very cinematic images that feature human beings and human beings move a certain way and human beings have appendages that look a certain way and you know knees bend at a certain angle and elbows bend and fingers move and sometimes Dolly doesn't want to give you all of those appendages in anatomically correct places that like would you know beget a traditional human movement. So we figured out quite early that our movement was at least with the humans, the human characters

• 1:01:32Speaker 00

  was going to be a very rudimentary look, you know, like it looked at at especially with the climbing up of the mountain, that was probably like where the motion was most evident and was going to be featured the most, like you needed to show, you need to show them moving up the mountain. That was done in a couple ways that I can go into. But when we were using the Dolly generated images, you kind of had to just go the coarse puppeteer way in terms of after effects, like literally like taking, taking an arm, moving it, you know, very time consuming way and or a knee and moving it up and then

• 1:02:16Speaker 00

  down. And with our, we leveraged our amazing animator, I want to call him out, his name is Matt Sessions, he's been working with Waymark for a long time and he did a tremendous job like breathing life into this project with the with the motion. So I definitely want to want to call him out. We were playing into the into the characters, we had no choice, you know, this is what Dolly was giving us and we had, this is what we're working with. We were working with characters that, you know, might not have a hand or might not have a complete foot. So it's like, how do you animate that? You know what I mean? He's like, well, you gotta, you gotta keep, keep, keep them climbing up the mountain, so to speak. So we were doing our best there. And also, like, in terms of the climbing,

• 1:03:01Speaker 00

  like, we were still like, really inspired by like the Akira Kurosawa movie. The Blizzard, which is featured in in like the anthology piece Dreams, which is like just now being remastered, I think this or released this week by Criterion, which is really cool. But like in that piece, like the characters move a certain way. It's a very sluggish kind of movement that that like we felt good about. It's like, okay, like, you know, Kurosawa is is doing this like up the mountain like we can do it. So like it was it was a great inspiration to have. And also something that I was looking at when I was seeing storyboards come together.

• 1:03:44Speaker 00

  And then I was seeing the early animations is like, it was just coming off as like, this is a graphic novel come to life. And, um, you know, sometimes you just you just want to see the graphic novel. You want to see the comic book just move a little. It doesn't have to be realistic. It doesn't have to be, you know, like in 24 frames per second movement. It's just it's just you want to see a little movement and like and and we're good. And that that goes a long way. Um, and that's kind of how we got that look. And, you know, whether it's grotesque or not, like, that's up to the viewer to decide. I'm not going to comment on that. There's also different other motion animation that we set into into play

• 1:04:27Speaker 00

  to really breathe life into this thing, to get the kind of scope that we wanted in terms of like a group of people moving up a mountain and not like really concentrate on moving every single appendage. It's like it would be too time consuming. We kind of we we utilized a a 3D models from Mixamo, which are basically like 3D characters that you can like buy and and and and animate in a kind of a coarse way and and like intersperse them throughout the frame. So like that helped us give us the a little more fluid motion throughout.

• 1:05:09Nathan Labenz?

  So do you think you could have done like a different genre at this stage? You know, I'm kind of wondering like, what if we came back and said, all right, we want to do a romantic comedy with the same tech? Would that just be impossible, given?

• 1:05:24Nathan Labenz?

  You know, the sort of limitations of how realistic, you know, is it, is it, would it be possible to make somebody, you know, the object of romantic desire with this, with this tech? Or is that just not quite there yet?

• 1:05:35Stephen Parker?

  I mean, maybe if you just want to do it as still images, honestly, like we can get something very photoreal, but I think like, you know, that's a very good question. Like I'll let Josh jump in, but I mean, my immediate reaction to romantic comedy is like, I can't think of anything harder to try to do where like subtlety of facial expression, where like so many back and forths between people. Yeah, I mean, we're definitely afforded a lot of things by the genre, by the scope, by the kind of interest in world building that we had around the project. A much more character driven thing would just be really, really difficult right now.

• 1:06:14Speaker 00

  I agree. I mean, one of the biggest challenges of this piece was to try and mine, mine emotion out of these characters. You know, it's like, we could get like really amazing, you know, photorealistic frozen nomads, nomads or whatever. And like, but they'd be, you know, like, for example, like when we were, when we were building the, um, the avalanche scene, like, all right, we need people in here looking up, looking up at, you know, up at the mountain. And sometimes it would just give us people looking up, you know, in, you know, like, like a romantic gaze, even though in our prompt, it was just like in utter terror. They are, they are, they are about to die and, you know, panic, exclamation point,

• 1:06:56Speaker 00

  exclamation point, like trying to throw everything at it to try and like, you know, receive something back that resembled like a human being in distress. And sometimes like, we just wouldn't get that back. You know what I mean? And we had to go in there, um, and manually sometimes, and you know, tweak eyebrows, tweak lip position. And, and what's, what's really cool, we started implementing at the very tail end of the project was, um, you know, Photoshop has like, has like a little AI emotion element to it. So we were able to kind of tweak, you know, some facial stuff with using Photoshop and also some, some after effects techniques as well.

• 1:07:39Speaker 00

  And also with in painting, you know, there's a lot of in painting in this, trying to get. Trying to get a smile, trying to get a frown, trying to get that. So I think, yeah, to answer your question, um, a, a romantic comedy might prove to be difficult. Now, I mean, there's this new tech that we're just starting with the Frost 2. I don't want to get, I don't know if we want to talk about that.

• 1:08:03Nathan Labenz?

  I want to talk about it, if you're open to talk about it. And, you know, in, not to like spoil the story or anything, but just the, the fact of how fast things are changing. It hasn't been that many months, you know, since you were doing the first version and released the, the first edition of this. And tools are changing, things are getting easier. So I'd love to hear, you know, what are the big advances that are making the second edition, you know, different in terms of your process and different in terms of the possibility of what you can create? What are you using tool wise? And what more can you create than before? Voice is another one, too, that I'm really interested in. And we've seen, you know, it seems like we're hitting some thresholds right now in terms of the viability of like deep fake

• 1:08:47Nathan Labenz?

  voices for better or worse. You know, would you consider a, a AI voice, you know, for the next generation? I mean, there's a lot of, um, different frontiers, too, that could be considered.

• 1:08:57Stephen Parker?

  Josh just put together a, a trailer for the Frost 2 and I'll let him speak to all of the, the new tech there. And, um, I think maybe just, uh, short preamble there is, we obviously started creating the Frost at a time when the tech to us was much more new. And then an interesting phenomenon working on the project is like, we're totally eclipsed by the tech by the end of it. Right? Um, we also, the Frost ended up being a way bigger project than we thought it would be initially. Like, we imagined it as, you know, just a couple minute piece to begin with, and then ended up with like a part one running at 13 minutes by the end of it. So we kind of have this thing where like, our first pancake

• 1:09:42Stephen Parker?

  ends up also like being the whole branch. And it's like, how do you do that effectively? Um, it's a bit kind of like live iterate and react for us, uh, you know, over the course of time. But one of the things lately, uh, we've started to I think like become attracted to is this idea of the Frost as an IP, a recognizable IP or storyline that people hopefully at some point become familiar with and then can watch sort of morph through these AI technologies as new tech comes along. And I think that's just, you know, for us, we like that idea because the world of AI is kind of moving so quickly.

• 1:10:22Stephen Parker?

  It's like, you know, you only kind of get one chance very often to see a thing, think about it, and then it's on to the next, you know, 10 things on the day. So the idea of an intellectual property of some sort, a storyline that you can sort of follow along with and also experience new tech with is becoming a really cool idea to us. So I'll let Josh talk about that new tech.

• 1:10:50Speaker 00

  Yeah, so for the Frost 2, we, at least for the trailer and for a lot of our tests that we're that we're running, we're using Runway Gen 2, which is basically a text to video model, which is something, you know, that when we were making the Frost back in, you know, starting the Frost back in January or December of last year, we didn't even think it's like, oh, that's, that's two, two years away, whatever it is. At least I was thinking. But I mean, it's here. And I think like as, as we were finishing, like cutting this trailer, as I was finishing cutting it, like they just released a newer version of Runway Gen 2 where instead

• 1:11:35Speaker 00

  of giving you a four second output, which is kind of limiting, it gives you an 18 second output, which is plenty of time to give any filmmaker to tell, tell the story of a shot. And or, or a scene, if you want to just keep that one, you know, keep that one shot running. So or just like it's, it's a, it's tremendous advancement. And it seems like it's only getting better. But I mean, just interacting with that was, was pretty different in that.

• 1:12:12Speaker 00

  The images were moving, you know, we're, we're not longer getting still images and then breathing life into them like the life is already being, you know, breathed into them. And so it's like there's a, there's, you know, with that comes more issues. It's like, no, well, now this person's head is, you know, resembling a tomato more than it is like a humid head. And yeah, we can't use that shot anymore. So it does, it does different weird stuff, you know, it's like you're counting different, different little weird, weird quirks that it has. But like, you know, I feel like that'll get ironed out and like as, as we work with it more, we'll be able to kind of articulate

• 1:12:56Speaker 00

  our best practices there. But uh, yeah, it's just super exciting to, to have worked with that.

• 1:13:01Nathan Labenz?

  So the new technique is now just straight text prompting with Gen 2. That's the focus at present. Yeah, that's interesting.

• 1:13:08Stephen Parker?

  There's also images involved in those prompts at times too, which, you know, it's cool. Cause like you can go over to a Dolly or mid journey, like craft sort of the image that you want runway to use in order to create that scene. And now you can do a couple of things. You can just do a pure text prompt. You can do that image plus a text prompt to sort of like ask for the type of movement you want, or you can just use that image only and just see what runway spits out as a result of that image. So it's kind of three things available to, to there. Now in painting, as Josh was saying, like in painting, out painting, these things are, I mean, they can

• 1:13:48Stephen Parker?

  be done, rotoscoping can be done, but like it's way more taxing to do that than typically to just kind of like prompt for another generation. So there is kind of less flexibility at the level of video output. But you know, it's kind of a We're building a flow now where we're sort of like achieving the image first. Uh, in many cases, not all, but in many cases, um, and sort of using that as a starting point for the generation of these clips. You also mentioned AI voice, Josh, you are using AI voice.

• 1:14:27Speaker 00

  When we were voicing Frost Part One, we auditioned, um, a ton of AI voices just, just because there's so many different characters and it's like, you know, it's a job to go in and, and, um, and audition people, hear different clips from people, reading lines, it's a whole, it's a thing. And, um, so we were hoping to maybe employ, uh, some, uh, some AI VO and it just wasn't, it wasn't happening. It wasn't, it didn't sound real. It didn't, it there were just, it was too quirky. It was too robotic sounding, whatever. Um, but now during, you know, our time with the Frost Two, it's just like

• 1:15:06Speaker 00

  the advancements are significant, especially applying them in a dramatic context. You know, I don't know anything about like a, anything outside of, you know, a drama, but within this dramatic context, it's like some of them really, really excel. So yeah, we were blown away.

• 1:15:28Nathan Labenz?

  What are the tools that have jumped out to you the most right now in the AI voice?

• 1:15:33Speaker 00

  I like Eleven Labs.

• 1:15:35Nathan Labenz?

  Their pro voice clones are insane recently. I have been just amazed by that. I'm waiting for mine. Uh, that I can, you know, delegate the whole podcast, uh, venture to the AI as well. Also Play HT, you know, former guest, uh, on the show, they just released something in the last few days that I think we, we really want to check out too, because man, it is. You know, to be able to prompt emotion on top of just the text, I don't know if if Eleven Labs has that, you can tell me if they do, but the ability to go in and be like, you know, this is said in anger, you know, versus said in surprise versus whatever. I mean, just a whole new layer of control that has just, as far as I know, just come online

• 1:16:20Nathan Labenz?

  with their last release. So it's, you know, your toolkit is expanding exponentially right now.

• 1:16:27Stephen Parker?

  Yeah, I mean, it is a bit of a sign of it. Like, it is really, really nice as artists to start to see these tools begin to incorporate more of a toolset that we need in order to kind of like get emotion out of things. And you know, like Josh was talking about trying to mine for emotion side of Dolly. I really think a lot of that is like, a lot of the emotion has just kind of been RLHF, like to death out of these data sets. And it's understandable at the level of like, you know, creating images that feel scary to people, faking real world events, disasters, that sort of thing. But like, at the level of drama or action movie or sci fi, you know, like, you need people to look scared at times. You need

• 1:17:13Stephen Parker?

  to be able to put explosions into the scene. You need people to sound angry in their voice delivery or happy. You need them to be able to cry, argue with each other, love each other. Like, the full range of human emotion and visual expectation, like, we need that as artists in order to create effectively with these tools. So I think, like, we're starting to see it. There's a delicate balance there. I can understand, like, why it's not there initially, but we're spending so much time talking about what's in these data sets. We don't really spend a lot of time talking about what artists really need in order to, like, get the full range of capability out.

• 1:17:56Nathan Labenz?

  I think that's maybe a good place to transition to the last thing I wanted to ask you guys about, which is just the broad future of all this stuff. I mean, obviously for context, You know, we've got multiple Hollywood strikes going on right now, writers and actors. And in general, people are, you know, kind of expecting a lot of disruption in this space. I think, you know, you could look at that from the standpoint of production, you know, what roles, you know, become more or less important? How does the mix of jobs change? How do the budgets change? Uh, I think it's also really interesting to consider like what gets created, you know, and how does that change?

• 1:18:37Nathan Labenz?

  I think a, uh, an important point that, uh, our good friend and CEO Alex always says is like, we just never would have done this, you know, otherwise, right? Like, we didn't have the budget to do the traditional production, uh, and never would have dreamed of it. So this is something that just simply could not and would not have existed before. And that's an important, you know, an additional layer to, to bring to the whole thing because it's not just about the way things are produced changing, but also what can be produced changing. But there's just so many different, you know, fallouts from this economic, cultural, you know, atomization perhaps of people kind of falling into their own.

• 1:19:18Nathan Labenz?

  I mean, we're already, you know, very concerned about echo chambers and information silos and, you know, algorithms, you know, curating content just for you in a way that, you know, may or may not always be healthy. But it seems like there's also, you know, a lot of potential for things to be created, you know, just for you in the future. So how, you know, you guys have been really at the forefront of this. And I wonder what, you know, if you try to peek around the corner, you know, beyond Frost two, you know, but kind of extrapolating that to three, four, five. But, you know, where do you think this goes? Or can you even, you know, predict that at this point?

• 1:19:59Stephen Parker?

  I mean, um, a few things there. We did this for fun. You know, we're not a film production company. We're a technology company at Waymark. We're, you know, our mission is fundamentally about, uh, people being able to make their own commercial, you know? So we, we were able to pursue this project really as a luxurious byproduct of the fact that we're out there kind of looking at new tech on behalf of our customers and trying to figure out, you know, what's the best, uh... Ways are there are for us to like incorporate this stuff and and be the best practitioners we can so that you know when we do have times where we need to like construct a prompt or something in our own workflow in the case of our customers

• 1:20:39Stephen Parker?

  we know how best to do that, right? So this is really about us kind of understanding what's possible and that that's really like what got us to the Frost and what gives us this kind of fun area to play in. That said, like because this is fun, because this is not for profit, because this is kind of just, you know, to share and hopefully share a lot of knowledge as well about, you know, the creation and the process. You know, my own opinion is just kind of like, I think there's still room for a lot of things. There's still kind of like room for all of us. You know, the tech is certainly going to transform roles. It's going to create a lot of new

• 1:21:22Stephen Parker?

  cool and interesting roles that we haven't thought of yet. It's going to enable, you know, much, much smaller groups of people the pursuit of these projects. I, you know, I would like to think of that as a lot of people without means previously to become filmmakers or do things creatively or, you know, gonna have the capability to do that very, very soon. And I think that is a a very exciting idea. I also think some jobs are going to go away. Certainly, that's kind of just the reality of technology in our world. That's kind of couched at the level of artist. I don't necessarily think I buy into that argument so much. I think like,

• 1:22:04Stephen Parker?

  yeah, maybe if you have only one modality of art creation and you never plan on expanding that or trying something new, then okay, like the, you know, there's going to be issues for you down the road, but if you embrace these tools and think about, you know, the use of them in your own workflow, I think just about every artist Can find something cool and find new paradigms and new possibilities for themselves. I think, uh, like, you know, the ability to fine tune now in AI on your own work. If I were, you know, an individual fine artist, who maybe hasn't given this a shot yet, or hasn't really thought about it. The idea to go fine tune on your own work and then kind of explore with the AI

• 1:22:48Stephen Parker?

  within the sort of realm of what you're already doing, sounds like a very new and exciting thing for me, uh, to think about. Also, you know, when we prompt for things like a specific artist or a specific production type, we're only kind of getting an amalgam from the AI that's based off of, you know, a myriad of things that's seen in the data set. We're, we're getting results back that like have an artist noted properly. We're also getting results where an artist isn't noted properly. Uh, and so like we're really getting a melange that like, though it may say one artist like represents any number of people. And in the case of film, like is representing, you know, everyone from screenwriter to grip to,

• 1:23:34Stephen Parker?

  you know, just armies of people untold that are represented there in the data set, but are not like equivalent to the artist's name that someone might put into a prompt. So I think that is an important thing for us to consider and then maybe to build on, you know? Like, I'm excited about the idea of, uh, you know, directors coming along and, and authoring definitively a fine tuning that other people in society can maybe purchase or license the ability to use, you know, what they like, exactly what they wanted the machine to understand their artist thumbprint as.

• 1:24:13Stephen Parker?

  And, um, for people to be able to like mix and match those definitively authored fine tunings with others. I think that is a new kind of like, like economic explosion potential that.

• 1:24:28Stephen Parker?

  I'm really trying to talk about right now more openly and be

• 1:24:35Stephen Parker?

  a big advocate for because I think that is maybe like a healthy new world where artists and creators can intersect and find a lot of benefit. And then last thing I'll say, and then I'll shut up is just that I in terms of what I put out into the world versus what I've created, I'm going to hit a million images generated this year here in about two months, running my numbers. You know, I put out a small fraction of that into the world in terms of what I let people see. That's okay with me. I have my own like worlds that explore I explore my own kind of stories I enjoy by myself and that is going to happen for a lot of people in a lot of ways.

• 1:25:19Stephen Parker?

  Going forward and I think we can just kind of try to be a bit more understanding about that and appreciate that for what it is.

• 1:25:30Speaker 00

  That's a hot topic right now. Um, I have friends and family in Hollywood who are, you know, worried about this and ask me about it all the time. So it's definitely controversial. I could tell you from my experience that you know, directing this movie and and trying to see this production through, this is not easy. This stuff, it was exceptionally hard, maybe even harder than like a traditional animation. It just because of there's just you're you're working with an unknown artist in the room who can give you can give you exactly what you want or who can give you

• 1:26:15Speaker 00

  some random wonderfulness or some, as other people say grotesque image. And so it's a huge challenge and I do believe that like there are still going to there's there needs to be a

• 1:26:33Speaker 00

  in order for it to be good, I think, like a human. Vision. There needs to be like, like a big 500 foot vision that a human being has. At least I'm talking about making a narrative film. I don't know how far away we are from that, but uh, from, from where the human being is not engaged at all. But um, because right now you can't really just ask chat GPT to write you a screenplay, feed that screenplay into another AI and then all of a sudden you have um, a movie. It's not really how it works. Um, there's a lot of tweaking.

• 1:27:15Speaker 00

  There's a lot of human conversations. There's a lot of, you know, critical thought that needs to go into this thing. And uh, so I think it's a, it's a great tool and it's improving every day. So, you know, 20 years from now, 10 years from now, that might happen. You might, you might be able to type in a prompt and, and, and get your um, your ready-made movie. But right now, um, it's, you, you still need, you still need the human to kind of be the storyteller, in my opinion.

• 1:27:48Nathan Labenz?

  Well, the AI assisted enabled short film that you guys have created is The Frost. It has been reviewed all over the place on the internet. You can watch it on YouTube and we will certainly be keeping an eye out for The Frost 2 with the whole range of new techniques and uh, look forward to continuing to follow your work as you guys continue to pioneer what it means to create truly high level content with AI. This has been a ton of fun, guys. Josh Rubin and Stephen Parker, thank you for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show.

• 1:28:31Nathan Labenz?

  So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.