Video generation with realistic motion

We seem to be experiencing a surge of video generation tools, models, and applications. However, video generation models generally struggle with some basic physics, like realistic walking motion. This leaves some generated videos lacking true motion with disappointing, simplistic panning camera views. Genmo is focused on the motion side of video generation and has released some of the best open models. Paras joins us to discuss video generation and their journey at Genmo. Sponsors: Domo https://ai.domo.com – The AI and data products platform. Strengthen your entire data journey with Domo’s AI and data products. Featuring: Paras Jain – LinkedIn https://www.linkedin.com/in/jain-paras, X https://x.com/_parasjChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Genmo https://www.genmo.ai/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep, technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at fly.io. Launch your AI apps in five minutes or less. Learn how at fly.io.

• 0:45Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris? Doing great. Happy New Year. This is our first show of 2025. Happy New Year. Yeah, this is the first one we're recording for the year. First, first time jumping back on the mics to talk about AI and definitely something that I think will be a theme in 2025, which will be, of course,

• 1:26Daniel Whitenack?

  multimodal AI in general, but I think something that a lot of people are wondering where it's going to go, I guess, in 2025, which is, which is video generation. Yeah. So we're very pleased to have Paris Jane with us today, who's CEO at Genmo. How are you doing?

• 1:45Speaker 02

  I'm doing great. Happy New Year, everyone. It's really wonderful to be here.

• 1:49Daniel Whitenack?

  Yeah, welcome. I know we've been trying to make this one happen for a little while, and I think the timing worked out well, because, like I say, people are thinking a lot about video generation and how that will evolve in 2025. Maybe from someone that's working in this area and has been thinking about it deeply. Maybe, you know, a lot of people, listeners have have just started thinking about this topic recently, but but you've been thinking about it deeply for for some time. Could you give us a little bit of a sense of kind of what happened, what led up to video generation and kind of where it is in 2024? And then kind of as we're entering this new year,

• 2:30Daniel Whitenack?

  what's the current state of video generation, I guess, more generally, in terms of what people can access in actually released systems or released models?

• 2:40Speaker 02

  Yeah, absolutely. There's been a long path to kind of where we are today. I think, you know, given so much excitement in what you might call the left brain of AI, that is like language models, reasoning, your O series of models, you know, I think the right brain has kind of lagged progress for quite a while, right? Like, people didn't really widely use kind of creative AI at a huge scale. And I think video is the ultimate creative modality here, right? If you think about it, so much of how we communicate as humans is through visual mediums, and specifically just video through through through motion. And so I think it's incredibly exciting in video being this ultimate form of creative multimodal synthesis. It was always really exciting, but the technology really kind

• 3:21Speaker 02

  of was far behind, I think, what people really wanted from it. And so it's interesting, my co-founder worked on some of the earliest image generation models and then 3D generation, and then video was always this kind of big modality we wanted to target. What I think was really interesting in 2023 and 2024 was first the development of the image generation, which was kind of a precursor to video. But but even then, the gap from image generation models to video generation models was always really big, because if you think about it, an image might have thousands of pixels and, you know, or even a million pixels, but a video would have, you know, hundreds of millions or even billion pixels in it in just a short clip. And so there was a huge gap to cross. I mean, so compute has scaled a lot, and that has enabled larger models. And so I think, you know, top of mind, since we, you

• 4:06Speaker 02

  know, rescheduled this podcast, Sora came to market, right? And I think that was really exciting. That was a watershed moment for a lot of people to kind of see what was possible with video generation. And to me, I think this is a really Early bellwether of like, you know, what is to come? I think we're still really early here.

• 4:20Daniel Whitenack?

  Yeah. And for those that don't know, Sorez is from OpenAI, right?

• 4:24Speaker 02

  Yeah, correct.

• 4:25Daniel Whitenack?

  Yeah. And I mean, you talk about some of the, some of the challenges with video generation being a kind of different animal. I know that some people might, you know, if they've been longtime listeners of the, of the show. We've had episodes talking about kind of stable diffusion and, and these sorts of models for, for image generation. What are the, the main, I guess, if you want to be a video generation model builder, what do you have to think of differently, both in terms of kind of the, the type of model that you would use?

• 5:06Daniel Whitenack?

  And also kind of, you know, the process that you'd have to go through in terms of, of curating data and that sort of thing.

• 5:13Speaker 02

  Yeah. I mean, I think first and foremost, video data is really data intensive, right? Like you just think about it compared to even images or text, like text is tiny. Images were more expensive, but video is like 100x more in terms of data volume in a short clip than you might have for images. And so when you think about training these models, that's really the most important challenge is how do you build architectures? And then systems that can scale to process large data sets. Um, that was a big ball neck for the community. I mean, again, at Genmo, we've been innovating heavily to actually make that possible. But I think this was why progress took a little bit longer than say for images or language to come to market. For companies that are training this though, like again, they're having to curate massive scale data sets easily

• 5:54Speaker 02

  in the, you know, petabytes of data essentially just to pre-train these models. And that's really intensive. Many practitioners are beginning to fine tune these models too, but even that remains more challenging than your staple diffusion, for example.

• 6:05Chris Benson?

  That's got to make it hard for new entrants to come into the, into the field. Just, just the sheer volume of what you have to get set up, you know, ahead of time to handle that is probably be, I would imagine beyond what, what most organizations are really able to do unless they're, you know, have specific expertise or experience in the area or something.

• 6:25Speaker 02

  Yeah, absolutely. I mean, it took us a long time to get ready to pre-train models. We'll talk about this more, but we open sourced one of the state of our video generation models. And part of the goal here was to kind of let other people have a chance, right? Let them pick up a model and begin to fine tune it. And they're kind of skipping past like a massive volume of technical infrastructure they otherwise would have needed to have built.

• 6:46Daniel Whitenack?

  And maybe talk a little bit about that data side because I know that this is one of the things that's been, I mean, it has been a struggle on the text side, but I think especially on the image and video side where there's a lot of questions about, hey, well, where, where can you actually source all of this video and imagery? And, you know, what are the, what are the rights associated with that? But I also imagine there's, there's definite curation that's needed in terms of like all these prompts that, that I've seen people do with like, oh, generate this, you know, it's shot with a Canon DSLR or whatever, like all, all of that sort of thing has to be,

• 7:31Daniel Whitenack?

  has to be curated on the, on the prompt side as well. So yeah, could you, could you talk a little bit about that, that data curation and what the, the source of that, kind of where, where you could even get videos and then, and then the, the curation process?

• 7:46Speaker 02

  Yeah, I mean, I think pre-training in general, and that's true for image models, text models, audio models and videos, they rely on like, you know, large volumes of internet scale data. But I think what's uniquely challenging with video is it's just, you know, it's easy to kind of get drowned out in the noise, right? And one, one angle here that I think is really interesting that we, for example, zeroed in on was how do we learn high quality motion with a video model? And it turns out the vast majority of video you find on the internet doesn't move. It's like a static object or it's someone talking. And if you think about that, that doesn't actually teach a generative model about the world. It doesn't teach about physics. It doesn't teach it about how objects interact. And so it's not going to learn strong reasoning. And so the way we think about it is really the goal with a video model is to learn physics

• 8:29Speaker 02

  and realism and the, the laws that govern our world. And so you might think about inertia, mass, optics, you know, fluid dynamics, all these kind of base properties and how they all interact. That's really the goal with video generations to learn and engine that can simulate this. Because The output is a video and we can consume it. It's creative and it's beautiful. But the hard step here is finding data that can really help you learn these base rules of the world. And this was one of the most fundamental gaps we had to cross. It's kind of non-trivial.

• 8:59Chris Benson?

  I'm kind of curious, as you're describing that, I would imagine that some things that you're training for are harder for the model to learn than other areas. And that, I mean, if you think about just, you know, narrow it down to just, you know, animals and mammals and humans, and they move differently, and the physiology and the anatomy is a bit different across those. And that all has to somehow be inferred by the model if they're going to make a video that's realistic. What have you, in your experience, as you, you know, you talked about motion being so important stuff. What are some of the harder things that's been to get right over time? Not just in where you're at today, but like for the industry, what is the industry? And maybe early on, what have you struggled with?

• 9:41Speaker 02

  Yeah, I think it's a really, it's kind of funny that one of the test cases people use now to test different video generators is gymnastics. And I think the reason for this, this hilarious videos online you'll see of Sora or other video generators doing gymnastics. And I think one of the answers is video generation models just can't do it now. And it's really complex human motion. It's really rare. Obviously, you talk about data curation, for example. Like, there isn't that much complex motion. We see people like doing twists and twirls and backflips and stuff in nature, right? And so it's kind of interesting is like, that is, that requires fundamental understanding of how human kinematics behave for you to simulate that properly without it feeling disturbing. And so this has been one of the challenges for people. I mean, for example, early on when we were training,

• 10:25Speaker 02

  I mean, we've gone through three fundamental pre-training foundational models in our history of a company. And what's interesting with Mochi, which is our latest model, and the prior one Replay, was walking was actually a really basic thing that was really hard to nail. It turned out most video generators early in, kind of early to mid 2023, they would make humans kind of hover as if they're hovercrafts, like their feet would not move. They would just kind of levitate off the ground and move. And so the models were not capable of synthesizing, forget gymnastics, just walking. And so that was one of the critical watershed moments we had to cross, for example, as a company.

• 10:59Chris Benson?

  I would suggest that might have been some folks that had a little too much bourbon in their eggnog over the holidays, right there, kind of that floating thing.

• 11:06Daniel Whitenack?

  I've definitely seen the like Jedi vibe, which is kind of cool in one respect, but not that awesome if you don't want it.

• 11:16Speaker 02

  One thing here is I think like we've invested heavily in evaluation infrastructure at Genmo and as part of it's like, how do you benchmark these capabilities? Like, one of the test cases we have is, you know, you might have a woman drinking a glass of water with ice and you want to look at, hey, does the ice move realistically? There's a water flow, but also like, you know, what's interesting is, once in a blue moon, like, the character will try to drink the water through the side of the glass, which is just not physically cohesive, consistent. And you'll actually see this with some of our competitor models. It's something we had been trying to develop. And I think just that isolated test case alone communicates a lot about the video generation model's capability to just understand the laws of reality, right? Like, it's kind of, yeah, it's a Jedi mind trick. Like, you just cannot, you should not be able to do that, right?

• 11:59Daniel Whitenack?

  How, like, when you're using those test cases that you've developed, is that a lot of human review? Or how does that, like, how do you create kind of the tooling around that? Because I know there's sort of like comparisons between this image and that image, right? And/or this frame and that frame, and you can compare closeness and all of that. But there could be a lot of sort of closeness in the, in the overall image. But if the woman is drinking from the side of the glass, there's kind of a major failure moment, even though maybe everything around is really good.

• 12:38Speaker 02

  Yeah, I mean, look, there's a, there's a, there's a lack of external publicly available quantitative benchmarks. I think one of the ones that are publicly available are these leaderboards. Or Artificial Analysis has a video generation leaderboard. I mean, we are the number one open source model and kind of neck and neck with closed models there, and that's just human preferences. Hundreds of thousands of people look at two videos side by side and they say this one's better or this one's better. And you kind of get like a chess style Elo rating. And I think this has been one of the best or better public benchmarks. Um, you know, internally, one of the ways we think about this, though, is as we're measuring these capabilities such as world understanding in physics, it's very hard for a human actually to rate by that. It turns out like when we as humans look at two videos side by side and you're saying, which one do you prefer? You often prefer the one that might have slightly higher resolution

• 13:21Speaker 02

  or more detail. But if you actually think about if I'm going to use this in an actual production application, like film production or gaming or something else, like I probably actually care more about the motion. And so we actually have to override the human intuition, your first order intuition to say select for detail and use these test cases as sort of like functional testing of how we can measure these capabilities. You know, I, in my career, I started out actually in self-driving. I worked at one of the early companies applying deep learning to self-driving perception. And, you know, I took a lot, I take a lot of inspiration to how we built functional safety testing, for example, for deep learning systems, right? And in that way, you're going to enumerate these test cases and use cases. And you can actually say yes or no, do you kind of pass that, that test case scenario, right? And so, you know, whether it's a human has to do that review and we're starting to

• 14:05Speaker 02

  develop more automated metrics, I mean, it's just producing more structured forms of valuation, I think are really important because otherwise, the world is just too intricate for us to test everything, right? So we have to kind of go use case by use case and just measure progress. And it turns out as you scale the models and scale the data sets, we begin to see percentage completion rates improve. And this gives us a semi-quantitative benchmark of progress.

• 14:38Speaker 04

  Well, friends, AI is transforming how we do business, but we need AI solutions that are not only ambitious, but practical and adaptable too. That's where Domo's AI and data products platform comes into play. It's built for the challenges of today's AI landscape. With Domo, you and your team can channel AI and data into innovative uses that deliver measurable impact. While many companies focus on near applications or single model solutions, Domo's all-in-one platform is more Robust with trustworthy AI results without having to overhaul your entire data infrastructure, secure AI agents that connect, prepare and automate your workflows, helping you and your team to gain insights, receive alerts and act with ease through guided

• 15:21Speaker 04

  apps tailored to your role and the flexibility to choose which AI models you want to use. So DOMO goes beyond productivity. It's designed to transform your processes, helping you make smarter and faster decisions that drive real growth. And it's all powered by DOMO's trust, flexibility and years of expertise in data and AI innovation. And of course, the best companies rely on DOMO to make smarter decisions. See how DOMO can unlock your data's full potential. Learn more at ai.domo.com. That's ai.domo.com.

• 15:58Daniel Whitenack?

  So Paris, I'm wondering, you mentioned this kind of history of pre-training at GIMMO and the most recent model, which of course we want to talk about, but I'm sure that that most recent model is informed by things that you tried in the past and kind of your history there. So could you give a little bit of a snapshot of kind of the history of your team and how they, how they approached this problem, how you all approached this problem and the kind of generations that you went through with that?

• 16:32Speaker 02

  Absolutely. So we're just about two years old at this point. We actually started work on the company Christmas 2022. So it was a holiday and there was a Jay and I were both the co-founders of the company. First and foremost, we're brothers. I think that's really unique.

• 16:48Daniel Whitenack?

  That's awesome.

• 16:48Speaker 02

  So, you know, and we didn't really plan to start a company with brothers. I mean, it's a little weird. I mean, it's like, you know, normally have sibling rivalries and things like that. I don't know. We didn't have much of that, but it turned out our skill sets were super complimentary. Both of us were doing our PhDs in UC Berkeley. I was working on large-scale distributed systems, you know, in the UC Berkeley AMP lab and RISE lab. And this is the same lab that created Apache Spark and, you know, Ray and the AnyScale project. And so really hardcore machine learning systems for scaling large language models. That was when my dissertation topic was on. And concurrent to that, Jay was working on the foundations of modern image generation. So. You know, he had joined Berkeley to work on early image generation models. This is kind of like in your GAN era.

• 17:31Speaker 02

  And I think for him, one deeply unsatisfying thing was a generative adversarial network was like a Mirage. It wasn't actually like a grounded loss objective that was learning real motion or dynamics. It kind of was like this game, but you got image generation aside artifact. So I think his story was really interesting in that he ended up kind of writing his paper, DDPM, or the denoising diffusion probabilistic models paper, which is one of the foundations for how we think about image generation with diffusion today. It's one of the most highly cited papers in this area. And that came from, I think, early inclination that how do we build video image models that understand physics and realism? Instead of just kind of like artificially playing this, you know, game that results in image generation, grounding it in real generative pre-training.

• 18:14Speaker 02

  And so that's some of the early academic history of the company. But starting the company, we decided to do video because like it seemed impossible. Back in 2022, it was just completely outside the frontier. And we said fundamentally, we need a new architecture to solve this. And so let us discover both from a systems perspective and distributed systems perspective, but also machine learning perspective, what the right approach to do that is. I mean, so yeah, it's been about two years since founding. We've gone through three large pre-training runs. And in each time, we learned something new about the world and integrate that into our approach and our framework and architecture for how we train these models. I think the single underpinning thing, though, is motion. We always joke like Genmo doesn't really have a, an explanation, but we kind of retroactively applied like this idea of generative motion, right?

• 18:57Speaker 02

  Genmo is like, we care so much about motion in video that that's really a core element of our founding history and our framework for how we approach video generation.

• 19:07Chris Benson?

  I'm wondering, you got me, I had a question for a moment or two is you were talking through it. You kind of talked about that evolution and, you know, kind of starting with GANs, the generative adversarial networks and, and finding your way across kind of the architectural progression that you guys have found. Could you talk a little bit about that in terms of, of like, you know, if you were coming into it during, you know, the age of GANs right there, and that was the thing. But, you know, what did, I'm kind of curious, like at a high level, what was the problem with that? Why did that not work for you? What did you look to next? You know, could you kind of give us kind of a highlight skip over the top of a couple of different major architectural twists and turns to give us a sense of what your journey might have been like?

• 19:48Speaker 02

  Yeah, so I think the earliest form of image generation models, I think, started to work well were auto-aggressive image generation models. This is very similar to a large language model. You do, you kind of take a, you know, you might take an image and you make it a single vector, a line. So if it's like, uh, you know, twenty eight by twenty eight image, now you have seven eighty four pixels in a straight line. And you just go one by one by one and decode the next one. So that was the earliest form of image generation. There are models like um pixel R N N or pixel C N N or Image G P T, which was from OpenAI, which were the earliest works here that worked well. But the problem is like, images have millions of pixels. This would never scale to produce high resolution images. I think what's interesting from when a Jay was working in this, like, early on in twenty eighteen, twenty nineteen, was um.

• 20:34Speaker 02

  I think I remember he trained in auto aggressive image generator model. And the first models he trained were trained on like L sun, which is a data set of bedrooms, basically. But what was so interesting is that in, and it would be like a little five by five or ten by ten pixel region. It would start to put artwork on the background of people's bedrooms. Why? Because that's just like what nature looks like. That's what real estate listings look like. But it was the first indication of A I generated art in some sense with, with an early image generation model. The problem is this would not scale, right? Because you're kind of going pixel by pixel by pixel. So it would take hours to make a small image. I mean, so GANs were the next kind of major approach, I think, that really worked well for this. And GANs are trained again with this like generative adversarial objective. It's kind of this like dueling game between a generator and discriminator, but they were really hard to train.

• 21:20Speaker 02

  It turned out they were getting these bad, you know, uh, states where, you know, you might get like mode collapse, for example, was one of the biggest issues. It would mean you could produce images of like a single domain, but you couldn't like produce everything in the world with a GAN. So you could get a really good model for making faces or a really good model for making, you know, bedroom, bedroom pictures or making a really good model of like tigers. But it was really hard to say, train a model on all of ImageNet, meaning cover thousands or uh thousands of different categories, right? And so diffusion models was a really exciting approach that Jay began to work on. Because it was, it had the potential to provide that kind of mode coverage. You could learn diverse representation in the world that generalize beyond just a single. Single domain like faces or animals to everything. And so, you know, that that was what will kind of result in DDPM.

• 22:05Speaker 02

  And I think since then, I mean, you've had latent diffusion, the stable diffusion approach. And then video generation is, I think, the next major evolution. But but the the learning paradigm has mostly remained remained similar to this like learning through this like, you know, diffusion setup or kind of iterative de-noising, right? This is the formulation of this diffusion problem. But like, it's remarkable to see how how far that has scaled, like literally 10,000x in pixel scale from the earliest diffusion models to kind of where we are with video generation.

• 22:35Daniel Whitenack?

  I guess on that front, like, how did you decide? I guess, because I know that part of what you've done, and I'm assuming what the intention was with the models that you've created is to open source them in one way or another. And as you mentioned earlier, release things into a community where people could experiment and try things and fine tune. How did you think about kind of size of the model? And and that sort of was that kind of purely driven by what was needed to produce kind of a certain size of video or certain resolution, a certain kind of performance metric that you were after?

• 23:18Daniel Whitenack?

  How how did you make some of those, I guess, trade off decisions? Maybe also the compute that you had had access to?

• 23:25Speaker 02

  Yeah, I mean, first of all, pre-training is incredibly GPU intensive. And we have access to more than 1,000 of H100 grade GPUs. And so, I mean, that is incredibly GPU intensive. But I think it's also a question of how you utilize that hardware effectively. And one of the critical challenges with video is they have really long sequence lengths, like training a video generation model is equivalent to kind of training a million token length kind of context window for a new language model. And so this introduces a huge set of challenges that are kind of orthogonal to kind of parameter scalings you might see typically with large language models. What I think is interesting is, though, certain capabilities only emerge at certain parameter scales. So like I talked about walking, like, it's very difficult to get walking to work with

• 24:07Speaker 02

  like a, you know, a one or two billion parameter model or something smaller than that. It just like. You won't learn that capability. So you do need a certain amount of scale for it to work. But at the same time, you're not seeing models that are like 100 billion or trillion parameter scale, as you see kind of with the frontier grade language model. So we open source Mochi one. It's a 10 or 11 billion parameter scale model. So it's it's big, a lot bigger than your conventional grade, older grade of video generation models, but it again, it still is runnable on a consumer grade GPU. People can access it and they can use it. That was a very intentional choice by us to kind of right size it for the community while making sure it wasn't too small to limit its capabilities.

• 24:45Daniel Whitenack?

  And what reasonably, I know one of the things that I've noticed over the, over time, as I've experimented with different video generation, either demos or products, there's definitely an element of it to where you can only generate so much. I'm imagining that there, as you mentioned, there's a sequence that is being generated and in a way similar to a sequence that's generated out of a language model, there's iteratively or iterations of calling the model, which is more compute intensive, the more you generate. Is that a true assumption about language or video models? Or is there, I think people are somewhat

• 25:28Daniel Whitenack?

  familiar, at least if they've been around the podcast or have done their own research in terms of how language models generate tokens, right? So I have a prompt, I, the model generates a token and then that's added to my prompt and then I iteratively generate another token. And so the model is being called the more that I, more, the more that I'm generating. Is that same thing true for kind of generating these sequences of videos? What are the kind of concerns around actual compute and usage of of these models in a realistic environment?

• 26:04Speaker 02

  Yeah, I think video generation models share common elements with large language models, but they also differ in some key ways. So first and foremost, a language model decodes tokens auto aggressively, one at a time. So if you want to generate, you know, a thousand tokens, however many, like whatever, let's say 500 words. You need to do 500 forward pass or 1,000 forward passes of the model. In a video generation model, like every pixel is kind of generated at once. So each forward pass produces all of the pixels that you see in your video across space and time. And we do multiple denoising steps. So you start with a kind of a pure noise sample and through maybe 50 or 100 forward passes, all of those pixels eventually become full resolution.

• 26:47Speaker 02

  So you'll actually see if you use our product, we stream those pixels as they're getting denoised in real time to your browser. So you'll see a full video, like not just a frame, but a full video, but it's kind of blurry. And slowly the video gets, you know, sharper and sharper and the details begin to resolve. Like you'll see blobs that become more and more detailed. And eventually you get fine details like hair or teeth or, you know, you know, plant leaves and so on. Like that appears in the last stage of this. And similarly, the motion might start with coarse grain motion, but eventually becomes much more detailed and realistic as the denoising process proceeds. So it's kind of a different axis in which we do the compute, like just like you do tokens and decoding. Like in video models, you have this denoising step. But one really important thing to talk about architecturally, at least with Mochi as we open source it is,

• 27:32Speaker 02

  it's kind of a multi-stage model. There's first what we call it's variational autoencoder, which or VAE, which refers to essentially video compression. There's just too many pixels in video for us to learn over natively in the model. It's just way too expensive. So in Mochi, we trained this 100x video compression model through the variational autoencoder setup that takes the input video and actually projects and makes that sequence that we talk about. So you're going from something that's like, you know, hundreds of millions of pixels or something down to something that ends up effectively taking about, you know, 50,000 tokens equivalent, 50 to 100,000 tokens equivalent in a language model. And so we do that compression stage first and then in that latent space, that is actually what the diffusion model is learning, right?

• 28:16Speaker 02

  So that that 10 billion parameter model is learning to kind of reconstruct, you know, that 100x down sampled or compressed space.

• 28:23Chris Benson?

  Do you envision that there's ever a point, you know, with with compute, you know, growing so fast, is there ever a point where you think compression will no longer be needed? And you'll be able to do, you know, very large and detailed videos without the need for that, just because compute is so available in the future? Or do you think that's unlikely and we're going to keep chasing it with compression and doing other things?

• 28:45Speaker 02

  So there was the first diffusion models actually were what we would call pixel space models. So they were done at the full resolution of the sequence. And so this is actually still doable for images. I think what's interesting is that this like latent diffusion setup has outperformed the pixel space approach, even in images where it is feasible computationally to still do that. You know, I think it is interesting, though, because like there has been a lot of hybridization of architectures between like auto aggressive setups and diffusion setups. That was one trend at like, for example, our team went in Europe this year, then in 2024. And, you know, several people have begun to explore combining different elements of auto aggressive models, diffusion models, both in pixel space and latent space. I think it's like a really diverse space that like is just extremely underexplored.

• 29:29Speaker 02

  For example, when we open source Mochi, we actually developed a new architecture. We call it the asymmetric dit. It was just an evolution on the kind of area that people were in. I mean, people leverage this diffusion transformer setup for the architecture. It's part of why it's so expensive, but, you know, we began to take some early steps to do architectural exploration. So I hope we can eventually, long story short, like find some global optima between compression and the actual generation part. Today, we kind of factorize it for computational reasons, and I think it'll just get more and more blurry as we kind of combine these different elements.

• 30:09Daniel Whitenack?

  Well, Peraz, you've mentioned Mochi. Would, uh, in, you know, this, this is the latest wave of what you have have created at Genmo. Could you talk a little bit about Mochi in relation to previous models and also Mochi? I mean, you mentioned that there's, uh, Mochi is achieving kind of top performance on certain benchmarks. Could you kind of help us understand where it fits into the ecosystem of video models out there and also kind of what it represents to you all in kind of progression from your last generation to this generation?

• 30:48Speaker 02

  First and foremost, before I dig into this, my belief is video generation is super early. I think we're 1% of the way there. So I think people look at this stuff and it's really surprising, but there's a huge gap between reality and where the state of video generation is, right? And I, and I think that mindset is really important because when we looked at the field of video generation as of, you know, mid 2023, when we kind of had our last generation model, um, sorry, mid 2024, when we had our last generation model replay, was they would synthesize high resolution videos, but they just wouldn't move. They weren't that interesting, right? So you would see a video of a person and they would just stand there and, and maybe there was camera motion. So the camera would kind of orbit the person or pan a little bit, but, but the subject wouldn't be moving. And to us, that would indicated some kind of learning failure with the video generation

• 31:33Speaker 02

  setup as of, um, and the last generation of these models. And so that was first and foremost, the most important thing we wanted to solve for video generation was solve motion and subject motion specifically. And so Mochi one is kind of neck and neck with the latest frontier grade kind of closed source models, your Google veils or, you know, Sora is in that way specifically by motion benchmarks, actually. And I think this is really important and subtle, but, um, that was kind of the key component we wanted to solve with video generation. The, the second one that was really important for us to solve in Mochi was prompt adherence. It was really common. I think many people have this experience with video generation. As you say, I want, I want X, right? Like you might say, uh, you know, I want, uh, you know, a classic test for this is a,

• 32:16Speaker 02

  is a, you know, like I, I want a, a dog wearing a hat, holding a teacup, but it, it'll make that, but the order of those things and the composition of those elements is wrong. Right? So they might be sitting next to it, but not holding it. We talked to a user and user study about video generation. They described the state of video generation was kind of like pushing on a rope. You, you kind of want the rope to go one way, but it, it, you just can't get it to go, right? You just, it's, it's just really hard. And so with Mochi, we also invested heavily in prompt adherence in addition to motion. And so prompt falling is, I think is really important element that, that will be critical to make these, these systems practically usable. We'd love to talk about like, you know, we open sourced this also because there was no good open model, let alone closed.

• 32:57Speaker 02

  There was a few of these closed models, you know, runway, um, and Sora had been kind of previewed in their blog for, for several months, but, but nobody had actually. Train and release an open model. And so that was holding this field back. And because we're so early, our viewpoint is releasing this model and creating this bedrock foundation for people to actually do the research on aspects like motion and prompt adherence was going to be critical for the field. And it benefits us as a company because people are building on top of our models, right?

• 33:24Chris Benson?

  So what kinds of things are you seeing people want to do with the model? And what are of the different categories of use cases, you know, that people might be addressing? What are the ones that are the high value?

• 33:35Speaker 02

  Yeah, I think everyone's first experience is just play. So like, people just want to open it up and they want to see something wild, right? Like a baby riding a dog, right? And so I think that was always a funny one. That was like, you know, you might have these things that just don't happen in the real world that you want to see the model do. And then so people start with that and they explore the surface area. But when we look at actual real use cases, I think what's really interesting is this video generation technologies are beginning to work its way into like enterprise content creation workflows. And I think of this as like creation, and then there's like editing, right? And these are two kind of halves of practical application of video generation. So creation, I mean, first and foremost, like many people are starting to begin to explore using video generation as a substitute for stock video. Like if you can't find exactly what you want to stock

• 34:20Speaker 02

  catalog, you can just go generate it and it's going to come with all the right adequate licenses. It's exclusive to you, right? No one else gets that video because you made it, right? And it's N equals one. And so that's actually really powerful for a lot of content creation workflows. Video is just really hard and expensive also to iterate with, right? You shoot it once, and if it's not perfect, you know, you might want to re-prompt and re-edit it. And so I think that's an exciting application, for example, in the brainstorming and pre-visualization and storyboarding process of content production. That goes way faster if you have a tool like a video generator in the loop. And then editing.

• 34:51Chris Benson?

  Actually, that's exactly where I was about to go was on the editing is kind of how does that, how do you envision that fitting in as that becomes a problem that people are attacking aggressively? What does it mean to edit video, you know, in the context of video generation? You know, if you're generating the video from scratch?

• 35:10Daniel Whitenack?

  What does it mean to edit a video like that? And how might that be done? Is anyone really thinking about that right now? Is that on the table?

• 35:18Speaker 02

  So we released Mochi one as open source. We didn't know what people would use it for. And one really exciting thing, within two weeks of open sourcing it, one of the community members built this workflow called Mochi Edit. It's a full video editing pipeline built on top of our open source model. And with it, you can add, remove, or change an object. So it's a crazy video. You can search up Mochi Edit on GitHub. And what was the demo that I think he showed me that was really cool? Is they took a video of a person talking and they said, give him a hat. And it actually put in a fully realistic, exactly 3D trapped hat on him. It just looked totally realistic. And I think that full process with the conventional videoing and pipeline between tracking and rendering and compositing everything would have taken like, you know, two, three weeks, honestly.

• 35:59Daniel Whitenack?

  Very cool. Do you see, I mean, I know there's certain, if I remember right, Coke did like a commercial, Coca-Cola did a commercial for their winter advertisement with, with Gen AI. Do you think, well, this is maybe a wider question, but how do you think people kind of in 2025, you know, how are we going to experience video generation? Kind of at the general public level? Do you think it will start to like, in what ways will it start to filter into people's everyday, everyday lives? Because Chris and I, well, everybody remembers like,

• 36:42Daniel Whitenack?

  we are talking about lots of language models before ChatGPT on the podcast. But, you know, we weren't talking about them at Thanksgiving dinner, right?

• 36:52Speaker 02

  Not at all.

• 36:53Daniel Whitenack?

  And so you do have those moments of like the Coca-Cola video where people were talking about this more widely. But that's probably not like the ChatGPT moment of video generation. Any thoughts on kind of how general public will kind of start to intersect with this technology in the, in the coming year?

• 37:14Speaker 02

  And you think? The early adopters are certainly here for video generation. I mean, our platform has more than well past more than 2 million users who use it just beyond open source and open source is probably some many multiples of that. But I think that still represents this like drop in the bucket compared to conventional media. And I think the one of the biggest limiters like I shared was like your ability to control it and like, you know, once you can actually get something out of it, the wow moment is almost instant. Like you'll ask it for something that just couldn't exist in the real world and you see it in front of your eyes. I mean, that that is a jaw dropping experience for most people, right? But I think the hard part there is the tech has required too much expertise with prompting and understanding of how to actually get good results of the model to make it usable.

• 37:55Speaker 02

  I think 2024 is the year that we will see, you know, instruction following and prompted here and solved here that make this makes this stuff actually follow what you want to say. And I think of this as like going from G P D three, which was just like a unaligned language model in some sense, which kind of would ramble about whatever topic at end, but not in a particularly useful way towards chat based instruction tuning, right? That that was the breakthrough moment for language models. I think very similarly for video models, it kind of comes to the moment where like somebody can pick it up and use it without being an AI expert. You know, today, you many people are already talented and mid journey or other kind of conventional forms of image generation that kind of translate into video. And I think this is really one of the critical moments that has to be solved for this to

• 38:40Speaker 02

  have like breakout exposure. But, but I mean, I just imagine a world like I think in five years, we're going to hit a point where, you know, there might be a poor kid in Mumbai or Kenya or something who just has a phone and a good idea, push the button on their phone and it wins an Academy Award, right? Like that, that's going to change the world. I don't think we're that far from that, to be honest.

• 38:59Daniel Whitenack?

  Yeah, I, I think that there's, I love how you've framed that in that kind of expanded agency sort of way. So instead of like AI models generally, I think the, the way people think about them as a bummer is like, oh, these, these things are going to automate everything, every video I'm going to see, I'm never going to see cool videos again because they're all going to be AI generated without creativity. But I think the fact that, you know, what we're seeing with language models, what we've seen even with image generation is there's there's so much creativity that the the human can bring into that, but it also democratizes a lot of potential, um, you know, production

• 39:43Daniel Whitenack?

  and that sort of thing to those that have amazing ideas, but maybe not access to a Hollywood film crew, right? So I I love, I love that there is still that element in kind of your vision of that human human agency being expanded upon and even, you know, people, people getting to tell stories that maybe they they wouldn't otherwise. So I I love that.

• 40:06Chris Benson?

  I I've got a question for you. It's a little bit of a random one, but um, interesting, you know, people ask me this a lot. What does creativity mean as we go forward as we're having these tools? And, you know, human creativity is coming to bear. You're having these tools that, you know, some people consider them creative in a sense, some people don't know, but what does that look like? What is that, that person and a tool together going and doing that, that thing the Kenyan boy is doing? How do you think about that? Like, how do you contextualize that?

• 40:38Speaker 02

  You know, I think human ingenuity and creativity is the root of all like interesting form of content. Like if you have AI, like, I know people are scared, hey, I was gonna automate all this stuff. But if you, if you look at what like L L M's will just ramble on about, it's just like a the aggregate average of all their training inputs. And that's not particularly interesting or novel to anybody, right? Like, I think the greatest films come from someone with a new idea, right? And a new lens on the world, right? A new interpretation of, of what it means to be human, right? And and live in the world that we do. And, and from that you have great media, right? There's some, some, and I, and I think that will forever be true. The human's role here is always going to be pushing the frontier. I mean, language models learn and video models learn by just

• 41:23Speaker 02

  averaging and aggregating compressing all information around them. But but in some sense, they won't ever be able to really push the frontier alone. Like a human plus now a video model, though, is something entirely different beast, right? Now you have something I like to call term creative amplification is possible, right? Like the human alone is producing the creativity, but with that video model, it now amplifies in such a way that just wouldn't have ever been possible with this old, older. Older generation media in the older world, right? Like that iteration cycle might have taken years, an entire lifetime to kind of go through and discover an idea space. And now somebody can do that, you know, within a matter of like months or weeks, just iterating on new ideas and testing them out and seeing them visualized for them.

• 42:02Daniel Whitenack?

  I guess that kind of leads us naturally into, that was a great kind of vision wider, but what is your vision for Genmo specifically? What's kind of, what keeps you up at night? What are you most excited about kind of as you move into a new year with a lot of new possibilities?

• 42:22Speaker 02

  So I think our vision has been very consistent over a long period of time, which is to build frontier models of video generation. But the goal was to unlock the right brain of artificial general intelligence. It's completely neglected. I mean, OpenAI and kind of these frontier models have taken over the left brain. And we said, hey, this other side is just as capable and just as important as the left brain here. And so, you know, I term that as like thinking, imagine AI that can simulate anything possible or impossible, right? And I think the first step here is creativity, is media, people creating like, you know, like I described, this vision of empowering creators. But like longer term, I actually think this is really interesting in that if we can explore this world of synthetic realities, it'll unlock huge progress. And, you know, I think like embodied AI, for example, and that's when this tech starts

• 43:07Speaker 02

  to become really powerful, right? Like I started in self-driving my career and the big problem is there's too many edge cases to simulate, right? And then even if you get millions of miles on the road, there's still new things that will happen. But I think for the first time, a video model will enable training, robust agents that can operate in the real world and actually understand all the possible realities that they can just simulate through that, right? Like that's entirely new paradigm that I think we're starting to see explored, even in reasoning with the OO1 style of models as well. But to me, that's, that's one of the most exciting long-term tenure potentials that we'll see for video generation. And then we at Genmo are kind of like trying to work towards that future.

• 43:47Daniel Whitenack?

  Well, thank you for how you're digging in in this space. It is truly inspirational and really appreciate you taking time to chat with us as you head into head into those innovations. Exciting stuff. Please come back when you release whatever the next is. You're welcome back to chat about it. Thank you so much, Paris. It's great to chat. Thank you, Daniel. Thank you, Chris.

• 44:22Speaker 00

  All right. That is our show for this week. If you haven't checked out our change log newsletter, head to changelog.com/news. There you'll find 29 reasons, yes, 29 reasons why you should subscribe. I'll tell you reason number 17, you might actually start looking forward to Mondays.

• 44:42Speaker 05

  Sounds like somebody's got a case of the Mondays.

• 44:45Speaker 00

  28 more reasons are waiting for you at changelog.com/news. Thanks again to our partners at fly.io, to Breakmaster Cylinder for the Beats, and to you for listening. That is all for now, but we'll talk to you again next time.