The Engineering Behind the World’s Most Advanced Video AI

Is video AI a viable path toward AGI? Runway ML founder Cristóbal Valenzuela joins Lukas Biewald just after Gen 4.5 reached the #1 position on the Video Arena Leaderboard, according to community voting on Artificial Analysis https://artificialanalysis.ai/video/leaderboard/text-to-video. Lukas examines how a focused research team at Runway outpaced much larger organizations like Google and Meta in one of the most compute-intensive areas of machine learning. Cristóbal breaks down the architecture behind Gen 4.5 and explains the role of “taste” in model development. He details the engineering improvements in motion and camera control that solve long-standing issues like the restrictive “tripod look,” and shares why video models are starting to function as simulation engines with applications beyond media generation. Connect with us here: Cristóbal Valenzuela: https://www.linkedin.com/in/cvalenzuelabRunway: https://www.linkedin.com/company/runwayml/Lukas Biewald: https://www.linkedin.com/in/lbiewald/Weights & Biases: https://www.linkedin.com/company/wandb/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The best way to understand video models is to understand them as basically universal simulation engines. They can effectively try or will try to simulate, hopefully, everything. And we're simulating entertainment and media first, but you can see how the models will start to simulate way more than just that.

• 0:15Lukas Biewald?

  You know, I always think like, wow, like, how is Runway going to compete against the resources of Google? Do you have any thoughts on how that's working?

• 0:22Speaker 00

  When we started seven years ago, I think people just never thought it was that interesting to build video models in the first place. I feel now we've created an industry, but I think still like, resources here are not enough. The only reason we're able to do this is because we have this incredible team that's incredibly committed to achieving the vision that we have, and it's the manifestation of that.

• 0:40Lukas Biewald?

  What are some things that your latest model can do that a previous generation couldn't?

• 0:44Speaker 00

  Now, Gen 4.5, which we just announced, has a really good margin with respects to everyone else who comes after. And so being able to train not a language alone, but on like observational data, on real data, allows the model to grasp reality and how the world works in a much more consistent way to hopefully do much more than just like video generation.

• 1:06Lukas Biewald?

  You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. All right, I got a text this morning from Chris, the CEO and founder of Runway ML, that his latest model is on top of the video arena leaderboard. And so I just wanted to record a quick podcast to ask him how he did it and what's working. And I hope you enjoy it. All right, so you're at the top of the video arena leaderboard?

• 1:35Speaker 00

  We are, yeah. Quite a moment.

• 1:39Lukas Biewald?

  So can you just, can you tell me how that leaderboard works, first of all?

• 1:42Speaker 00

  Sure. So it's, all leaderboards are asking basically the internet world, people at large, to rate between two different outputs. And so you go basically through pairs of examples, and then people vote. And as models get better, better, better, you eventually have someone who's like on the leaderboard, on a ranking. And so for video, that's, I think, it's been On video have been happening for the last couple of of years, one or two years. Um. And so, yeah, now gen four point five, which we just announced. Um. Is on the top of a leaderboard. It has a really good margin with respects to like, uh, everyone else who comes after.

• 2:25Speaker 00

  Um. And it's a pretty remarkable feat, to be honest. It's a. It's very hard. Video is one of those like, um, ex- very, uh, expensive, hard modities to ve- be very good at. And, uh, we managed to to get the best video model in the world out there.

• 2:41Lukas Biewald?

  It's really amazing. I mean, you know, we've been, you know, we've been friends for a while and, you know, been obviously, you know, working with you guys for a long time and. You know, I always think like, wow, like, how is runway? You know, got to compete against the resources of Google and yet you you like continue to do it. Do you have any like thoughts on how that's working?

• 2:58Speaker 00

  Uh, yeah, you know, uh, we we started like, I don't know, seven years ago, and and I think we met like when we were really, really early in the journey. I don't think it was like, uh, competition or a leaderboard, because I think people just never thought it was that interesting to build video models in the first place. Uh, I feel now we've created an industry, we've we've helped others kind of come on board and. And it's true now, it's it's some of the biggest and, uh, uh, most. Uh, I guess, well-funded companies in the world are, uh, building vers- things to runway and and it's great. It feels like a great. Uh, push for the market. It helps, I think, uh, models improve consistently across the board. Um, but I think still like, uh, resources here are not enough. Like, you need, you need a consistent vision and.

• 3:41Speaker 00

  And an obsession to to build this. Um, I I think also the the underestimated part of a lot of what we've done is how efficient and effective we've been. With the resources that we have. I think it's easier to train models who have hundreds of billions of dollars to spare. But managing to like get to the best, uh, model in the world with. Uh, small set of resources, uh, requires you to be very creative about. Uh, you know, efficiencies and optimizing both training and inference and. It's it's a real, the the only reason we're we're we're able to do this is because we have this incredible team. That's incredibly committed to like, uh, kind of achieving the vision that we have and. And it's the manifestation of that.

• 4:23Lukas Biewald?

  And I guess like what what makes, I mean, I just I'm continuously astonished by like how well these video, you know, models do like, I mean, how do you like, what, what, what do you think you do like particularly better in this version of the model? Like, kind of where do you look to improve models at this point?

• 4:40Speaker 00

  I think, you know, in all, in all parts of research, like taste model matters a lot. And taste doesn't mean like aesthetics. It does means aesthetic in some point, but it's not the only thing. I think taste, I think, how do you train these models? And there's like thousands of different knobs and parameters and chips and tricks and like things we know have worked that you need to keep in mind. And I think I don't think there's one single like element to why models are getting better over time. I think it's it's a combination of these these elements. It's I was like listening to Elias like podcast from from a couple of days ago. And I really like the idea of phrasing like this era's the era of research, but with bigger computers.

• 5:21Speaker 00

  So we're back to like, you know, fundamental research, you need to understand and spend a lot of time doing doing science, trying experiments. And so if you're really good at trying experiments, it's, I would say it's one of the things that works really well here.

• 5:34Lukas Biewald?

  But like specifically, I guess, like, where is the frontier here? Like, like, where do models like still kind of underperform? Like, what are what are some things that like your latest model, you know, can do that like a previous generation couldn't? Or or like, where in the leaderboard do you think you're like, you know, getting an advantage and why?

• 5:52Speaker 00

  I think there's definitely a lot of things that models can do today that they couldn't do before. But it's still like there's other stuff that they're not able to like do today that I'm sure will be able to solve in a hopefully in the future. I think overall, this idea of like world understanding has become more evident that the models are really good reasoning systems that understand like spatially temporal spatial like consistency that can understand like cause and effect, that can understand like just the world. And the implications of it are kind of like pretty broad. There's ways of customizing or fine tuning these models so they can be also useful in other domains. And that becomes like extremely interesting from an like general intelligence perspective where

• 6:34Speaker 00

  I think one of the bottlenecks of language models is that language is always constrained by by by what language actually is, which is a human obstruction of reality, right? We've we've created this uh this mechanism for us to communicate with each other and describe the world, but it's not an accurate representation of the real world. It's an obstruction of the world. And so being able to train not a language alone, but on like observational data, on real data and video data, allows the model to um kind of grasp reality and how the world works in a much more consistent way. Um, and a lot of the work we're doing now is kind of heading towards like what that means and how you

• 7:15Speaker 00

  ex-scale and extrapolate that sort of like the abilities to hopefully do much more than just like video generation.

• 7:23Lukas Biewald?

  Are there any particular like prompts or things you've tried recently where you're like, ooh, that's like super cool?

• 7:28Speaker 00

  Yeah, there's there's a lot. I have like internal benchmarks on which I've I always try, you know, to see how models are doing.

• 7:35Lukas Biewald?

  Ooh, can you tell one or two of those?

• 7:37Speaker 00

  Uh, sure. There's like, uh, I have this this kangaroo uh pushing another kangaroo, uh, like a baby kangaroo in a in a stroller. It's it's complex because like you need the motion of the kangaroo to be consistent with how actually they move. Uh, but just hearing how the camera should follow, it's a hard prompt. And uh now the model has has kind of nailed it pretty well. Uh, I think physics in general, like the model does pretty good, like object permanence and like movements of things. Like human motion, it's one of the things that is very hard to like accomplish well. Uh, the model does pretty good at that. And and if something we spend a lot of time and if if you're if you ever want to try the model at some point, like being able to follow cameras like movements with precise like descriptions.

• 8:20Speaker 00

  So you can like pan and zoom and kind of like focus and go back to another focus all in a single sequence. It it breaks a little bit of this like, I would say, AI feel that a lot of this previous models had that were like very kind of like consistent with single shot kind of tripod cameras, which which I think a lot of great stories and great videos are more like more than that. And so we've we managed to like really nail, I would say, how camera movements have a work in the model.

• 8:48Lukas Biewald?

  I've always felt like you've had this perspective of, hey, like, we don't want to just make the most like flashy, like impressive model, but we want to make one that's like really useful for like, you know, telling stories. And I feel like you probably are focused on some things outside of what might like win a competition like this, like, you know, character consistency and things like that. Is that, that must also be a consideration for you, right?

• 9:09Speaker 00

  Yeah, of course. I mean, I think storytelling and media is one of the most like compelling use cases of the models in the first place. And it's the obvious one, it's the first one, but you really need to think about these models as, I would say, the next frontier of intelligence, where the models can reason and be used in other sort of domains beyond just like arts and media. And I think in a way, like two or three years ago, the question we had to ask ourselves was like, can you actually make great art and great films and great media with models? And I think that question has been like solved by now. And you have, we have a film festival that we've been hosting every year. The last, this year that we had, it was like, it was the packed Lincoln Center in New York.

• 9:53Speaker 00

  We have artists from all over the world come to it and make these amazing like films. And so in a way, like, I think that answer is already like solved. But now you can see how the models are becoming way more than just like for for entertainment and something we're looking into it.

• 10:08Lukas Biewald?

  So one thing that happened in one of our recent, you know, hackathons that we do is someone took one of, like a model like this and they used it to then train a robot to to use its arms correctly by generating synthetic data with video data, which just seemed like super cool and evocative. Is that the kind of thing you're talking about or?

• 10:30Speaker 00

  Correct. Yeah, there's ways of doing like physical or embody like AI with with world models with the models that we have. And and I think that that idea is, I would see largely still unexplored because like the consistency of the models wasn't there yet. And you can think about how you can manipulate the physical world with this simulation. They're basically simulation systems. And it's something we've been kind of advocated for a long time is the best way to understand video models is to understand them as basically universal simulation engines. They can effectively try or will try to simulate hopefully everything. And we're simulating entertainment and media first, but you can see how the models will start to like simulate way more than just that.

• 11:12Lukas Biewald?

  Well, give me some more examples.

• 11:14Speaker 00

  Uh, well, I think gaming is another one, nonlinear experiences like narrative, linear content as I would say like well films, videos, like things you you think you you've seen that are pre-made. But but the moment you have really good consistency and permanence and like a bit more of the deterministic like world building. Um, and you have it in real time, then you can have all this like nonlinear kind of uh type of content, uh we maybe we can call it just experiences, they're like live experiences in which you can either move around or have conversations with people or like see things play out in the real world in in real time. That for me might might be, it might resemble like a video game, and I think there's a lot that we're borrowing from the language of games.

• 11:59Speaker 00

  But effectively, it does feel like a new medium altogether. Because the way you interact with it, the way you make, the the way it's developed in the first place, it's different. And I think that thinking about it from well, from an art, from an artistic perspective, thinking about it as a new medium is incredibly exciting. This is like all these new things you could do with it. But also, it's like all the like adjacent things that you can also kind of use these models for, like if you want to learn, like I've, I've pretty much every time I want to learn something, I probably watch a YouTube video these days. Um, imagine a world where we can have uh like customized learning experiences and videos that are completely generated on real time just for you. I think we're not that far away from that.

• 12:40Lukas Biewald?

  It's really exciting. Um, I guess one one selfish question, you know, I love using this model with my my daughter who's who's six and and we um we play this stuff all the time, but one of the things that drives her nuts is like you can't you often put like kids in the models, it often gets like flagged. Have you thought about this at all? Like, I do think like kids are kind of the new generation of creators, but the fact that like there's so many guardrails, I mean, I I understand why you'd want to have guardrails, but I also think it's um, I don't know, it's it's like, I would say it's my my daughter's biggest objection to these media generation models.

• 13:13Speaker 00

  Yeah, I mean, look, there are um, that's our problem, like you've you've and and everyone has different approaches. I think we've taken a a more like safe and consistent approach to how we think we should develop the models and moderation will get better and like the ways to um, you know, uh. Making sure the outputs are safe will get better over time. I've done a really incredible job at that. We have a team of just dedicated to trust and safety to make sure we know how how uh video moderation can be done effectively. Um but yeah, I mean, I also have a daughter. You want to make sure I guess you can like make stuff with them and it's just a fun experience like there's I would say there might be a way where you could do something like what um

• 13:54Speaker 00

  what a lot of streaming services do these days where you can like have us kids like safe kind of an option where some kind of rules have been applied and use the power and can decide which ones. Um and if you're even a move them, you can remove them. I think probably that will be a maybe a consistent or a more interesting way of uh working towards solving that.

• 14:12Lukas Biewald?

  Well, if you ever if you ever uh release that, you'll have a fan in me.

• 14:16Speaker 00

  Yes, totally. We're working on that.

• 14:18Lukas Biewald?

  Awesome. Um well, thanks so much. Congratulations on uh on on coming.

• 14:23Speaker 00

  Of course. Yeah, well, thank you for the support. It's been quite a a a in a great journey and uh by the way, all the models are are trained always with weights and biases. So uh big shout out to you guys. Uh you've you've helped us a lot to get here.

• 14:36Lukas Biewald?

  Thanks so much. Thanks so much for listening to this episode of Grading Descent. Please stay tuned for future episodes.