The Timeline for Realistic 4-D: Devi Parikh from Meta on Research Hurdles for Generative AI in Video and Multimodality

Video dominates modern media consumption, but video creation is still expensive and difficult. AI-generated and edited video is a holy grail of democratized creative expression. This week on No Priors, Sarah Guo and Elad Gil sit down with Devi Parikh. She is a Research Director in Generative AI at Meta and an Associate Professor in the School of Interactive Computing at Georgia Tech. Her work focuses on multimodality and AI for images, audio and video. Recently, she worked on Make a Video 3D, also called MAV3D, which creates animations from text prompts. She is also a talented AI-generated and analog artist herself. Elad, Sarah and Devi talk about what’s exciting in computer vision, what’s blocking researchers from fully immersive Generative 4-D, and AI controllability. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Devi Parikh - Google Scholar Text-To-4D Dynamic Scene Generation named MAV3D (Make-A-Video3D) Full Research Paper Website with examples of image to 4 D generation Devi’s Substack Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @DeviParikh Show Notes: (0:00:06) - Democratizing Creative Expression With AI-Generated Video (0:08:31) - Challenges in Video Generation Research (0:15:57) - Challenges and Implications of Video Processing (0:20:43) - Control and Multi-Modal Inputs in Video (0:25:50) - Audio's Role in Visual Content (0:39:00) - Don't Self-Select & Devi’s tips for young researchers

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Text prompts are democratizing creative expression, and the Holy Grail is AI-generated and edited video. Elad Gill and I sit down with Devi Parikh. She's a research director in generative AI at Meta, a leading researcher in multimodality in AI for visual, audio, and video. And she's an associate professor in the School of Interactive Computing at Georgia Tech. Recently, she worked on "Make a Video 3D", which creates animations from text prompts. She's also a talented artist herself. Devi, welcome to No Priors.

• 0:35Speaker 01

  Thank you. Thank you for having me.

• 0:37Sarah Guo?

  Let's start with your background and how you got started in computer vision. I've heard you say you choose projects based on what brings you joy. Is that how you got into AI research?

• 0:47Speaker 01

  Kind of, kind of, yeah. So my background is that I grew up in India, and then I moved to the US after high school. And I went to a small school called Rowan University in Southern New Jersey for my undergrad. And that is where I first got exposed to what at the time was being called pattern recognition. We weren't even calling it machine learning, and got exposed to some research projects. There was a professor there who showed some interest in me, thought I might have potential to contribute meaningfully to research projects. And that's how I got exposed. And I really, really enjoyed what I was doing there, decided to go to grad school, to Carnegie Mellon. I knew I was enjoying it, but I wasn't sure if I wanted to do a PhD. So at first, I wanted to just get a master's degree

• 1:31Speaker 01

  with a thesis where I can do some research. But the year that I applied, that the ECE department at CMU decided that there wasn't going to be a master's track for thesis. Like, either you can just take courses or you go to a PhD. And so they slotted me onto the PhD track, which I wasn't so sure of. But my advisor there was reasonably confident that I'm going to enjoy it and I'm going to want to keep going. So yeah, that's how I got started in this space. At first, I was doing projects that didn't have a visual element to it.

• 2:02Sarah Guo?

  How did you pick a thesis project?

• 2:04Speaker 01

  So at first, I wasn't, I was working on projects that didn't have too much of a visual element to them. But when I got to CMU, my advisor's lab was working in image processing and computer vision. And I always thought that it was pretty cool that everybody gets to kind of look at the outputs of their algorithms and see what they're doing. Whereas if it's kind of non-visual, then yeah, you see these metrics, but you don't really have a sense for what's what's happening, if it's working, if it's not. And so that's how I got interested in computer vision and that then defined the topic of my thesis over the course of my PhD.

• 2:39Sarah Guo?

  So you have been working in machine learning long enough that, as you said, it was called pattern recognition and you've worked across a bunch of different modalities. How does, how has that changed your, your research path? Because like things like diffusion models and GANs and large transformers, none of that existed when you were first starting. And I, I think you have managed to sort of translate or transition your interests in a way that keeps you on the cutting edge. How, how has that happened?

• 3:09Speaker 01

  Yes, I think, I mean, you can always kind of look back and try and find patterns. Like when you're actually doing it, you don't necessarily have a grand strategy or anything in mind. But when I look back, I think one common theme that led to me transitioning across topics a little bit was that I was interested in seeing how we can get humans to interact with machines in more meaningful ways. And so kind of even my transition from kind of non-visual to visual modalities and hindsight, I feel like was essentially that. I felt like you can't interact with these systems too much if it's sort of these abstract modalities that you're looking at. And then when I was working in computer vision, I wanted to find ways for humans to be able to interact with these systems more.

• 3:50Speaker 01

  So I started looking at kind of these attributes and adjectives of like, oh, something is furry or something is shiny. And using that as a mode of communication between humans and machines, both for humans to teach machines new concepts and for machines to be more interpretable and explaining why they're making the decisions that they're making. And that slowly led to the sort of more into natural language processing, where instead of these kind of just adjectives and attributes, looking at more natural language as a way of interacting. So a lot of my work in visual question answering, where you're answering questions about images, image captioning. Was coming from there. And then over time, I sort of started thinking of are there ways to go even deeper in this interaction? There are other ways where AI tools can enhance sort

• 4:32Speaker 01

  of creative expression for people, give them more tools for expressing themselves. And that's how I got interested in AI for creativity. And I was dabbling in kind of a few fairly random projects a few years ago. Kind of my bread and butter research was still multimodal vision and language. But then sort of I enjoyed what I was doing with AI for creativity. And a couple of years ago, that took sort of a little bit more of a serious turn, but I made it more of my sort of full fledged research agenda. And that's how I started working more seriously on generative modeling, including transformer based approaches, diffusion models for images, for video, for 3D video, things of that sort.

• 5:12Sarah Guo?

  So you became a professor and you also, you know, now work in industry at Meta. What brought you there?

• 5:18Speaker 01

  So this was, I've been at Meta for about seven years now. And so this had started when I was transitioning from Virginia Tech to Georgia Tech. So I was an assistant professor at Virginia Tech. And I was getting started at Georgia Tech. And in that transition, I decided to spend a year at FAIR. At the time, it was called Facebook AI Research. Now fundamentally, I research at Meta. And I knew colleagues there from, some of them had been at Microsoft Research. Before that, I had interned at MSR. I had spent summers at MSR even as a faculty member. And so I had a lot of colleagues who I knew. And so I thought it would be fun to kind of spend a year, collaborate with them, get to know what FAIR is like. So that's what that was. It was supposed to be a one year stint.

• 6:01Speaker 01

  And then I was going to go back to Georgia Tech and kind of continue with my academic position. But in that one year, I enjoyed it enough. I think FAIR enjoyed having me around enough where we tried to figure out, is there a way to keep this going for longer? And so for many years after that, for five years or so, I was splitting my time. But every fall, I would go back to Georgia Tech, to Atlanta, spend the fall semester there, teach. And then the rest of the year, I would be in Menlo Park at Facebook, now Meta.

• 6:30Sarah Guo?

  And you transitioned from fundamental AI research to a new sort of generative AI group. Can you talk about why that's interesting to Meta or sort of what, what kind of things you're working on now?

• 6:39Speaker 01

  Yeah, yeah, I mean, yeah, that's a very exciting space right now. There's a lot happening both within Meta and outside, as I'm sure many of the people listening to this are aware of. But yeah, so that the new organization was created a few months ago, so not a long, not a long time ago. And it's looking at things like large language models, image generation, video generation, generating 3D content, audio, music.

• 7:06Speaker 01

  Yeah, all sort of, all sorts of modalities that you might think of. And why is it interesting? I mean, like right now, if you think about all the content, there's so much content that we consume in all modalities and all sorts of surfaces. And it makes a lot of sense to ask that instead of, maybe not instead of, but in addition to all of this consumption, can more of us be creating more of this content, right? And so almost everything that you think of images, video, you can ask this question, like for any situation where you're searching for something, trying to find something, it's relevant to ask, well, could I just create what it is that I have in my head? And so when you think of it that way, you can see how it can touch a lot of different

• 7:49Speaker 01

  things across a variety of products and surfaces. So, yeah.

• 7:53Speaker 02

  Yeah, that makes a ton of sense. I think we'll come back and ask you some questions about images and audio and a few other things, since you've done so much interesting work across so many different areas. But maybe we can start a little bit with video generation, in part, you know, due to the fact that you had a really interesting recent project called Make a Video. And in that approach, users can generate video with a text prompt. So you could type in, imagine a corgi playing with a ball, and it would generate a short video of a corgi playing with a ball. Could you tell us a bit more about that project, how it started, and also how it works? And what's the basis for the technology there?

• 8:27Speaker 01

  Yeah, yeah. So Make a Video, it started because, I mean, this was a couple of years ago where, this was before DALL-E 2, by the way. So this was like after DALL-E 1 had happened. And I feel like a lot of people don't even remember DALL-E 1 anymore. Like, people don't even talk about that. It's fun to go check out what those images look like. And that had, that had blown our minds at the time, but now when you go back, you're like, wait, like, that's not even interesting. But anyway, so we had seen a lot of progress in image generation. And so it seemed like the next kind of entirely open question, where we hadn't seen much work at all, was to see what can we do with video generation? And so that was kind of the, the inspiration behind that.

• 9:06Speaker 01

  And for make a video, the approach specifically, the thinking was we have these image generation models by this time, we had seen a lot of progress with diffusion based models from a variety of different institutions. And so the idea was, is there a way of leveraging all the progress that's happened with images in a very direct way to sort of make video generation possible? And so that led to this intuition that what if we try and use images and associated text as a way of learning what the world looks like and how we, how people talk about the visual content and then separate that out from trying to learn how the how things in the world move. So separate out appearance and language and

• 9:51Speaker 01

  that correspondence from motion of how things of how things move. And so that is what led to make a video. And so the there are sort of multiple advantages of thinking of it that way. One is there's less for the model to learn because you're directly bringing in everything that you already know about images to start with. The second is all of the diversity that we have in our image data sets that image models already know all sorts of fantastical depictions of like dragons and unicorns and things like that, but you may not have as much video data easily available. All of that is inherited. So all the diversity of the visual concepts can come in through images, even if your video data set doesn't have all of that. And the third benefit, maybe the biggest

• 10:36Speaker 01

  one is that because of the separation, you don't need video and associated text as spare data. You have images and text as spare data, and then you just have one labeled video to learn motion from. And so these were kind of three things that we thought were quite interesting in how we approached make a video. And so concretely, the way it works is that when you initialize the model, you're starting off with image generation sort of parameters that have already been learned. So before you do any training for make a video, you you we set it up so that it can generate a few frames that are Not temporally coherent. So there is going to be independent images like the corgi playing with the ball is just going to be independent images of corgis playing with

• 11:19Speaker 01

  blue balls, but they're not going to be temporally coherent. And then what the network is trying to do as it goes through the learning process is to make these images temporally coherent. So that at the end of training, it is generating a video rather than just unrelated images. And that's where the videos come in as training data.

• 11:37Speaker 02

  That's a great explanation.

• 11:39Sarah Guo?

  One question, just like if we if we use an example of something that is not going to be in your video training set, right? So I want a flying corgi, for example, right? How should I think of this in terms of like interpreted motion?

• 11:57Speaker 01

  Yeah, so one way of thinking of it could be that you may not have seen a flying corgi, but you've probably seen flying airplanes or flying birds and other things that fly in images and in video. And so from images, you will have text associated with it. So you will have a sense for what things tend to look like when someone is saying, oh, this is X flying or Y flying. And then in videos, you will have seen the motion of what stuff looks like when it when it flies. And in images, you'll have seen what corgis look like. And so it's hard to kind of know for sure what it is that these models learn. Sort of interpretability is not a strength of many of these deep, large architectures. But that could be one intuitive explanation for how the model is managing to figure out what a flying corgi might look like.

• 12:41Speaker 02

  Yeah. Well, what are some of the major forward looking aspects of this sort of project and research?

• 12:48Speaker 01

  I think there's a ton to do in the context of video generation. Like if you look at make a video, it was very exciting. It was sort of first of its kind capabilities at the time. But it's still it's a four second video. It's essentially an animated image, right? It's kind of the same scene, the same set of objects that are moving around in reasonable ways. But you're not seeing objects appear, objects disappear, you're not seeing objects reappear, you're not seeing scene transitions. None of this is is in there. And so if you look at, if you think about the complexity of videos that you just regularly come across on various surfaces, this is far from that. And so there is a ton to be done in terms of making these videos longer, more complex, having memory so that if an object reappears,

• 13:33Speaker 01

  it's actually consistent. It doesn't now look entirely different.

• 13:37Speaker 01

  Things of that sort, sort of being able to tell more complex stories through videos. All of this is entirely open.

• 13:44Speaker 02

  Well, and I know these things are always extremely hard to predict. But if you look forward a year or two, what do you think the state of the art will be in terms of length of video? Complexity of the scenes that you can animate, things like that.

• 13:54Speaker 01

  Yeah, that is, that is hard to say. And to be honest, I've actually been surprised that we haven't seen more of this already. Like Make-A-Video was, I think, what, nine months or so ago, maybe approaching one year? And it's not like, even from other institutions, it's not like we are seeing amazingly longer videos or significantly higher resolution or much more complexity. We're still kind of in this, videos equals animated images. And yeah, maybe the resolution is a little bit bigger, quality is a little bit higher. But it's not like we've made significant breakthroughs, unlike, for example, what we've seen with images. So I do, I, I, that has given me a sense that maybe this is harder than what we might think.

• 14:36Speaker 01

  And sort of our usual curves of like with language models or image models, we're like, oh, just six more months. And there's going to be something else that's an entirely different step change over this. I think that might be harder in video, and I wonder if there is something that we're kind of fundamentally missing in terms of how we approach video generation. So it's not quite answering what you asked me, but I do think that it might be a little bit slower than what we might have guessed, just based on progress and other modalities.

• 15:02Speaker 02

  What do you think is the main either challenge or bottleneck that you think has slowed progress in this field or not slowed it? I mean, obviously, there's been, you know, every, a lot of people are working very hard on these problems. But to your point, it seems like sometimes you have these fundamental breakthroughs. And sometimes it's like, it's an architecture, like transformer based models versus traditional NLP. And sometimes it's, you know, iterating on a lot of other things that already exist in the preexisting approaches and just sort of solving specific engineering or technical challenges. If you were to sort of list out the bottlenecks to this, what do you think they're likely to be?

• 15:33Speaker 01

  Yeah, I think there's a few different things. One is videos are just sort of, from an infrastructure perspective, harder to work with, right? They're just sort of larger, more storage and sort of more expensive to process and more expensive to generate and all of that. So there's just that iteration cycle that is much slower with video than it would be with other modalities. So that is, is one. The second is, I don't, I don't think we've still figured out the right representations for video. Right? There is a lot of redundancy in video. One frame to the next frame, there's not a whole lot that changes. We still kind of approach them fairly independently as sort of independent images.

• 16:12Speaker 01

  Even if you're generating it as kind of one after the other, or even if you're generating in parallel and then making it finer grain. So I think maybe that could be something that helps with a breakthrough that if you really figure out how to represent videos efficiently. And the third is this hierarchical architecture, that if you want longer videos, there are just so many pixels that you're trying to generate, right? It's a very, very high dimensional signal compared to anything else that we're doing. And so just thinking through how do we even approach that? What sort of hierarchical representation makes sense, especially if you want these scene transitions, if you want to have this consistency, which may be a form of memory.

• 16:53Speaker 01

  Figuring those architectural pieces out, I think maybe another piece of this puzzle. And then finally, data, right? Data is kind of gold in anything that we're, we're trying to do. And I don't know if as a community, we've quite built the muscle of thinking through data, sort of massaging the data appropriately and all of that in the context of video. We have that muscle quite a bit with language, quite a bit with images, but with video, we're perhaps not quite there yet.

• 17:20Speaker 02

  But what would be the ideal training set for, for video in that case? Or what's lacking from the existing approaches?

• 17:27Speaker 01

  Yeah, I think what's lacking may not be so much the data source itself, although that is certainly a challenge as it is with other modalities. But I think it might also be the data recipes that do we want to start with training with sort of these very short videos where not much is happening. The scene isn't really changing, but then that also tends to limit the motion. There's just not much happening and so you can end up with these kind of animated image looking things. And on the other hand, you have sort of very complex video might be multiple minutes long with all sorts of scene transitions. And that's ideally what you want to shoot for. That's where you want to get. But if you just kind of directly throw all of that into the network, it's unclear if the models will be able to learn all of that complexity well.

• 18:09Speaker 01

  So I think thinking through some sort of a curriculum may be valuable here. And I don't think we've quite nailed that recipe down.

• 18:17Speaker 02

  I feel like every generation of sort of technology shifts always runs into video is the hardest thing to do. And if you look at sort of just the first substantiation of the web, one of the reasons YouTube sold was the infrastructure point you made earlier, where just dealing with that huge amount of streaming and the cost associated with it and everything else, even in the prior generation of just, you know, can we host and stream this effectively? In part led to them, you know, getting sold to Google reasonably early in the life of a company. So it's interesting how video is always that much more complicated.

• 18:46Speaker 01

  Yeah, yeah. And same thing for computer vision, right? Like here we're talking about generation, but even just understanding with images, with image understanding, there was so much progress that was being made. And videos was always kind of not only trailing behind, but just sort of continued to be harder, even sort of the rate of progress was slower, not just the absolute progress. And I think, yeah, I think we're seeing some of that for generative models as well.

• 19:09Sarah Guo?

  Debbie, I know this isn't within your like core field, but I'm sure you also pay attention. Like, how do you think advances in video may like impact robotics?

• 19:22Speaker 01

  So I think there the video understanding piece is probably more relevant than the video generation piece. And video is like if you think of embodied agents, right, they're sort of moving around and consuming visual content, which inherently is video, right? They're not looking at static images. And so I think that video understanding piece is is very relevant there. What's also interesting in the context of embodied agents or sort of robotics, physical robots that are moving around, is that it's not passive consumption of videos, right? It's not like how you and I might be watching videos on YouTube or anything else. It's that the

• 19:57Sarah Guo?

  I'm yelling at the screen. I'm not passive.

• 20:02Speaker 01

  It's that the the next visual signal that the robot will see will be a consequence of the action that the robot had taken, right? So if it chose to move a certain way, that's going to change what the video looks like in the next few seconds. And so there's that interesting feedback loop there, where it knows what action it had taken, it sees how that changed the visual signal that it is that it is now getting as input. And so that connection makes it adds a layer of interestingness to how it can process the video, sort of in contrast to with sort of regular computer vision disembodied tasks where we think of sort of it just streaming a video is just kind of happening and you're not controlling what you're seeing.

• 20:42Sarah Guo?

  You started by saying that human interaction was a big driving force in in your research interests and, you know, going beyond like metrics as outputs and even language as as inputs. How do you think about controllability in video and like how important text prompting is to sort of the next generation of creation?

• 21:07Speaker 01

  Yeah, I think that's I think that's very important, exactly to your point, that if we want these generative models, not just for video, but for any modality, to be tools for creative expression, then it needs to be generating content that corresponds to what someone wants to express, right? Like it has to bring somebody's voice to life, and that is not possible if there aren't good enough ways of controlling these models. And so text is one way. That's better than random samples. That's that's one way in which I can I can say what I want. But right now, for the most part, You type in a text prompt, you get an image back, a video back. And either you take it or leave it, right? Like, if you like it, that's great. If not, you're just gonna try again. And maybe you tweak the prompt a little bit.

• 21:52Speaker 01

  You sort of try a whole bunch of these prompt engineering tricks and and and hope that you get lucky. But it's not really a very direct form of control. And so I think of more control, at least in two different ways. One is to allow for prompts that are not just text, but are multimodal themselves. So for image generation, for example, instead of just text, it would be nice if I can kind of sketch out what I want the composition of the scene to look like. And and the model would be expected to kind of respect that. For video, instead of just text as input, maybe I can also provide an image as input so that I can tell the system that this is the kind of scene that I want. Maybe I can provide sort of a little audio clip as input to convey that this is the kind of audio or sound that I want associated with it.

• 22:37Speaker 01

  Maybe I also bring in a short video clip and expect the model to sort of bring in all of these different modalities in a reasonable way to to generate a video. So that's one piece where I can bring in more inputs as a way of more control. And the second piece is sort of the predictability part, that even if I bring in all of these modalities as input, if the model then goes off and kind of does its own thing with these inputs, maybe it's reasonable, but that's not what I'm looking for. What do I do, right? Like, do I just go back and try again? It would be ideal if there's some way of having iterative editing mechanisms where whatever I get back, I have a way of communicating to the model what it is that I want changed, in what way,

• 23:16Speaker 01

  so that over iterations, I can get to the content that I intended in sort of a fairly reasonable way, without having to sort of spend hours learning a new tool or something like that, right? If that can be done in a very intuitive interface, I think that would be pretty awesome.

• 23:32Sarah Guo?

  Where do you think we will get to in terms of the frontier of like controls for video generation over the next couple of years or five years?

• 23:40Speaker 01

  I think control sort of tends to lag behind the core capability. Like even with images, I feel like we first had to get to a point where these models can actually generate nice looking images before we start worrying about, well, is it really doing what I wanted it to do? And I feel like we're not quite there with video yet. So get random good first. Exactly, exactly, exactly. Like at least get random good first, then maybe let me give it text, then let me give it these other prompts. Um, so I do think we'll first probably see more progress in just the core capabilities of sort of text to video generation. Um, before we look at uh uh prompting, although we are, and this is in the context of sort of me generating something from scratch, right? Which is where I might want this iterative control and things like that.

• 24:23Speaker 01

  A parallel scenario is where I already have a video and I'm trying to edit it in an interesting way. I might want to stylize it and all of that. I think we're already seeing that even in products um with with runway, for example, right? So I think that we'll probably see much more of uh, we're already seeing that. And I think we'll see more of where you already have a video that you're starting with. And then you're trying to edit it, um, which has similarities to, but is a little bit, is a little bit different in my mind. Um, compared to sort of generating something from scratch and wanting control over that.

• 24:52Speaker 02

  The other um potential uh part of output for videos obviously is text to speech or some sort of voice or other ways to sort of accompany the video or animate it. What is your view in terms of the state of the art of text to speech systems and how those are evolving?

• 25:07Speaker 01

  I think I've um, I have, I haven't tracked the text to speech quite as much. What I have tracked a little bit more closely is um things like text to audio, where you might say that the sound of a car driving down the street and what you expect is sort of a sound of a car driving down the street to be generated. Um, and so there uh the state of the art right now is um sort of roughly sort of a few second to tens of seconds long audio. Um, and I would say that roughly. It probably works reasonably well, uh, one in five times or so. Um, it's like, because there aren't concrete metrics, it's kind of hard to articulate uh where state of the art is. But um, hopefully this is, this is helpful.

• 25:51Speaker 01

  And I, I do think that um, audio added to visual content um, makes it much more expressive and much more delightful. And I do think that it tends to be um, under invested um, uh, both for audio, similarly for music. I think it just makes the content much more expressive, much more delightful. Um, but I feel like we don't do enough of that.

• 26:12Speaker 02

  Yeah, it's interesting too, because there are actually very large sound effect libraries out there. They're very well labeled as well in terms of what the exact sound effect is and the length and the components and all the rest. And so it's interesting that the state of the art hasn't quite caught up with, with, you know, what used to be a really interesting old business where you generate an enormous amount of IP for different sound effects. And then you just license them out.

• 26:32Speaker 01

  Yeah, yeah.

• 26:33Speaker 02

  Which it seems like eventually that industry is likely to go away, so.

• 26:36Speaker 01

  And even with audio, similar to what we were talking about with video, there is the right, like the same kinds of challenges and dimensions exist that you want the piece to be longer. Um, you may want compositionality, right? I might want to be able to say that, well, first it's the car driving down the street. And then there is a sound of, I don't know, a baby crying and then something else. And maybe I'm saying that two of these sounds are happening simultaneously, which is not like, that's something that can happen in audio where you can have this super imposition. Um, but in video is not something that would uh, where it's quite as natural. And so all of that isn't stuff that these models can do uh, very well right now. If I described a complex sequence of sounds, or if I tried to talk about these different sounds simultaneously, um, these models can't do that very well.

• 27:19Speaker 02

  Where do you think we'll see the first application areas? Or what, what do you think are the first sort of use cases that we'll see immediately? And then how does that evolve over time?

• 27:27Speaker 01

  Yeah, I think um, I'm not too much of a product person. So I feel like I, I don't know if I have the strongest intuitions there. Um, but I think for like, kind of like I was touching on earlier, that a lot of these situations where we find ourselves searching for things to express ourselves, um, I think thinking of whether that can be generated um, so that it's a closer reflection of what you're trying to communicate is likely things that we'll see. And I know we're not talking about sort of L L M's and conversational agents and all of that too much. But I think A I agents is going to be a thing that we'll see a whole bunch of um, across many different surfaces. And then thinking about what media creation looks like in the context of AI agents is is another dimension to this.

• 28:10Speaker 02

  Yeah, that makes a lot of sense. I mean, there's there's all sorts of obvious sort of near term applications in terms of generating your own animated GIFs, or to your point, midstream video editing, or, you know, different types of shorter form animations or other things that you could do, marketing, etc. And so, you know, it definitely feels like there are some near term applications and some longer term ones. And then the thing I always find interesting about these sorts of technologies is the spaces where they kind of emerge in a way that you don't quite expect, but end up being a primary use case. You know, it's sort of the Uber version of the of the mobile revolution where you push a button and a stranger picks you up in a car and you're fine with it, right? And it feels like those sort of unexpected delightful experiences are are going to be very exciting in terms of a lot of areas of this field.

• 28:53Speaker 01

  Yeah, yeah. And and to your point that this technology is brand new, right? So it's not like there are existing product lines or ways of thinking about product that we can kind of directly plug into and kind of see, oh, did the metric go up, did the metric go down? I think there's a lot of just kind of thinking about where do we anticipate people will be excited to use this. And I, as you said, I think I there's a very good chance that there will be things that we don't necessarily foresee, but it's kind of come up as very exciting spaces.

• 29:22Sarah Guo?

  I think an interesting cynicism has been that like there aren't that many artists out there, or like people don't want to create imagery when when looking at some of these generative, like much less video, when looking at some of these generative technologies. But, you know, the the recent history of social media would say that's like certainly not true, right? If you look at Instagram democratizing photography, or TikTok democratizing short form video creation by just like reducing the number of parameters of control, right? As you said, like, you know, sound makes video much richer, but it's also really hard to produce any one of these pieces. So you just take one control away and like, you

• 30:05Sarah Guo?

  know, record with your phone and you get you get something like TikTok. But I think it's really exciting, like the explosion and usage of things like Mid Journey, right? Because the traction suggests there are an awful lot of people who are actually interested in generating high quality imagery for a whole range of use cases, professional or not.

• 30:24Speaker 01

  Yeah, yeah. And I think there's people across the entire spectrum, right? Like, on one hand, you can talk about artists who already had a voice, were already involved in sort of um creating art. And then the other end of the spectrum are people who don't necessarily have the skills, may not have had the training, but are still interested in being able to express their voices a little bit more creatively than they would have otherwise. And so I do think that there is one question of whether or not artists want to be engaging with this technology. And there is the other question of does it kind of just lift the tide for all of the rest of us to be able to be more expressive in what we can create and what we can communicate. And so I think both of those ends are relevant here.

• 31:09Speaker 01

  And with artists, there are artists who are like whose sort of brand is AI artists, where they are explicitly using AI as the tool of choice for expressing themselves and their entire practices around that. Someone like Sophia Crespo or Scott Eaton and others. So there's also, and this was before Midjourney or anything like that, right? Like, they've been doing this for years, even with like GANs, for example, that existed that were popular before diffusion models and all of that. Yeah.

• 31:37Sarah Guo?

  You're an artist yourself, both, you know, digital, AI-driven analog. It's some of it's behind you. Like, how does that impact like your view of this?

• 31:47Speaker 01

  I I kind of always hesitate a little bit to call myself an artist. I feel like somebody else should be deciding whether I'm an artist or not, but then there's this whole community.

• 31:55Sarah Guo?

  We'll say you're an artist.

• 31:56Speaker 01

  Yeah.

• 31:57Speaker 02

  By the way, we should mention some of your lovely macrame art is on the wall behind you as well. So I think it looks great.

• 32:04Speaker 01

  Yeah, thank you, thank you. And yeah, so to be honest, I don't know if I, it's hard to kind of look back and get a sense for did that play a certain role in it or not. I know for sure that it plays a role in just how excited I am about this technology. That anytime there's some new model out there, whether it's from the teams that I'm working with, or if it's something external, I'm definitely very enthusiastic to want to try it out and see what it can do, what it can't do, and sort of tell people about it. And so just kind of my baseline level of excitement around this technology is high in part because of all these other interests that I have. I I'm pretty sure that my emphasis on control is probably also coming from that. But I feel like I want to be using these tools to kind of have them do the thing that I want to do.

• 32:49Speaker 01

  Um, and sort of text prompts are restrictive in that way. Um, and I mean, the we talked about it in the context of control, that if you can bring in multiple modalities as input, that definitely gives you more control, but it also means that there is more space to be creative. Right? I can now pick interesting images or interesting videos or interesting pieces of audio and pair that up with like this really interesting text prompt. And just kind of see what happens. Like if I put all of this in, you don't know what the model is necessarily going to do. And so it's also just more norms to play with um as you're as you're trying to um interact with these that yeah, there's just more more space to be creative. If there's more ways of more norms to control these models.

• 33:30Sarah Guo?

  Yeah, I was talking to um Alex Isra, who's an LA based artist. And uh he's not a technical guy, um but an amazing artist. And he was describing this new video project he wants to do that involves use of AI. And I was very inspired by uh like how specific the vision was and like thinking through the implementation a little bit for somebody who doesn't come from the technical field. And I imagine there'd be a whole crop of people who look at the capabilities as as as another tool for expression.

• 33:59Speaker 01

  Yeah, yeah. And so and there are some people who have a very specific vision and they just want the tool to kind of help them get there. And then there are others whose process involves sort of bringing the model along where the unpredictability and sort of not necessarily knowing what this model is going to generate is a part of their process and is a part of the final piece that they create. So some view them view these models very much as tools. And then others tend to view them as more of a collaborator um in this process of of creating. And it's always interesting to see what end of the spectrum different people lie on. Yeah.

• 34:34Sarah Guo?

  Okay, so as we're nearing the end of our time together, we want to run through a few rapid fire questions, if that's okay. Maybe I'll start with one, just given your breadth in the in the field. Um, is there an area of image, audio, video generation, understanding, control that you feel like is um just underexplored uh for uh people looking for research problems?

• 34:57Speaker 01

  Yeah, so one is the control piece that we already talked about quite a bit. And I think the other is multimodality, like bringing all of these modalities together. Right now, we have models that can generate text, a model that can generate images, models that can generate video. But there's no reason these all need to be independent. You can envision systems that are sort of ingesting all of these modalities, understanding all of it, and generating all of these modalities. And I haven't, I'm starting to see some work in that direction, but I haven't seen a whole lot of it that goes across many different modalities.

• 35:29Sarah Guo?

  You just got back from CVPR and presented there. Can you mention both what you were talking about and then sort of the project or work that most inspired you there?

• 35:39Speaker 01

  Yeah, so I was at CVPR. I was on a few different panels and I was giving some talks. So one of it was on vision, language and creativity at the, at the main conference. And so, yeah, that was kind of what I was representing there. In terms of something exciting that I saw there, not necessarily a paper, but there was a workshop there called Scholars and Big Models, where the topic of discussion was as these models are getting larger and larger, making a lot of progress, in what way can sort of academics or labs that don't have that sort of as many compute resources, what should their strategy be? How should they be approaching these things? And that I thought was a really nice discussion.

• 36:21Speaker 01

  In general, I tend to enjoy venues that talk about kind of the meta, like the human aspects of the work that we do. We always, we have a lot of technical conversations, but we don't tend to talk about these other components. And so that workshop is something that I enjoyed quite a bit.

• 36:36Sarah Guo?

  Is there a prediction you'd make about the impact of all these technologies on social media? Since you're working at the intersection?

• 36:44Speaker 01

  Yeah, there's going to be more of it. That's one, that's one prediction that I can very confidently make, that we are going to see all of these tools sort of show up where millions and billions of people can be, can be using these in various forms. And I'm excited about that. Like I said, I think it kind of enhances creative expression and just sort of communication and we're going to have these entirely new ways of interacting with each other. Then even the social entities on these networks might change, right? Like when we talk about AI agents and you think about them being sort of part of the social graph. What that does and how that changes, how we connect with each other. All of that is fascinating and I can't, I can't wait to see how that evolves.

• 37:29Speaker 02

  It actually feels a little bit under discussed in terms of how much impact it's already had in some ways, even if it sometimes is transitory like a lensa. Where, you know, there is, my understanding is tens of millions of people at least who ended up using that product. There's character in terms of the engagement there. One could argue mid journey for certain types of art you share with friends, et cetera. And so I feel like there's already these social expression modalities, bot based interactions, et cetera, that are already impacting aspects of social or communicative media in ways that people don't really recognize it as such. So to your point, it's very exciting to see where all this is heading.

• 38:02Speaker 01

  Yeah, yeah.

• 38:04Sarah Guo?

  David, you also write quite a bit online and in terms of sort of giving back to the community and advice for researchers or young people newer to the field. You talk about time management and other topics. What's one one piece of wisdom you'd offer to people in terms of productivity and joy and AI research?

• 38:23Speaker 01

  Yeah, so time management is something that I'm quite excited about. And so, yeah, if I have a blog post on that. It's sort of main philosophy is that you should be writing everything that you want to do down on your calendar. So it's kind of the point is that it should be on your calendar. It shouldn't be a to do list. And the reason it should be on your calendar is it forces you to think through how much time everything is going to take. If it's just a list, you have no idea how long it's going to take and that's not a good way to plan your time out. So that's kind of the main thesis of it. That I hadn't anticipated, but it resonated a whole lot with many people, which was kind of surprising when I put it out there. So yeah, if anyone's interested, you should check that out. In terms of advice outside of what I've written, one advice that has stuck with me over the years is don't self-select.

• 39:08Speaker 01

  That if you want something, go for it. If you want a job, apply for it. If you want a fellowship for any students who might be listening, just apply for it. You want an internship, just apply for it. And yeah, like, don't, don't assume, don't question, oh, am I good enough? Am I not? It's on the world to say no to you. If you are not a good fit, the world will tell you that. And so, yeah, there's nothing to lose by just kind of giving it a shot. So don't self-select.

• 39:35Sarah Guo?

  It's a great note to end on. Devi, thank you so much for joining us on No Priors.

• 39:39Speaker 01

  Thank you. Thank you for having me.

• 39:41Speaker 02

  Thanks for the time.