Generative Video WorldSim, Diffusion, Vision, Reinforcement Learning and Robotics — ICML 2024 Part 1

Regular tickets are now sold out for Latent Space LIVE! at NeurIPS https://lu.ma/LSLIVE! We have just announced our last speaker and newest track, friend of the pod Nathan Lambert https://www.latent.space/p/rlhf-201 who will be recapping 2024 in Reasoning Models like o1! We opened up a handful of late bird tickets for those who are deciding now — use code DISCORDGANG if you need it. See you in Vancouver! We’ve been sitting on our ICML recordings for a while (from today’s first-ever SOLO guest cohost, Brittany Walker https://x.com/brittwalker_?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor), and in light of Sora Turbo’s launch https://www.youtube.com/live/2jKVx2vyZOY (blogpost https://openai.com/index/sora-is-here/, tutorials https://www.youtube.com/watch?v=360ZqfabuPQ&list=PLOXw6I10VTv8q5PPOsuECYDFqohnJqbYB&index=1) today, we figured it would be a good time to drop part one which had been gearing up to be a deep dive into the state of generative video worldsim, with a seamless transition to vision (the opposite modality), and finally robots (their ultimate application). Sora, Genie, and the field of Generative Video World Simulators Bill Peebles, author of Diffusion Transformers, gave his most recent Sora talk at ICML, which begins our episode: * William (Bill) Peebles - SORA https://icml.cc/virtual/2024/39514 (slides https://slideslive.com/39022361) Something that is often asked about Sora is how much inductive biases were introduced to achieve these results. Bill references thesame principles brought by Hyung Won Chung from the o1 team https://www.youtube.com/watch?v=kYWUEV_e2ss- “sooner or later those biases come back to bite you”. We also recommend these reads from throughout 2024 on Sora. * Lilian Weng’s literature review of Video Diffusion Models https://lilianweng.github.io/posts/2024-04-12-diffusion-video/ * Sora API leak https://x.com/EHuanglu/status/1861446152924864908 * Estimates of 100k-700k H100s https://www.factorialfunds.com/blog/under-the-hood-how-openai-s-sora-model-works needed to serve Sora (not Turbo) * Artist guides on using Sora https://www.fxguide.com/fxfeatured/actually-using-sora/for professional storytelling Google DeepMind had a remarkably strong presence at ICML on Video Generation Models, winning TWO Best Paper awards for: * Genie: Generative Interactive Environments https://icml.cc/virtual/2024/oral/35508 (covered in oral https://icml.cc/virtual/2024/oral/35508, poster https://icml.cc/virtual/2024/39080, and workshop https://icml.cc/virtual/2024/workshop/29960#collapse-sl-39299) * VideoPoet: A Large Language Model for Zero-Shot Video Generation https://icml.cc/virtual/2024/oral/35537 (see website https://sites.research.google/videopoet/) We end this part by taking in Tali Dekel’s talk on The Future of Video Generation: Beyond Data and Scale https://icml.cc/virtual/2024/39508. Part 2: Generative Modeling and Diffusion Since 2023, Sander Dieleman’s perspectives (blogpost https://sander.ai/2024/09/02/spectral-autoregression.html, tweet https://x.com/sedielem/status/1820233922287919263) on diffusion as “spectral autoregression in the frequency domain” while working on Imagen and Veo have caught the public imagination, so we highlight his talk: * Wading through the noise: an intuitive look at diffusion models https://icml.cc/virtual/2024/workshop/29968#collapse-sl-39511 Then we go to Ben Poole https://icml.cc/virtual/2024/37921 for his talk on Inferring 3D Structure with 2D Priors, including his work on NeRFs and DreamFusion: Then we investigate two flow matching papers - one from the Flow Matching co-authors - Ricky T. Q. Chen (FAIR, Meta) https://icml.cc/virtual/2024/37945 And how it is implemented in Stable Diffusion 3 with Scaling Rectified Flow Transformers for High-Resolution Image Synthesis https://icml.cc/virtual/2024/oral/35548 Our last hit on Diffusion is a couple of oral presentations on speech, which we leave you to explore via our audio podcast * NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models https://icml.cc/virtual/2024/oral/35499 * Speech Self-Supervised Learning Using Diffusion Model Synthetic Data https://icml.cc/virtual/2024/oral/35493 Part 3: Vision The ICML Test of Time winner was DeCAF https://joltml.com/icml-2024/test-of-time-decaf/, which Trevor Darrell notably called “the OG vision foundation model”. Lucas Beyer’s talk on “Vision in the age of LLMs — a data-centric perspective” https://slideslive.com/39022228/vision-in-the-age-of-llms-a-datacentric-perspective?ref=search-presentations was also well received online, and he talked about his journey from Vision Transformers to PaliGemma. We give special honorable mention to MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark https://icml.cc/virtual/2024/oral/35497. Part 4: Reinforcement Learning and Robotics We segue vision into robotics with the help of Ashley Edwards, whose work on both the Gato and the Genie teams at Deepmind is summarized in Learning actions, policies, rewards, and environments from videos alone. http://Ashley Edwards, whose work on both the Gato and the Genie teams at Deepmind is summarized in Learning actions, policies, rewards, and environments from videos alone Brittany highlighted two poster session papers: * Behavior Generation with Latent Actions https://icml.cc/virtual/2024/poster/33379 * We also recommend Lerrel Pinto’s On Building General-Purpose Robots https://icml.cc/virtual/2024/39069 * PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs https://icml.cc/virtual/2024/poster/35217 However we must give the lion’s share of space to Chelsea Finn https://joltml.com/icml-2024/invited-talk-chelsea-finn/, now founder of Physical Intelligence, who gave FOUR talks on * "What robots have taught me about machine learning" https://icml.cc/virtual/2024/invited-talk/35253 * developing robot generalists https://icml.cc/virtual/2024/39071 * robots that adapt autonomously https://icml.cc/virtual/2024/39293 * how to give feedback to your language model https://icml.cc/virtual/2024/workshop/29963#collapse-sl-39699 * special mention to PI colleague Sergey Levine on Robotic Foundation Models https://icml.cc/virtual/2024/39068 We end the podcast with a position paper that links generative environments and RL/robotics: Automatic Environment Shaping is the Next Frontier in RL https://icml.cc/virtual/2024/oral/35495. Timestamps * [00:00:00] Intros * [00:02:43] Sora - Bill Peebles * [00:44:52] Genie: Generative Interactive Environments * [01:00:17] Genie interview * [01:12:33] VideoPoet: A Large Language Model for Zero-Shot Video Generation * [01:30:51] VideoPoet interview - Dan Kondratyuk * [01:42:00] Tali Dekel - The Future of Video Generation: Beyond Data and Scale https://icml.cc/virtual/2024/39508. * [02:27:07] Sander Dieleman - Wading through the noise: an intuitive look at diffusion models https://icml.cc/virtual/2024/workshop/29968#collapse-sl-39511 * [03:06:20] Ben Poole - Inferring 3D Structure with 2D Priors * [03:30:30] Ricky Chen - Flow Matching * [04:00:03] Patrick Esser - Stable Diffusion 3 * [04:14:30] NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models https://icml.cc/virtual/2024/oral/35499 * [04:27:00] Speech Self-Supervised Learning Using Diffusion Model Synthetic Data https://icml.cc/virtual/2024/oral/35493 * [04:39:00] ICML Test of Time winner: DeCAF https://joltml.com/icml-2024/test-of-time-decaf/ * [05:03:40] Lucas Beyer: “Vision in the age of LLMs — a data-centric perspective” https://slideslive.com/39022228/vision-in-the-age-of-llms-a-datacentric-perspective?ref=search-presentations * [05:42:00] Ashley Edwards: Learning actions, policies, rewards, and environments from videos alone. http://Ashley Edwards, whose work on both the Gato and the Genie teams at Deepmind is summarized in Learning actions, policies, rewards, and environments from videos alone * [06:03:30] Behavior Generation with Latent Actions https://icml.cc/virtual/2024/poster/33379 interview * [06:09:52] Chelsea Finn: "What robots have taught me about machine learning" https://icml.cc/virtual/2024/invited-talk/35253 * [06:56:00] Position: Automatic Environment Shaping is the Next Frontier in RL https://icml.cc/virtual/2024/oral/35495 This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  Welcome to the Latent Space coverage of ICML 2024. This is Charlie, your AI co-host. We know it's been a few months since ICML actually happened, but now that all the talks are available online and we are in final preparations for NewReps 2024, we figured this was a good time to release our conference recap to get you in the mood. As a side note, regular tickets are now sold out for Latent Space Live at NewReps, where we have announced our dream speakers to recap the best of 2024 across the top voted domains in vision, open models, post-transformers, synthetic data, small models, agents, GPU scaling, and a special 2024 in AI keynote

• 0:44Speaker 00

  from our friend and fellow podcaster Sarah Guo of Conviction Capital. Today, we are announcing our very last speaker and newest track, friend of the pod Nathan Lambert, who will be recapping 2024 in reasoning models like OpenAI's O1. See you in Vancouver. Coming back to ICML, this is a very special episode in more than one because it is the very first episode not hosted by swyx or Alessio. We are continuing to experiment with guest hosts, adding different opinions and voices to the show. And in this case, to cover conferences we personally weren't physically able to attend. So we're very grateful for our friend Brittany Walker of CRV

• 1:28Speaker 00

  to step in as your guest co-host for ICML 2024. Our goal with these conference recaps is to give you an audio experience of what it's like to be there and to provide a filtered recommendation of papers and backstories of authors that will be useful for the AI engineer today and tomorrow. Brittany worked enormously hard to put together the poster chats you will hear, and we're very grateful. Given that OpenAI has launched Sora Turbo today, we have bumped up our planned second episode to release first, since generative video happened to be a huge focus at ICML. Let's not bury the lead and go straight into the Sora talk from Bill Peebles, first author of the Diffusion Transformers paper

• 2:12Speaker 00

  and research scientist leading Sora model development.

• 2:17Speaker 00

  Since we're talking about video models, you may wish to tap into the show notes for direct links to the public talks. However, we think there is still value in editing the audio for eyes-free browsing. We believe this is the most recent public academic discussion of Sora before the Sora Turbo public release today, so we hope this episode is valuable background for anyone getting up to speed on video diffusion. Watch out and take care.

• 2:44Speaker 01

  I'm Bill, and thanks a lot to Joanna for organizing this conference. Really excited to be giving a talk here. So I'm going to be talking about Sora today. So this is Video Generation Models as World Simulators. This was joint work with my good friend Tim Brooks and also some other wonderful colleagues at OpenAI. So let's dive right in. So Sora is OpenAI's first video generation model. And in advance, I'm sorry for any kind of FPS delay with screen sharing videos. This is always the hardest part about working with videos, is showing results to other people over the internet. But this is a sample from Sora, and the text prompt is a stylish woman walks down a Tokyo street filled with warm glowing neon, and you can see the rest of it. Sora

• 3:27Speaker 01

  is capable of generating 1080p video up to a minute long. And what's remarkable about Sora is kind of all of the simple things that we take for granted about the visual world, it really begins to pick up on when you train on video data at scale. So if you see that blue sign in the background, even when there's a shot change and it's occluded, it's maintained. And we see this very consistently for a large number of samples from Sora. So it really has a good understanding, not only, for example, of how light interacts within scenes in complicated ways, but object permanence and lots of other capabilities that have been very difficult for video generation models to grok in the past.

• 4:04Speaker 01

  So of course, it can do more than just photorealistic styles. So this prompt is a gorgeously rendered paper-craft world of a coral reef, rife with colorful fish and sea creatures. So Sora, again, is capable of generating in non-photorealistic styles, can also do a number of scene transitions. So we didn't stitch these video samples together. This is all one continuous output from Sora. It's capable of figuring out that if you want a scene with a variety of sea life, maybe there should be a shot of seahorses, turtles, etc.

• 4:34Speaker 01

  And it's also capable of modeling complex scenes. So this prompts this beautiful snowy Tokyo city is bustling. And so there's a large number of people in the scene. You can see the camera is flying through. And while it's doing that, it's able to have, you know, interactions between people like this couple is holding hands. There are people selling goods at the stalls. There's soccer pedals flying through the air. So Sora has really begun to pick up on the intricacies of how scenes should look and do a great job at rendering them.

• 5:02Speaker 01

  One final example here is a movie trailer featuring the adventures of the 30-year-old spaceman. So what's cool about this is Sora's kind of zero shot learns that you should have character consistency throughout a number of scene transitions. So, you know, it knows movie trailers do not normally like change the leading actor halfway through. And so the man is the same across these different environments and different scenes. And all of this is just learned automatically by training on video data at scale. So now I want to go into a few technical details about Sora. A lot of the inspiration for Sora came from language models. And in particular, this notion of a unified representation of text data.

• 5:43Speaker 01

  So, you know, one of the key ingredients to the success of LLMs over the years has been this idea that, you know, you could take stories, you can take code, you can take math. But at the end of the day, all of this information is represented with a unified vocabulary being tokens, which makes it very easy to train on data at scale. This imbues language models with very generalist capabilities and makes them polymax at like a number of tasks. Now, we were really thinking like what the analog of this would be for visual data. And, you know, in particular, you know, there's no shortage of very diverse sources of visual data in the world. You know, there's vertical video, there's square images out there.

• 6:23Speaker 01

  You have like every kind of data of different durations, of different resolutions, of different aspect ratios. And the question is, how can you train on all of that in a unified representation? So we don't have to throw away any visual data. And so this is really one of like the key ingredients for the success of Sora is coming up with this unified notion of kind of a visual representation on which we can train on, you know, internet scale visual data. And so in order to accomplish this, uh, we use a VAE, kind of inspired by latent diffusion models from Robin Rambock. And what we do with this is encode all of this information into one unified latent space.

• 7:04Speaker 01

  So the idea here is on the far left, you know, we have like a video of a butterfly, uh, swimming underwater. We go through this visual encoder and this will compress videos both spatially and temporally into a single sequence of data. And at the end of the day, we do this, of course, so we can train transformers on this sequence of data. We train diffusion transformers at scale. And the benefit of this is we get a number of just great properties of scaling transformers up specifically for video and image data. So, you know, the name of the game here is like, how does visual quality improve as you throw more flops at the problem? And we find that improves like pretty steadily, which is great. So on the far left here, you know, we have a base compute trained Sora model.

• 7:48Speaker 01

  So this is trained with a small amount of compute. And you can see it gets like some details, right? So for example, it kind of has some idea of like, if a camera is moving through a scene, there should be some notion of consistency. But all the textures are wrong, uh, and it's not high fidelity. If you four X the amount of training compute you pump into that model, it begins to figure out kind of like, you know, what dogs look like, what humans look like. Um, but the visuals are still not great. And if you really crank up the amount of flops you're pouring into these things to thirty two X, you begin to see that it gets a lot of these like fine grain details, right? The interaction of like the owner's hand with the dog, all of like the snowy textures on the ground. And so we're finding that these models scale extremely effectively if you kind of nail the basics, right?

• 8:31Speaker 01

  So in particular, if you can create this, you know, setup where you have this unified representation of visual data and crank up diffusion transformers, uh, they can really start to learn to do amazing things. Another cool property of Sora is how generalist it is at test time. So you know, when you actually want to sample content, you can do it at any aspect ratio and resolution. And this is really great from the perspective of kind of like controllable generations, specifically as it relates to, you know, different devices. So now if I'm like watching a movie on my iPhone and then I transition to watching it on my laptop, those are going to use two totally different aspect ratios. And normally you either have to just like pad with black bars or crop it.

• 9:16Speaker 01

  But with models like Sora, it's now possible to generate content natively for any device, which is pretty exciting to think about the possibilities of how that can affect content creation in the future. So the sea turtle here is just rendered out with different aspect ratios.

• 9:31Speaker 01

  Another exciting aspect of this very generalist training recipe is we can kind of move on from the days of just like cropping data for training generative models. So you know, back when I was like in grad school, I was always like spending time, you know, cropping to like 256 by 256 resolution to train like whatever version of StyleGAN I was working with. And while that works well, it has certain downsides. So you know, there are certain biases actually within data. For example, the photographer's bias of centering objects. And so on the left here, we have a baseline Sora model where we don't train with native size video and image data. Instead, we actually do this like hard cropping to center. And you can see that the model essentially inherits some weaknesses of this cropping strategy, right?

• 10:15Speaker 01

  Sometimes like the scuba diver is going to be off-center, which isn't actually ideal framing. If you do this native size training, it's actually much more effective at composing scenes. So you inherit some nice benefits of the training data in the model by just, you know, not throwing away pixels and training on everything you have.

• 10:36Speaker 01

  So Sora is also an image generation model. So the prompt here is "digital art of a young tiger under an apple tree in a matte painting style with gorgeous details."

• 10:46Speaker 01

  Here's another sample. We find that Sora in particular really excels at photorealistic kinds of content. So there's a lot of details, kind of in the woman's face here, which it does a great job at rendering out. This is at 2K by 2K resolution.

• 11:04Speaker 01

  And of course, we can interact with Sora in other ways beyond just text. So all of the results before were text to video or text to image samples. But Sora can also accept visual inputs as conditioning. And so here we were seeding it with an image from Dolly 3 and then having Sora extend this out in time. So Sora is capable of kind of understanding, you know, what's going on in an image and then extrapolating from there. And so we had a lot of fun with this. So these are Dolly 2 samples on the left here. And so Sora can take video conditioning or image conditioning at any temporal index. So here we condition the model in the middle of the sequence with the Shiba Inu. And then we extend it both backwards and forwards in time from that position.

• 11:48Speaker 01

  And you can see it's able to animate the dog's face. Of course, it can also do kind of more fun animated styles here.

• 12:00Speaker 01

  We've been using this to make emojis internally. We have this nice Sora Slack emoji now.

• 12:07Speaker 01

  And another cool thing with Sora is its ability to extend backwards in time. So of course, you know, whether you're doing like temporal like outpainting like forward or backwards in time, it's all kind of like the same to these models. And so here we have the model end in the same way, which is with this San Francisco logo. But all of the events leading up to it are re-sampled by the model. So it's very flexible in how you can use this to edit or extend videos.

• 12:40Speaker 01

  Another cool aspect of Sora is its zero-shot editing capabilities. So there's been a ton of great work from the academic community over the years on finding creative ways to use diffusion models to do, for example, like image editing tasks. So, you know, one really nice work in that area is SD Edit. And we find that techniques like this, of course, just work right out of the box with Sora, because it's a diffusion model at the end of the day. So these are SD edit results. So the top left is the source video. This particular source video was generated with Sora, but of course it doesn't have to be. It could also be a real video. And you can use a variety of different text prompts to re-render this scene automatically. So for example, on the top right, we can rewrite the video in a pixel art style.

• 13:26Speaker 01

  And as you would expect of SD edit, if you kind of, you know, use the right like noise level, you can get it to maintain most of the structure in the scene. And just only update the style, which is cool. So towards the end of this video, you can see that there's like a cave that the car in the top left goes into. And across all of these like re-rendered styles, you see that it preserves like some notion of like a cave or like an overhang that the car goes through. Another thing that's cool is Sora is like kind of smart about figuring out whether or not, you know, certain correlations make sense. So for example, in the bottom right, you can say, change the video to a medieval theme. Sora knows there weren't cars in the medieval times, so instead you get a red horse carriage. So it's kind of fun to see where Sora takes liberties in re-rendering your video.

• 14:10Speaker 01

  Another cool capability that Sora can do is blend between videos. So the far left and far right videos here define the endpoints of this interpolation. And the middle video is Sora's imagining of how you connect the dots. And so you can see you get these kind of fantastic creatures in this case, where you can never quite see where it goes from being a chameleon to a bird. It happens very seamlessly.

• 14:38Speaker 01

  And you can use this for all kinds of scenes. They don't even have to be like particularly related. So on the far left here, we have a drone flying through the Colosseum. And the far right is the butterfly flying underwater. And you can see that Sora is able to come up with a pretty reasonable interpolation between these two videos. So you gradually see the Colosseum decay and move underwater. And at some points, the drone morphs into the butterfly very suddenly, because it kind of like has put these two things into correspondence automatically and infers that like this is like a reasonable thing that should focus on blending between.

• 15:16Speaker 01

  And here's an example of blending two scenes with totally different styles. So the far left is like a photorealistic aerial drone shot. And then the far right video is it's kind of nice like gingerbread village. And it comes up the really creative way to make this work. So rather than kind of morph the whole style of the scene in one shot. It decides that maybe this like gingerbread village is kind of hidden off to the side of this. This photorealistic town and it zooms in.

• 15:49Speaker 01

  So one other technique that we use for Sora is this notion of video recapping. And this is a technique that was actually pioneered by Dolly three by some other folks at OpenAI. And the high level idea is during training diffusion models and really alternative models benefit from having a much cleaner source of conditioning than we've historically given them in the past. So you know, there's like very crude text captions out there like all text, for example, which doesn't actually contain a lot of information about the scene. They're like very coarse keywords, for example, sometimes the content's like pretty unrelated actually to what's in your image or video, etc.

• 16:32Speaker 01

  And one of the key breakthroughs in Dolly three was generating synthetic captions that are much more detailed and contain much more mutual information with the content that you actually want to generate. And so what we saw with Dolly three, this is an example figure from Dolly three, is that this really improved the controllability of the model and enabled you to create much more intricate scenes with a lot more ease than in the past. And so we took inspiration with Sora to also apply this technique to video. And one of the features of this is that at test time, when you're actually interacting with the model, rather than just directly kind of upload a prompt to Sora,

• 17:16Speaker 01

  we'll actually use GPT under the hood to essentially up sample a user's base prompt into a much more detailed video description. And so this figure here is the system prompts that we used for Dolly three. In order to do this up sampling, it's actually pretty involved to get this to work well. And so there's a lot of like prompt engineering, even at OpenAI, to get these systems to be reliable. But under the hood, this is, this is what we're doing to achieve some of these kind of like finer grained control of Sora.

• 17:51Speaker 01

  So the last topic I want to talk about is this notion of like emerging simulation capabilities. And this is really the aspect that we are most excited about with Sora looking forward. You know, we, we often get asked the question, you know, at OpenAI, you know, how does video generation really relate to the core mission of AGI? And on the Sora team, we're actually really passionate about this being a model for world simulation moving forward. And so what, what exactly does that mean? Like, how do we actually use these models long term to do interesting tasks and to really extract intelligence out of the world? You know, what we believe is when we really scale up video generation models, they're going to get so good

• 18:31Speaker 01

  at simulating such a variety of complex scenes with, you know, different agents in them that it's going to need to ultimately learn an underlying model of how people interact, of how people do tasks, of how people think. If it's truly generating, you know, high fidelity content, like at some point, right? If the conversation I'm having at a dinner table within Sora is not realistic, that means it's failed to do its job of like accurately learning like the distribution of like human behavior. And so as we approach the limits, you know, of achieving like the irreducible loss there, we think like pretty amazing things are going to emerge from these models and it's going to play a really key role in developing more intelligent systems in the future.

• 19:12Speaker 01

  So Sora is obviously not there today, but we already see some cool phenomena by training on video data at scale that we just want to highlight. And we think this list is only going to grow in the future as Sora continues to scale up. So the first one I'll talk about is 3D consistency. And so this is pretty clear from a lot of the samples thus far, but you know, even when you have these very dynamic scenes with a lot of people moving in them and the camera. Being non-stationary, you can see that a large number of elements in the scene really do move with like what appears to be like accurate geometry. And so this is achieved without any kind of, you know, hardcoded inductive biases for 3D within the model. It's all learned jointly end to end as part of, you know, large scale diffusion training. It was really important to us when we were doing this project.

• 19:57Speaker 01

  That whatever solution we came to for video generation was scalable and could just absorb a lot of flops. And one way to do that, right, is to really strip out these inductive biases. That in the past have sometimes been useful for achieving certain kinds of behaviors at low scale. But it's not clear that when you really crank up the training compute. If they'll either help or hinder you. And so we find that it's totally fine to like not have these kinds of inductive biases as long as you're training at scale. Here's another sample. This one's kind of fun. So it's an aerial view of Yosemite showing both hikers as well as a gorgeous waterfall. The hikers do some very extreme hiking right here. I would not recommend trying this at home.

• 20:43Speaker 01

  And Ben Mildenhall, who used to be at Google, he took some Sora samples when we released them. And then he trained a nerf on them and in his words, it nerfs. So this is another kind of a nice sanity check that the underlying geometry that Sora is learning for some scenes, not certainly all yet. Is actually pretty accurate. And so it's cool to see that this again just emerges automatically at scale without inductive bias. So the next capability I want to talk about is this idea of long range coherence. So this is one of my favorite samples. This is the Bling Zoo shop in New York City. Is both a jewelry store and zoo, saber toothed tigers with diamond and gold adornments, turtles with glistening emerald shells, et cetera. And so again, this is all one continuous shot from Sora.

• 21:27Speaker 01

  We didn't stitch it together. And what's cool about this is even when you have these sort of scene transitions. Sora kind of automatically, you know, figures out like the vibe of what you're going for. So in this case, you get this coherence of like, you know, the environment you're in, you see this like outdoor. Component at like the start of the scene and like it gradually like moves indoors, but it all creates this kind of coherent narrative. Which is awesome that you don't have to like, you know, manually stitch together everything. It can kind of just like figure it out in context. Of course, you can also do long range coherence and like the notion of a character consistency as we alluded to earlier. So this is the story of a robot's life in a cyberpunk setting. And you can see you get the same robot character across these different shots.

• 22:09Speaker 01

  So it really does understand this idea that, you know, if I have a long video with multiple cuts, I'm probably going to have some amount of like characters that show up multiple times. It's not going to be an entirely new cast, you know, like every two seconds. And you just figure this out automatically. Uh, object permanence is another big one. So, um, you know, in the past, video generation models have really struggled to keep objects in the scene under occlusions. And so, you know, this is an example sample where, uh, even though this Dalmatian is getting included multiple times in the scene, Sora understands that, uh, that dog should still be there even when the people pass. And this very simple capability, um,

• 22:51Speaker 01

  that we take for granted was used to be a very challenging problem for video generation systems. But again, you don't necessarily need any kind of inductive biases specific to objects for this to emerge. You really just need, uh, kind of like the right fundamental training recipes and to scale these models up. And so, uh, one other capability we're excited about is this idea of interacting with the world and updating state. So kind of by definition, like if you want a useful video generation system, uh, at some points, it needs to be able to like interact with objects in your scene and have those like interactions be meaningful. And by meaningful, I mean like they need to persist over time. So, you know, in the simplest case, if I'm like drawing, uh, or painting, in this case, um, some soccer pedals,

• 23:34Speaker 01

  I would expect that, you know, as I'm leaving brushstrokes, like they actually interact with the canvas and stick around. So we find that sometimes Sora can do this. This is probably one of like the flakiest capabilities of the model currently. Um, but in this case, uh, it does work. This is another example of a, an older man eating a burger and. At the end here, there's bite marks in the burger. So I think this is one of like the larger challenges for video generation systems. Moving forward is, you know, this idea that if I do something in the distant past, can the model really remember that and recall that and have it affect things in the future? So these are like kind of very simple examples of that, but there's still a long way to go in creating like, I think, like really compelling examples, you know, where

• 24:19Speaker 01

  like a past conversation or something influences like what the system outputs, like me, multiple minutes in the future, for example.

• 24:27Speaker 01

  And the last topic I want to cover here is this notion of digital world simulation. So, you know, when people talk about video generation models, of course, there's like a lot of excitement about this idea that, you know, we can learn the real world's physics, and I think that's extremely valuable and a very important direction. But what's cool is, you know, these systems are very general, so there's no need to like constrain ourselves to only learning about our world's physics. There's all kinds of other crazy worlds out there, like, you know, laptop operating systems, or like video game consoles, that, you know, Sora like models could also learn from. And, you know, you can have one model, which eventually is like extremely generalist and is able to render out scenes in all of these different environments.

• 25:10Speaker 01

  And so one step towards this is Minecraft. So the prompts here is Minecraft, the most gorgeous high-res AK texture pack ever. And this is just a straight output from Sora. It's not even particularly cherry picked, actually. It's pretty easy to get good samples here. And you can see that, you know, Sora is able to like implicitly control the player here with like a an intelligible, if slightly boring, policy, while rendering out the full environment, rendering out NPCs like these pigs. And we think this is like a really cool, like extremely crude, you know, proof of concept of how you know, Sora can do more than like, you know, just be used for creative purposes, right? It can really model whole environments. And in the future, be used to,

• 25:54Speaker 01

  you know, potentially extract information about like policies for,

• 26:00Speaker 01

  you know, implicitly this all lives somewhere in source like activations and weights. It's cool to see that it kind of automatically learns these things again, just by training on video data at scale. So here's one more sample. With this prompt, so it chose a different texture pack for this one. Uh, but again, you see the same kind of things. You know, you got like a chicken and a pig. Uh, it's able to control the policies for those in addition to the character. As the character jumps around, it's able to render out this environment and pretty high fidelity.

• 26:29Speaker 01

  And so we're, we're pretty excited to see, you know, all of the kinds of knowledge that you can pack into this one model, not necessarily only real world physics. Uh, of course, Sora has a lot of issues, uh, so it is kind of far from this ultimate goal of simulating everything. Um, and they're kind of fun failure cases though. So, uh, everything about this scene is sort of messed up. So the woman looks way too happy. The hands in the background are kind of cursed. The candles are blowing in, in all the wrong directions. Um, this is another one, uh, where a cup kind of spontaneously leaps in the air and cracks in a really unrealistic way. So even pretty basic interactions, uh, like glass shattering, Sora does not really understand yet, and there's a long way to go.

• 27:14Speaker 01

  Um, this is, I think, most people on the team's favorite failure case. Um, so the, the prompt is like archaeologists discovering a plastic chair. Um, but the plastic chair is a bit sentient and starts like flying and seems somewhat possessed.

• 27:30Speaker 01

  So it's always fun, you know, when you have these models where on your scaling curves, um, you know, they're, they're not pushed all the way yet. It's always fun to see, you know, kind of like the correlations they don't yet understand about our world and how they take somewhat creative liberties. Uh, and yeah, this one's pretty self-explanatory for, uh, for what's wrong.

• 27:54Speaker 01

  So, um, Sora is currently in a research phase and we do not have it in a product yet. Uh, we work with red teamers and artists to really get a handle on, you know, what are the potential risks of a model like Sora? Uh, should it be deployed one day? And also, how can we make it as useful as possible? Uh, for, you know, both existing kind of like artistic workflows, but also potentially entirely new ones as well. Um, and so, uh, one quote from Shy Kids, um, which is a group that we gave. Access to Sora 2, as great as Sora is at generating things that appear real, what excites us is its ability to make things that are totally surreal.

• 28:38Speaker 01

  And so we really love this idea that, you know, Sora is not necessarily replacing elements of the artistic workflow, but really enabling entirely new processes that have not been possible before. And so I'll play this Shy Kids video now.

• 28:57Speaker 01

  I'm not sure if you guys can hear the audio, but if not, you can find this video online, search for Shy Kids Sora.

• 29:04Speaker 02

  Well, they say everyone has something unique about them, something that sets them apart. Just in my case, you know, it's quite obvious what that thing is. I am literally filled with hot air. Yeah, living like this has its challenges. Windy days, for one, are particularly troublesome.

• 29:24Speaker 02

  There was a one time my girlfriend insisted I go to the Cactus store to get my Uncle Jerry a wedding present.

• 29:30Speaker 02

  What do I love most about my predicament? The perspective it gives me, you know, I get to see the world differently. I float above the mundane and the ordinary. I see things a different way from everyone else. Yeah, and I feel like it's because of that perspective, I'm reminded every day that life is fragile. We're all just a pinprick away from deflation. So I try to live life with lightness, a buoyancy, joie de vivre. I got a lot of ideas, keep them in the safe. With any luck, I'll find a way to share them with everyone else.

• 30:11Speaker 01

  And so this video was made with a combination of just using direct model output. And of course, also more like traditional video editing workflows on top. So it's been really cool to see how artists have embraced Sora and have begun to incorporate it. There's also been kind of some films that were at Tribeca, actually, which were made in a similar way with Sora. And I've been really surprised by like the level of creativity with the kind of the current levels of capabilities that Sora has today. It's really cool to see the community like lean in and use these models. So that said, that's pretty much the end of the talk. I have a few extra samples here.

• 30:51Speaker 01

  But yeah, thanks again to Joanna for scheduling this and happy to field any questions. I don't know if it's possible to communicate them over Zoom currently or not, but yeah, if not,

• 31:07Speaker 01

  that's about it. So thanks a lot.

• 31:10Speaker 03

  Thank you, Will, for this great talk. I think, yes, we can definitely, well, if you can hear me, then we can do a Q and A.

• 31:18Speaker 01

  I can hear you, so I think we're good.

• 31:22Speaker 04

  Hello. Thank you for the great talk. I wonder how far we are from, let's say, a video producer making a whole movie with zero actors. So maybe if like hypothetically, the video producer can upload characters, how they look like, and they can describe the scenes and tell you, oh, this character is now running away or on a bike, etc. With zero actors, can they actually do a full movie?

• 31:44Speaker 01

  Yeah, good question. So I think there's like a technical answer and like a cultural answer. So on the technical side, I don't think that, you know, there's necessarily any blocker to really, you know, making like character consistency work over very long time horizons. That seems like just a very achievable problem in general. And so I think in the near term, it will be possible for people, if they want to do that, to be able to, you know, create these kind of like synthetic characters and kind of use them as they desire. Now, I don't know if people will really want to do that in the near term necessarily. Yeah, we've been chatting with like a lot of directors, for example,

• 32:28Speaker 01

  and a lot of them mentioned how like, you know, for kind of like very simple scenes, how it's really convenient to use Sora's current capabilities. You know, for example, like have like a large crowd in the background, whereas maybe in the past that would be CGI driven. But you know, for like these really kind of like intricate and like meaningful like close up shots. And when you're trying to develop more of like an emotional connection with the audience, at least for like the near future, it seems like, you know, human actors definitely have like an edge on on models like sort of today. So I suspect in the future, there will be some kind of like mix and collaboration between the two. But yeah, it'll be interesting to see kind of like when and where people choose to use totally like,

• 33:12Speaker 01

  you know, digital characters versus traditional actors.

• 33:16Speaker 05

  Hi, I have two questions. The first one would be how big of a role played synthetic data in the training process?

• 33:25Speaker 01

  So I can't answer any questions about training data, unfortunately. So yeah, sorry.

• 33:33Speaker 05

  Okay, then the second one would be about how much control does the user actually have over the camera angle and the trajectories? Is it just prompting or can you actually define a full trajectory or what's possible?

• 33:45Speaker 01

  Good question. So currently, the only way that you can define notions like camera motion is either through text or through video conditioning, right? And so in the latter case, that would mean like if you see generation with like a video where the camera is like already moving in the way you want, and then you want to extend it out from there, you could kind of infer that via in-context learning to potentially get the right camera motion. Currently, there's not a more granular way to do camera control. I think this is something that, you know, we've definitely heard people want. And so it'll be interesting to explore alternative ways to more explicitly control those kinds of features, but right now it would go through text primarily.

• 34:27Speaker 05

  Hey, so we've seen all this nice visual output. Can you share something about audio and consistency, maybe what you observed there, or if you have any?

• 34:37Speaker 01

  Yeah, this is a good question. So for Sora, we were really focused on trying to push the envelope with visual generation quality. And we weren't very focused on, you know, for example, jointly generating audio. Um, I think it's a really interesting direction to get, you know, extremely high fidelity like joint video audio generation. But um, it's not something that we have with Sora currently. Um, I think, uh, you know, in the future, um, making these models more controllable and, you know, potentially like giving users all of like the modalities they want is certainly an interesting direction.

• 35:11Speaker 06

  I, Sora can also generate images. Do you think in the future, uh, video generation model will be stronger than text to current text match models? And we will stop basically training just on images?

• 35:26Speaker 01

  Yeah, I, I think so. Um, and part of the reason for that is, you know, there's a lot of information about the world, which, you know, if you're training on like huge data sets of images, you can probably infer to some extent, but I think, you know, there are still some things that like slip through the cracks that you only get by training on video data. So, you know, for example, um, the fact that like a model can, uh, really generate like an accurate fly through of a scene and like really understand occlusions. I, my guess is like, that actually helps like image generation capabilities and like understanding how, you know, some fingers on a hand may be occluded by an object, but that doesn't mean that like humans often only have like two fingers.

• 36:07Speaker 01

  It just means that, you know, there's like a physical interaction here that you didn't necessarily rock by only training on image data, or you didn't rock it efficiently, uh, and you get these concepts much more, uh, either data or compute efficiently from co-training on video. Um, so yeah, I, I suspect in the future, video generation models will like generally supersede image generation.

• 36:27Speaker 06

  And is already the case for Sora or not yet? It is.

• 36:32Speaker 01

  Um, so currently we haven't productized any of Sora's capabilities, including image generation. So today, you know, if you go, uh, to chat GPT or something, it's using Dolly three, uh, under the hood to do text image.

• 36:45Speaker 06

  Thank you.

• 36:46Speaker 07

  I, um, I was wondering if you can tell us a little bit about the size of the model on, for example, what is the maximum length in time that it can generate or resolution or amount of pixels, something similar?

• 36:59Speaker 01

  Uh, good question. Unfortunately, can't comment on that. Sorry.

• 37:05Speaker 07

  And even something closer to the order of magnitude. So for example, can we expect a user having a good model in the future? Or it will be something that only big clusters, some big companies can afford?

• 37:20Speaker 01

  Yeah, that's a good question. I think I wouldn't be particularly surprised, you know, of like the evolution of video generation models ends up looking a lot like the evolution of language models. So, you know, there will be a variety of models of different like capability levels and sizes, pretty similar to the ecosystem we have now. You know, there's like open source models, which at least historically have tended to be somewhat weaker compared to like these like larger scale, closed source models. But I'm curious to see how the, you know, how the whole ecosystem evolves, just a guess.

• 37:55Speaker 07

  All right. Thank you.

• 37:57Speaker 08

  Hi, thanks for the talk. I'm curious about how you are thinking about more sophisticated controls like bullet time or in zone.

• 38:11Speaker 01

  Yeah, good question.

• 38:15Speaker 01

  I think one thing we've heard from chatting with like directors and artists is, you know, they have like a very particular language that they use to describe certain kinds of like shots and like camera motion. And Sora out of the box is not so good at speaking that language. So a lot of kind of what we think about in terms of like improving users' interactions with this model is, you know, kind of like thinking about, you know, can we train the model to like use that same language? So it to some extent is kind of like a captioning problem. But I think the jury's still out there on, you know, the best way to like make these models controllable and is like, is that only through text or are there other kinds of inputs?

• 39:00Speaker 01

  I think it's a very interesting space currently and we're still just kind of at the start of exploring it.

• 39:07Speaker 08

  Thank you. One more short question is there was interaction between character and hamburger kind of thing. Is there any way to make more interaction from the user like like hitting the hamburger so that hamburger will be squished so that

• 39:27Speaker 08

  yeah, I think you get it.

• 39:28Speaker 01

  Yeah, yeah, that'd be cool. I don't know if Sora today can do that kind of more complicated interaction. I think there's no fundamental reason why that shouldn't be possible and why further scaling up these models shouldn't be able to achieve that kind of capability. It's just, you know, even this kind of like biting a hamburger thing. This sort of phenomenon was something that took a while within like, you know, the research process of Sora to like emerge and seem to require like at least a decent bit of compute for it to be like a non-trivial interaction. So I'm curious like, you know, what level of scale you might need to

• 40:10Speaker 01

  fully smash a hamburger and have it be like physically accurate. I think, I think, you know, we'll definitely get there eventually. It's just you never know where on the scaling curve you have to be for these capabilities to start popping up.

• 40:21Speaker 08

  Thank you.

• 40:23Speaker 09

  Hey, great presentation. I was wondering, you've shown this nice Minecraft example where the actually had some kind of agentic behavior a little bit that was shown. Do you think Sora could actually help enabling as a world simulator, for instance, as a foundation model, agents that interact with the real world? Make it better than what's there right now as like robotics, for instance?

• 40:46Speaker 01

  Yeah, definitely. You know, I don't know if like the current model is robust enough to be reliably useful to improve like real world policies, but I think it's inevitable that one day these models will power these kinds of systems. There's just so much information about the world that you learn by training on large scale video data that it seems inevitable that that knowledge should transfer to the real world at some point.

• 41:12Speaker 10

  Hi, I have question about inductive bias. Have you ever tried to train the model with some specific inductive bias such as physics or any rules in the video?

• 41:24Speaker 01

  No, we haven't. So from the onset of the Sora project, we were really focused on kind of training just like pure visual generative models of data with as few inductive biases as possible and really just ensuring that the foundation was solid for scaling. That was kind of like the core thesis of the project. And so we haven't explored incorporating inductive biases. I suspect, you know, for like certain kinds of like narrower use cases, it's possible that you can get some kind of win from doing that. And, you know, potentially if your model needs to be really small, but you don't need it to be extremely generalist, then potentially that could be a win.

• 42:06Speaker 01

  But we're really just trying to like scale up like the largest, most generalist model possible. And so to that end, we generally assume that they'll be harmful at some points, which is why we haven't explored them too much.

• 42:17Speaker 03

  Thank you, Bill. Let's thank the speaker again. Thank you so much for the fantastic talk.

• 42:24Speaker 00

  In their original blog post, OpenAI describes Sora as a world simulator, and we have been tracking the start of the summer of simulative AI. Google DeepMind is not resting, of course, having announced their VO model at Google Y2O this year with Donald Glover's endorsement. However, the focus at ICML is on GENE, short for Generative Interactive Environments, which is an 11 billion parameter foundation world model trained on unlabeled internet videos to generate action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model,

• 43:08Speaker 00

  and a simple and scalable latent action model. GENE enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature.

• 43:27Speaker 00

  More recently, DeepMind announced SEMA, their Scalable Instructible Multiworld Agent, and Genie 2, which extends Genie 1 from generating 2D worlds going into 3D worlds. Genie 2 is a world model, meaning it can simulate virtual worlds, including the consequences of taking any action, for example, jump, swim, etc. It was trained on a large-scale video dataset and, like other generative models, demonstrates various emergent capabilities at scale, such as object interactions, water effects, directional lighting, reflections, complex character animation, physics, and the ability to model

• 44:10Speaker 00

  and thus predict the behaviour of other agents. In particular, Genie 2 has long-horizon memory, meaning it is capable of remembering parts of the world that are no longer in view and then rendering them accurately when they become observable again, ensuring it generates new plausible content on the fly and maintains a consistent world for up to a minute. Finally, Genie's learned latent action space facilitates training agents to imitate behaviours from unseen videos, opening the path for training generalist agents of the future, which we will explore in the last part of this pod. But first, here is Google DeepMind's oral presentation of Genie.

• 44:52Speaker 11

  Hello, everyone. Good morning. Thank you for coming. I'm Jack and alongside Ashley, I'm super excited to be presenting our paper, Generative Interactive Environments, otherwise known as Genie.

• 45:05Speaker 11

  Genie was an amazing collaborative effort from this wonderful group of people at Google DeepMind.

• 45:12Speaker 11

  Our long-term goal is to train embodied agents that can safely perform complex tasks with long-horizon consequences in the real world. It's safe to say there's been amazing progress in our field in the past few years, but this still feels pretty far away. So what's missing?

• 45:29Speaker 11

  Fortunately, there's another pretty cool paper ICML, which has a nice way of thinking about this. In particular, they factorize agent capabilities by breadth and performance. And if you see in the bottom right cell, that's kind of what we want: general superhuman intelligence. Now, the good news is we've made some good progress on this grid, so we already have what they call emerging AGI, thanks to progress in foundation models. We also have superhuman agents in narrow domains. In the case of AlphaGo and AlphaZero, these agents have already been used to augment human intelligence. Note, however, this was made possible by access to the simulator for the game of Go, which we don't have for general intelligence. So the main claim that we make in this work

• 46:13Speaker 11

  is to get to the bottom right corner, the key missing ingredient is a more general environment.

• 46:22Speaker 11

  So, the main motivation for us is how can we possibly get to this more general environment?

• 46:29Speaker 11

  In parallel, it's pretty clear that video generation's having a bit of a moment. We've got a pretty packed out room for this set of auras. And this area of research is now front and center of progress in AI. What's pretty incredible, though, is that by making use of large video data sets, these models are increasingly understanding the physical world in ways they couldn't before. And so as a result, many people are starting to believe that these video models can actually be accurate world simulators.

• 46:57Speaker 11

  However, we believe that while these models may have world knowledge, they're not world models. Indeed, many of the text-to-video models that we currently see are only controllable at a very high level via text captions. Prompt the model and you receive a video clip. It may be consistent and beautiful, but it's not, therefore, a world model, because you can't take sequential actions in the environment to learn new behaviors. The challenge here is that we don't have data with action labels, and so the largest world model settings that we have are limited by action-label data sets, meaning we can only model existing environments, therefore not being able to generate new ones.

• 47:35Speaker 11

  It doesn't seem to make sense to use tons of compute to just recreate existing games that we already have.

• 47:41Speaker 11

  So this is the problem we're working on solving in this project. We want to make use of the vast amount of unlabeled videos on the internet, and we want to train an action-controllable video model, otherwise known as a world model.

• 47:52Speaker 11

  So to summarize, the goal of Genie is to learn what we call a generative interactive environment, purely from videos that's playable by both humans and AI agents. How do we do this? Well, the main idea is to use a latent action space, learned in a fully unsupervised manner. Intuitively, these correspond to clustering potential outcomes from a given frame of video. I'm now going to have Ashley to talk about how we do this and show some of our results.

• 48:18Speaker 12

  Thanks, Jack.

• 48:21Speaker 12

  So the Genie model consists of three main components, and we're typically training over a sequence of around 16 frames. So the first component is a video tokenizer, which takes in patches from that entire sequence and discretizes them into video tokens. The next is a latent action model, which is going to take in consecutive frames and discretize and compress them into what we call latent actions. And the main purpose of our latent action model is to try to encode what's going to change between our scenes, such that, given those latent actions along with our prior frames, we can use them for predicting our next frames. And this is what's going to be important for controllability. And the final component of our model is going to be our dynamics model,

• 49:05Speaker 12

  which takes in our tokenized frames along with our latent actions and predicts next frame tokens.

• 49:13Speaker 12

  So in order to actually interact with our model during inference time, we can take an initial prompt frame, take some actions, then generate the next frame, plug that back into the model, and continue this. And importantly, because we're learning discrete latent actions, we can actually just plug in integer values to interact in this way. And we found that it was really important to, in order to actually evaluate our model, to actually step into it and basically play it ourselves. Because it's one thing to measure the sort of quantitative performance of our model using approaches like FPD, but it's another thing to actually model controllability and be able to actually evaluate that. So actually interacting with it ourselves was very important.

• 49:56Speaker 12

  So Genie was trained on a dataset that consisted of around 300,000 hours of video game footage, consisting mainly of 2D platformer games. But we did find it was important to filter this down to 30,000 hours to get a more high-quality dataset. So before training our main model on this, we ran a few scaling analysis experiments that showed us that it was important to scale both model size and bat size. And once we did this, we found that we came up with our final 11 billion parameter, Genie model with a bat size of 512.

• 50:35Speaker 12

  So now let's take a look at some of our results. And so I want to point out that all of the results are going to be showing out-of-distribution examples. So some of them might be example coming from text-generated images.

• 50:50Speaker 12

  So this first video here is going to show some of the environments that Genie is capable of creating. And so I think the exciting thing here is it's showing that we can essentially step into our environments. This is showing real human interaction within these generated environments and take actions and change the world that we're experiencing. And it's important to point out that this is one of the main differences between what you see in Genie versus video generation models. It's that interaction that we're able to actually take actions within our environments. And this is why we can consider Genie to be a foundational world model, because we can take these actions within our generations. So this is showing another example.

• 51:32Speaker 12

  So given the same initial prompt frame, what we can do is we can take a different series of latent actions and plug them into our model. And what you see is that we're going to get very different and diverse trajectories. Again, this is showing human interaction with the model. And again, this is because we've learned this latent action space in an unsupervised manner.

• 51:53Speaker 12

  The other important thing is going to be consistency. It's one thing to be able to generate a diverse set of trajectories, but if you're having to figure out what your latent actions mean every time you have a new image, it's not really that useful. So we also wanted to measure how consistent our latent actions were. So given four different initial prompt images, we could plug in the same sequence of latent actions. And what you can see is that there are very similar trajectories and behaviors essentially happening across these different environments, which is telling us that indeed our latent action space, at least in these environments, is consistent. And again, just to point out, we were able to learn these latent actions without using any ground truth action labels or doing object detection or object segmentation or any sort of domain-specific information.

• 52:38Speaker 12

  So one sort of exciting and fun thing we found within the project was that we could actually plug in sketches, even though we were only training on 2D platformer games. So for example, on the left here, we see a sketch done by Richie from a team. In the middle, that's one of Jeff Cloone's children made that. And on the right, I did that one, but don't judge me too harshly. And we can basically plug these images into our model and again, create these environments. So we can see, for example, we're able to climb this ladder that Richie basically sketched out. And so I think it was in this moment when we really started to see the kind of creativity that Genie could enable. And so we also plugged in real-world images, which is again, very out-of-distribution from what the model was trained on. But for example, that's Jack's dog Doris on the left here.

• 53:23Speaker 12

  And we can again, sort of, generate these environments and I guess, interact with them, even though we didn't train on anything that looked like this.

• 53:34Speaker 12

  Genie also works on real-world data. So we trained a smaller model with 2 billion parameters, excuse me, on a robotics data set. And we again, see that if we take different prompt images, but plug in the same sequence of latent actions, we're getting similar behaviors, which is again, evidence that the latent actions are consistent. We're also able to simulate deformable objects with this model here. And finally, while we haven't yet shown that we can train agents within the genie model, we do show in the paper that we can take the latent actions learned from videos on the internet and use those for labeling unseen videos, which allows agents to actually imitate from me.

• 54:15Speaker 12

  So this indicates that genie can be used for training our generalist agents of the future. And speaking of the future, I'll pass it back to Jack to talk about future directions.

• 54:26Speaker 11

  Awesome. Thank you so much, Ashley. Right, back to me. We want to emphasize that what we've shown here is that this is even possible. Before we started this project, the idea of training an action controllable world model from videos seemed a bit like a pipe dream. And so as a result, this is the worst the genie is ever going to be. We're expecting to see rapid progress from here, which we think can have a huge impact in a variety of areas.

• 54:49Speaker 11

  So going back to our original motivation, we think that genie presents a clear path to generating unlimited environments for training agents. And so for a more formal write-up of how we see this could fit into a framework towards getting to more general intelligence, come check out our position paper on Thursday in the oral session. Not only that, but as Ashley mentioned, we noticed something pretty magical happening while playing our model, is that it enabled a new form of creativity as people such as Jeff Koons' children, as previously mentioned, were able to draw their own worlds and step in and play. And we think this is barely scratching the surface by what could be possible with this new form of generative AI.

• 55:26Speaker 11

  Okay, so to address the elephant in the room, so for those in academic institutions thinking it's just another industry paper that used tons of compute that you can't possibly work on, fear not, we've got something for you as well. So in the paper, we have a case study where we show you can train your own much smaller genie model on a mid-range TPU in just under a week. With this approach, you should be able to see some pretty consistent latent actions and given different initial prompts in the coin run environment. As an example here, you see different actions from this model that we did train in a few days. And we're excited to see that this isn't just a wild goose chase. We have actually got some students that have been able to reproduce this. So come along to the controllable video generation workshop on Saturday to see their poster.

• 56:08Speaker 11

  And finally, if 12 minutes wasn't enough for you, fear not, we've got a few other things going on this week. So we've got a couple of position papers, we've got the poster straight after this talk. And then we've also got some longer talks in workshops later in the week. And then many others in the team are here as well who would love to chat to you all. So yeah, that's a wrap. Thank you for your time.

• 56:27Speaker 13

  Thanks for showing this amazing work. So I have a question about technical details. So in the, in this training phase, you have two training phases, right? So in the first phase, you're training a inverse model for the latent actions. And then in the second phase, you're training a prediction model for this video generation, right? But in the first phase, when you're training the latent action model, you already has a latent model trained. So why is necessary to train the second step? I'm just wondering.

• 56:58Speaker 11

  Oh yeah, that's a great question. And so essentially what you're saying is why do we have a decoder for the latent action model that already predicts the next frame? And then subsequently train another one? And we found that there's actually slightly different trade-offs for this decoder. So we found, if you see in the paper, we're predicting the pixel space rather than token space for the latent action model. And we found that that really helps to get more controllable, consistent latent actions. And then that decoder itself is just predicting in the pixel space. So actually it would have, it would be pretty blurry if you were to use it as a generative model. Whereas we found that the masket objective wasn't best for learning latent actions. It just didn't lead to as consistent latent actions.

• 57:38Speaker 11

  So we have this like dual approach where we have different, two different dynamics models essentially we learn as part of the process. But you're totally right that this isn't the most elegant solution. And many in the team weren't overly ecstatic about it. But that's why we're saying this is the worst it's ever going to be. And hopefully some of you folks in the community can build a much more elegant solution in the next few months.

• 57:57Speaker 14

  Thank you very much. Really cool work that you guys are doing. I wanted to ask regarding the qualities that we can see on these world models. So on the videos, we essentially saw some amount of like physics, so jumping and then falling down because of gravity. And then we saw platforms and saw some ladders. Do the world models ever generate other entities? Think of it like maybe enemies going back and forth, that if you touch them, something happens? Or like, what other quilia do you think that have you guys observed?

• 58:26Speaker 12

  Yeah, that's a great question. So I would say in the sort of examples that we show, particularly out of distribution examples, it is very difficult for it to generate anything that's kind of exciting. We are able to move the character around, but typically you would just see it, I guess, repeating the patterns that it's seen in the background and that sort of thing. I think that's another sort of exciting direction for the future is trying to figure out how to make it a little bit more, the generation's a little bit more exciting and diverse.

• 58:55Speaker 14

  Yeah, thank you very much.

• 58:56Speaker 15

  Now, as well, one last question. I actually wanted to ask you something. What do you think is sort of like the cool killer application that you see in the future? If you could really scale this up and train this on anything?

• 59:10Speaker 11

  So I think there's quite a few applications, and really it's subjective, depending on your interests. So I think if you, I personally think this could have impact in quite a few areas. So you can imagine some of the domains we use, you're already quite fun to interact with it as someone in those settings, but I think it could have quite a large impact in areas such as robotics, because it's currently quite hard for robotics, for robots to generalize to completely unseen scenarios, but if you could generate a world model for any possible domain. And actually, we've seen, there's an open source genie model from 1X Robotics that works pretty well. And so I think that they obviously think so too, and they probably know more about that than me.

• 59:53Speaker 11

  And yeah, so I think there's a lot of potential applications, but we're just not focusing on one right now.

• 59:59Speaker 15

  All right. Thank you very much, and congratulations on the best paper.

• 1:00:05Speaker 00

  Because the Genie team were accepting their Best Paper Award in Vienna, we were able to catch them at their poster session live to tell a bit of the human story behind Genie. Over to you, Brittany.

• 1:00:17Speaker 16

  I am here with the Genie team. Generative Interactive Environments is the title of the poster. And I'm here with Jack. Jack, can you tell us a little bit about the origin story of the Genie project?

• 1:00:30Speaker 11

  Sure thing. Yeah, firstly, thanks for the chance to speak to you. So basically, Genie is kind of a fusion of a few different areas of research. Myself and some others who are working on open-ended learning and environment generation beforehand. And we were interested in world models and thinking about how we could scale them to internet videos. But obviously, the key challenge with that is that internet videos don't have action labels. So if you want to train a model that takes actions as input to predict the future, you don't have the action, so you can't train that way. And then, on the other end of the spectrum, Ashley Edwards had been working for many years on inferring actions from videos for a different purpose, for directly training agents with behaviour cloning. And so it seemed like a natural fit, really, to combine these ideas.

• 1:01:15Speaker 11

  And there were some pretty simple proof of concepts of people doing this at like very small scale. But no one had really gone to the generative angle of getting an environment generator from a large-scale dataset. And so when we first spoke with Ashley, you know, a year or year and a half ago, we were excited about this potential combining these ideas to build something completely new. And yeah, I guess that's where we got to.

• 1:01:39Speaker 16

  Nice. And can you give a little bit of an overview, I guess, of how the work went? What results you saw? That type of thing?

• 1:01:45Speaker 11

  Sure, yeah. So we started basically working on this 2D platform as dataset. So we have 280,000 hours of publicly available videos of 2D platform games. We found one important thing was to filter this down, because a lot of the videos aren't very good quality. So we trained a classifier with hand labels that we labelled as a team, a small subset. And then we end up with 30,000 hours of good quality videos. And then from that point, it was just a modelling problem. And so we did a lot of research on different approaches to get these latent actions. And what we ended up with is, we train it in kind of a, I guess, slightly quirky way, is that we predict pixels with a latent action decoder.

• 1:02:27Speaker 11

  And then that allows us to learn a discrete set of eight latent actions. And then we separately train a dynamics model that is using mask it, which is like a way of generating next frames.

• 1:02:40Speaker 11

  And we train that separately, given the latent actions that are produced from the video. Just predict the next frame conditioned on the actions. And then people working on different things, like the project was quite fast paced and a few of us kind of switched and wore many hats. And we started all getting different results in different areas. And then roughly around last summer, so probably just under a year ago, we realized that when we combined a few of these ideas, we actually had something that worked pretty well. And then we were really excited, obviously. So then we started working on seeing how the model scales. Because the key thing about this project is that if you can figure out how to generate worlds without action labels, then essentially there's nothing stopping you from using all of the world's videos because there's no reason why you need to wait to do that. So we started saying, okay, how can we scale this approach and what does it do?

• 1:03:25Speaker 11

  And then we produce these plots, which you'll see in the paper, if you have time to look at that, where we show that as you increase the model size from a few tens of millions to, in that plot, something like 2 billion, you just get an increased performance every single time you increase the scale. And then the same thing with batch size, when you increase the number of examples the model sees. And so we realized that we had produced a scalable model, so we then decided to go for what in the end was an 11 billion parameter model. And then, once we produced that, we just started all playing with it and seeing what we could do with it.

• 1:03:56Speaker 11

  And then, finally, obviously, the goal of this was originally to get an environment for agents. That's how we kind of started, Ashley on the behavior cloning side and myself and others on the like more auto curriculum and open ended learning side. But we realized it was really fun to play with the model. And so actually, maybe the more interesting use case is how it enables new forms of creativity. And so there's some examples in the paper of things like drawings from one of our co-authors' children. And they sent us photo of the drawings and then we were able to then prompt the model with those photos and play and move the characters and the photos around. And that's pretty cool, right? Because you're enabling people to create their own world, step into them and interact with them. Which was not really what we first thought of when we started the project, but

• 1:04:37Speaker 11

  I think it tells you that if you do kind of ambitious, somewhat crazy stuff, then maybe new things will emerge.

• 1:04:44Speaker 11

  So that was pretty fun. There's also a picture of my dog there too, so that's another example I like.

• 1:04:48Speaker 16

  Nice. Do you see it? What would you see in terms of, I guess, more near term potential applications for this? A lot of the folks who listen to the podcast are kind of on the AI engineer builder side of things. Any creative ideas there?

• 1:05:01Speaker 11

  Honestly, like, this is gonna sound a bit like, um, a bit of a non-answer, but there's so many applications, like, um, so you can obviously see the ones in the paper, we have examples where we show generating, um, 2D platformer-like, kind of, short game experiences. But you also can see the model works in robotics data too. And arguably, that latter use case, um, is maybe more, uh, promising in the short term. Uh, there's actually already been an open source, um, Genie model, um, released on 1X, um, GitHub repo in Paws Day World Model Challenge. And so, I think they're more expert in robotics than I am, but the fact that they think it's a potential, uh, good direction for robotics probably speaks volumes.

• 1:05:46Speaker 11

  Um, I think there's other use cases too, so things like maybe driving, uh, if you could generate scenarios for, for like, testing or even training, uh, autonomous vehicles, and then be able to interact in the world, in any custom situation, that could be very valuable. Um, but on our side, we're mostly just pursuing the, the, like, fundamental research, um, and not really focusing on one specific application.

• 1:06:09Speaker 16

  Yeah, and I imagine the world has, has already moved to, um, continue to move forward from a research perspective, since you guys put this out there. Is this a direction that you see yourself continuing to pursue? Or what have you been excited about lately?

• 1:06:24Speaker 11

  Um, yeah, so this work, I mean, ICML, the deadline is January, right? So it's already six months or so ago that we submitted this. Um, and yeah, I guess, most of the team are still working on it, so.

• 1:06:36Speaker 16

  Do you see coming out with a Genie V2 or a V3?

• 1:06:39Speaker 11

  Um, yeah, uh, I don't, I can't speak exactly about specific releases, but, um, hopefully, we'll have something new at some point.

• 1:06:50Speaker 16

  Has there been anything else that's come out in the research landscape that you feel has either reinforced, kind of, what you've worked on? Or, or contradicted on the flip side? How do you see it evolving?

• 1:06:58Speaker 11

  Yeah, it's a great question. So definitely reinforced. I think, just after Genie came out, there's been a flurry of, like, really amazing video generation results. So, um, the first one was clearly Sora. Uh, I think it, they definitely took the space by the scupper of the neck and really pushed capabilities quite significantly. Uh, and that was really exciting to see. It's quite a different, um, style of model in that theirs is, um, text-to-video, so it generates entire clips, um, whereas Genie is like, frame-by-frame level control. But nonetheless, like it does show you that with additional scale and like I guess brilliant execution, you can get much more high-quality videos generated already than I probably thought was possible. And then since then, I think there's been kind of the floodgates opened

• 1:07:41Speaker 11

  in this space. So from our own colleagues, the VO model came out and was announced at I O was really impressive as well. And then competitors, I guess, have also, other competitors have done similar things. So it's a really exciting time for that space. I think there's not really been anything that's action-controllable like Genie. But yeah, it's definitely an exciting time for video generation. So I think it's a good space to be getting involved in now. Yeah, given how fast the community is moving, I think in a few years' time, we'll have something pretty incredible.

• 1:08:14Speaker 16

  I just spoke with your, um, your video poet colleagues as well. How do you see this work like dovetailing with that work? Or how do you kind of work together, I guess, on the future of what video looks like or world models?

• 1:08:26Speaker 11

  Yeah, so I think that they're, they're maybe more interested in, in more like cinematic video experiences and like generating entire clips. Whereas Genie still remains like quite, quite fundamentally different because it's only generating one frame at a time. And it's like, it's a, it's kind of a video model, but it's also kind of like a auto-aggressive image generation model in a sense. So it kind of sits in its own, like it's kind of a new area. I guess a lot of researchers always claim that they're inventing a new area, but it is kind of a new area of research and hard to classify. It's a bit different. There's also, a lot of us come from an RL background, so we're much more thinking about agents, which I think is quite different to a lot of the video work, which is much

• 1:09:07Speaker 11

  more, I guess, focused on generative media and generating cinematic quality videos. But there's definitely some overlaps in the architectures and these kind of things, and the infrastructure and... We both want to use lots of compute, so I guess that's another thing we have in common.

• 1:09:21Speaker 16

  What do you make of all of the hype around the agent space? Do you see that continuing? Or do you see people getting tired of the agentic buzz?

• 1:09:29Speaker 11

  You're venturing into hot takes territory. So it's tricky because, I mean, a lot of us who've worked in RL, right? We've been working on agents for a long time. And because I think RL is often dubbed as reinforcement learning research, but really it's agent research. For a lot of us, it's agent research where RL is currently the best method to get agents. It seems like now that's shifted because people are starting with LLMs and then training them on top to get additional capabilities. But it's a lot of the same people that were doing RL research. So they've always been working on agents. It's just now it's called LLM agents. Before it was tabular Rasa RL. So I think, yeah, this hasn't changed a huge deal. It's just we're starting with base models rather than tabular Rasa

• 1:10:14Speaker 11

  in maybe some kind of toyish environments. It's still have a natural progression of that line of work. I think it's exciting, but I think the goal of genie is a bit different in that we're going more for an embodied AGI. And we want agents that can interact in the real world over a long horizon. And for that, I just can't look past how you would need a simulator of the real world, which I don't think we're going to build by hand. So I think they're kind of complementary in a sense. I think the LLM agents will become more capable in doing long horizon tasks in like tech space substrates. But I think that to then in the real world, take long horizon actions for some kind of VLM, it's going to need to be able to interact in the world. And we're not going to just be releasing them to do random exploration.

• 1:10:58Speaker 11

  So I think a real world simulator will play into that at some point.

• 1:11:02Speaker 16

  Awesome. Thank you so much for the time.

• 1:11:04Speaker 00

  Believe it or not, Google also won a second best paper award at ICML for video generation with Video Poet, DeepMind's take on zero-shot video generation. Video Poet is a simple modeling method that can convert any auto-regressive language model or large language model, LLLM, into a high-quality video generator.

• 1:11:27Speaker 00

  It contains a few simple components: A pre-trained MagVit V, two video tokenizer, and a soundstream audio tokenizer. Transform images, video, and audio clips with variable lengths into a sequence of discrete codes in a unified vocabulary. These codes are compatible with text-based language models, facilitating an integration with other modalities such as text. An auto-regressive language model learns across video, image, audio, and text modalities to auto-regressively predict the next video or audio token in the sequence. A mixture of multimodal generative learning objectives are introduced into the LLM training framework,

• 1:12:12Speaker 00

  including text-to-video, text-to-image, image-to-video, video frame continuation, video in-painting and out-painting, video stylization, and video-to-audio. Furthermore, such tasks can be composed together for additional zero-shot capabilities, for example, text-to-audio. Let's cut to Li Junyu, speaking for the Video Poet Oral Presentation.

• 1:12:37Speaker 17

  Good morning, everyone. This is Li Junyu from Google DeepMind. Excited to meet everyone at Vienna.

• 1:12:46Speaker 17

  This year, I believe many of you may have witnessed the significant progress on video generation, especially with text-to-video diffusion models. Today, I'm going to talk about a completely different approach, which shows that diffusion may not be a necessary component. We appreciate that the award recognizes the contributions of this work. Now, please allow me to introduce Video Poet, a large language model for zero-shot video generation. This work wouldn't be made possible without our talented team, with members coming from diverse backgrounds and moving forward along different paths. The core contributors were Dan, myself,

• 1:13:31Speaker 17

  Xue Ye, Jose, Jonathan, Brian, and Lu, along with many other video poets.

• 1:13:40Speaker 17

  Reflecting on the progress so far, we realize that video generation has already come a long way from the early days of GAN models. In case you have never seen generated videos from a large-scale model by the definition of 2016, here are two examples for classes of GORF and BIB. Since then, people have scaled up GAN models and developed pixel-space autoregressive and diffusion models, which were getting less affordable. Some works try to model it as a foreign language of images or videos, but lossy discrete tokenization poses inevitable limitations. Later, latent diffusion has become the dominating approach,

• 1:14:24Speaker 17

  given its appealing sample quality. Big companies and startups have ignited a rise of scaling up compute and data.

• 1:14:33Speaker 17

  Now, nearly 10 years later, models can easily generate a video clip from a text prompt, like this skeleton drinking soda. But is latent diffusion the only way to go as we embrace the LLM area?

• 1:14:49Speaker 17

  Absolutely not. In fact, this video is generated with VideoPoint, a purely LLM-based approach without diffusion.

• 1:15:00Speaker 17

  VideoPoint is a foundation model that takes inputs of text, image, visual dance signals, partial videos, audio combinations. It is capable of text-to-video, image-to-video, video stylization, video editing, video-to-audio, and many other tasks. In short, VideoPoint is an autoregressive LLM that synthesises videos with high-fidelity motion and matching audio from a large variety of condition signals. The diverse capabilities of VideoPoint are facilitated by defining a universal multimodal sequence-to-sequence problem. The condition sequence includes task indicators,

• 1:15:45Speaker 17

  inputs from text, visual and audio modalities, as well as output format controllers. The model generates the output sequence of visual and audio tokens in a fully autoregressive manner, just like a usual language model.

• 1:16:00Speaker 17

  In order to define the token space for each modality, we resort to a collection of uni-model tokenizers. MagVid V2 encoder and decoder define a bidirectional mapping between the pixel space and the compressed space of discrete visual tokens. It can tokenize image, depths, or optical flow, as well as cropped or masked videos. Soundstream does similarly for the audio waveform. Although text tokens can be directly fed in, we use a pre-train T5 to extract text features to reduce the burden of learning human language from scratch. The MagVid V2 tokenizer defends the visual language. It resembles a quantized VAE with a temporally causal 3D CNN architecture, processing pixels.

• 1:16:45Speaker 17

  This causal design enables joint training with large-scale image data and seamless support for long videos.

• 1:16:53Speaker 17

  For higher prediction bandwidth, we adopt a large vocabulary of over 200,000 words enabled by our scalable quantizer. The model is trained with both reconstructive and adversarial objectives. In a human reader study, our advanced video tokenizer achieves even better compression quality than VVC, the next-generation video codec standard. This tokenizer lays a solid foundation for high-fidelity generation of videos, especially for those with large motion.

• 1:17:23Speaker 17

  Similarly, the Soundstream tokenizer defends the audio language, which adopts a causal 1DCN waveform. It uses residual vector quantization to produce multiple levels of tokens, where VideoPoint uses the first form, and its quality is better than OPUS Audio Codex standard.

• 1:17:42Speaker 17

  Now that we have defined a multimodal token spaces, we can convert video data sets into discrete token sequences. Then we can use an out-of-the-box LLM transformer training infrastructure to learn these as foreign languages. In VideoPoint, we adopt a decoder-only prefix LLM architecture, where bidirectional attention is applied on the condition sequence, followed by causal attention on the target output.

• 1:18:09Speaker 17

  Compared to a diffusion transformer of the same size, the VideoPoint framework has significant flexibility and efficiency benefits at both training and inference times. It can flexibly train arbitrary tasks between any modalities together with variable length of condition and target sequences in a single model. With causal attention, the transformer learns the entire decoding trajectory for video in a single training step. At inference time, we can leverage various types of existing acceleration techniques, such as KV caching, so that the entire decoding flops are no more than one full forward pass.

• 1:18:46Speaker 17

  Video data comes from different sources in diverse formats. Text-to-video diffusion models usually require text-video pairs with high aesthetic value, which may be scarce and costly to curate. With our flexible design, VideoPoint can pre-train on a mixture of pre-existing data, where a large fraction remains unlabeled or noisy-labeled. In this paper, we have a large number of raw videos with audio from the public internet, some videos with noisy machine caption, and another set of videos with high-quality human captions. We also leverage image-text pairs to improve language alignment. After pre-training, we can have a second training phase of task-specific adaptation with the corresponding high-quality data sets,

• 1:19:31Speaker 17

  such as for text-to-video. More details about the training data can be found in the paper.

• 1:19:37Speaker 17

  We have a large mixture of training tasks on these data, starting with self-supervised ones, such as unconditional generation of various modalities. With an autoregressive model, they also imply the corresponding continuation tasks for video, audio, and both of them, as well as the image-to-video task.

• 1:19:58Speaker 17

  VideoPoint is trained to generate audio given a video or vice versa, and perform various types of video editing such as inpainting, outpainting, and interpolation.

• 1:20:09Speaker 17

  In addition, leveraging the captions, it learns to generate video, audio, and image from text. Video stylization is supported by DAP's optical flow conditions.

• 1:20:20Speaker 17

  After the LLM backbone generates video tokens, we can optionally apply a latent super-resolution module before decoding to pixels. It uses the magnet mask transformer with non-autoregressive decoding, which runs faster at small scale with multi-access windowed attention to handle long sequences at high resolutions.

• 1:20:41Speaker 17

  While VideoPoint has broad generation capabilities, much of the existing automatic benchmarks are defined around text-to-video. Here we compare with still-the-art methods on the commonly-used MSRVTT and UCF-1.1 zero-shot text-to-video evaluations. On metrics of clip similarity, inception score, and FVD, VideoPoint performs favorably against prior models, which were specifically designed for text-to-video. As automatic metrics get to be saturated and less indicative, we conduct user study with human readers to compare zero-shot text-to-video generation in various aspects. We compare against prior works including Finaki, Show1, VideoCraftur, Runway, and Pika,

• 1:21:25Speaker 17

  as well as concurrent works such as VOT and LUMIR. On access of text fidelity, video quality, motion interestingness, and motion realism, VideoPoint is preferred to prior works and preferred to concurrent works in majority cases.

• 1:21:41Speaker 17

  This is a collection of text-to-video samples by VideoPoint, and we highlight their high-fidelity motion. More samples can be found on the project page. Here we demonstrate the image-to-video capability, which can be potentially applied for 3D rendering as well.

• 1:21:59Speaker 17

  In addition, video stylization and editing are natively supported. VideoPoint can generate the corresponding audio for a video where it understands the content. Here we show a few examples where both video and audio are generated by VideoPoint.

• 1:22:32Speaker 17

  We hope our work can empower the community to explore in broader areas and greater depths. With LLM-style foundation models, we could further leverage their generalization capabilities for in-context learning of a new motion, character, or object for video generation in a customized and controllable way. We can even think of how a new modality can be added into the model at inference time. On the efficiency side, we probably want to care about how video generation can run in a real-time streaming fashion. This will not only enable interactive neural gaming, but may also facilitate neural user interface. Imagine for a neural-based operating system, it could have no more blue-screen crashes,

• 1:23:17Speaker 17

  but may reboot when it runs out of memory due to a long context. Further advancement would hopefully take us to a universal multimodal generating model that excels at text, video, audio, image, and beyond. Think about text-to-video as machine translation in 2018, when it first beat human performance. It took another five years before we have ChatGPT. I guess it will take sooner before we can reason and generate across modalities with LLM-level intelligence. Looking forward to seeing how it answers. Can you show me how to tie this shoe with a single hand in a live video? In summary, Video Poet represents a distinct approach to video generation. It challenges the diffusion monopoly

• 1:24:02Speaker 17

  with still-the-art visual quality while offering multitask flexibility which goes beyond the text-to-translation paradigm. It is a video-first foundation model with diverse generation and editing capabilities, building upon out-of-the-box LLM infrastructure for native integration. That concludes my talk today.

• 1:24:25Speaker 18

  Thanks for your great work. And actually, I'm very curious about the instruction-following ability of video point. So do you take some measures to evaluate the instruction-following ability quantitatively or qualitatively? And how does it compare with the traditional classifier-free guidance of diffusion models? Yeah, that's my question.

• 1:24:49Speaker 17

  Okay, that's a great question. First of all, I don't think there exists a very good quantitative metric to measure that thing, but it's a very promising future direction people should work on to evaluate these video generation models. Secondly, we did use classifier-free guidance in our model as well for auto-regressive. It works. And then I think it is very tricky to fairly compare with a diffusion model beyond a system-level comparison, because they use different latents, one is continuous, one is discrete, and you train on different of them, you have different reconstruction quality, then you... Well, you can calculate perplexity for language model. People don't really know how to do that for diffusion model.

• 1:25:31Speaker 17

  But I believe it really warrants further study on comparing these two methods systematically.

• 1:25:38Speaker 18

  Okay, thank you.

• 1:25:41Speaker 19

  Okay, great works and a great team. Hi, Li Jun.

• 1:25:44Speaker 17

  Hi.

• 1:25:45Speaker 19

  So we are also working on the topic of video and like that has the capability of generating video. So my question, as we all agreed on, that the video tokenization might be the bottleneck of, for example, the model, right?

• 1:26:07Speaker 17

  Yes.

• 1:26:08Speaker 19

  And so do you have any insights that you want to share about how to build the, you know, some very capable video tokenization technique?

• 1:26:21Speaker 17

  Okay. So first of all, the video tokenizer, it consists of encoder and decoder, but it basically learns a bidirectional mapping. Sometimes people use like diffusion decoder. On the other side of language model. But that means you are doing another generative model there. In video point, we only use as a pair of encoder and decoder, training with reconstruction and adversarial objects. It is a bidirectional mapping. In this case, it gets really tricky to train it for high-quality reconstruction. Because it will always be a lousy compression problem. And on the decode side, you always have one too many mapping. So the real help, I guess, from

• 1:27:05Speaker 17

  going away from the blurry reconstruction is the help from adversarial objective, which gets you sharp videos. Also, the 3D causal CNN in the Megaview 2 architecture, that helps a lot, especially when coupled with auto-regressive modeling. So you have full temporal causality for the training. It is very friendly for auto-regressive decoding. Okay, thanks. Maybe we can have one more chat. Yeah, of course. Welcome to the poster session. It's happening last minute.

• 1:27:39Speaker 20

  First of all, thank you for your work. And I have a question about open sourcing MagVit first. As I know, it was planted, but somehow MagVit first was not open sourced yet. And do you have maybe any plans about it?

• 1:27:57Speaker 17

  Okay, I may answer this question more confidently if you asked me one month ago. But now, as I joined for time, I don't really know.

• 1:28:09Speaker 17

  The good news for you is the Megaview 2 version 1 tokenizer was already open source like a year ago. Yeah, you can use that. And I think it will only take like another 100 lines of code to reproduce Megaview 2. So that you have the tokenizer, the full reconstruction and adversarial training logics.

• 1:28:29Speaker 20

  Yeah, I know. It's simple to reproduce, but it's hard to train, same to all the QVA-like methods. It's very difficult to find the right deep parameters.

• 1:28:40Speaker 17

  Yes, I agree with that part. It is tricky and it takes some hyperparameter search, especially it varies with the dataset statistics. Which I think there is a lot of room for future improvement. I believe the current solution is not perfect. And even for now, although I have worked on so many video tokenizers, my really dream, my real dream now is getting rid of them.

• 1:29:05Speaker 15

  All right, last question.

• 1:29:06Speaker 17

  Yeah.

• 1:29:07Speaker 21

  Thank you. Thank you for your talk. So you presented a really interesting direction in the multi-model foundation model.

• 1:29:15Speaker 17

  Thanks.

• 1:29:16Speaker 21

  From what I see, the whole architecture or the approach in training is very much sequence continuation, right?

• 1:29:23Speaker 17

  Yes.

• 1:29:23Speaker 21

  So I'm wondering if you work on some more capabilities or architectural components which help the model to generalize and to see, how to say, to simplify the connecting the dots, especially in video and audio signals. It is a very hard task for any model to see the structure behind the diversity. So if you're thinking of working on any in this direction.

• 1:29:58Speaker 17

  Okay, that's a great question. First of all, one of the advantages of Video Poet is we take the ARM training infrastructure from an Autobots version. We take it for granted and we actually made no modification to the model architecture and training recipe. So all you need to do is define your token space and curate your sequence datasets.

• 1:30:20Speaker 17

  I think that part requires some really smart designs, like you can have text to image as a prefix of text to video and you have video to audio tasks as the prefix of unconditional video audio generation, stuff like that. So you help model generalize beyond different tasks.

• 1:30:42Speaker 15

  All right, so thank you so much for the talk and congratulations again on your best paper award.

• 1:30:48Speaker 17

  Thank you.

• 1:30:50Speaker 16

  I'm here with Dan Konradjuk to talk about the Video Poet, a large language model for zero-shot video generation poster. Dan is with Luma AI, which is one of the leading companies in the AI video generation space. Dan, can you give me an overview of kind of how you started working on this and maybe a brief, high-level summary of what it is you have here on the poster?

• 1:31:12Speaker 22

  Yeah, so we started... Yeah, we started this project as mainly a way of thinking about video generation from a foundation model perspective. So, foundation models, typically, when you think about them, I guess at the time, they were all like language models, or visual language models. So they output primarily text, but we thought, what if we approach it from a video perspective? And this approaches the design from a very different perspective from how the current video generation models are, which are primarily diffusion-based. So, like, we thought, maybe we could envision a task where we take off-the-shelf language model.

• 1:31:53Speaker 22

  One of the things we changed the least about this project is like, we just take a language model, we don't do anything special with it. And our real innovation here is on the data side, like, how you design the tasks as input to the language model. And the way we designed it is, we translate all of our modalities, so text, video, image, audio, into one embedding space. That means you translate into one language that the language model can understand. So, typically, when you think of language, it's like human language, natural language, the type of text that you can read.

• 1:32:34Speaker 22

  But you can actually think of images, video, and audio as a type of language too. So we have this tokenizer, which we called MagBitV2, which takes this, for instance, a image or video, and translates it into a discrete sequence of tokens with a very large vocabulary. So there's like a vocabulary of, say, 200,000 tokens, and that can be input directly into a language model. So this language model speaks the language of video in some respect. And all we do is just train it on hundreds of millions of videos. I think we trained more than a billion images, and also, some of our dataset had video and

• 1:33:18Speaker 22

  audio pairs. And depending on how you order things, we have this bidirectional attention prefix. It just means that we input these, and the model has a way of incorporating all these modalities, this text, images. We also have some alternative types of dense input prediction for stylization, audio.

• 1:33:42Speaker 22

  And depending on the order, what you input in the beginning, you can condition it to output different things. So for instance, you input text, you can output video. So depending on your description, it outputs like, an astronaut starts dancing on Mars, and then it starts generating that output video based on how we trained it. Similarly, with our output audio, we can also do, for instance, take an input image or video and try to generate accompanying audio with it using some audio tokens generated by a sound stream, which is a previous Google paper that did kind of language modeling on these audio tokens.

• 1:34:28Speaker 22

  So our approach is primarily how we combine these tasks together. And we see we're not the first to show that you can use a language model for this type of generation, but we are a work that shows that you can actually scale this to a level that's actually competitive with existing works that do video generation. So you can see, it can do, because it's a foundation model, like it can do tons of tasks just based on how we were able to design it. Like for instance, you can do text to video. We can do image animation. Take an input Mona Lisa, and you just ask Mona Lisa to yawn, and all of a sudden, it just like, based on what you described,

• 1:35:13Speaker 22

  what you want the image to do, it just does it, which is really cool. And then we can chain this with other tasks, like for instance, stylization. If we use something like depth and optical flow conditioned, it just basically strips out all of the contents of the original video and condition it on just the depth and optical flow. If you describe it like, oil painting of a snowman with a red hat, opening their mouth to yawn, and then it just paints on top with the same motion as the original video. So that's another really cool thing that just was able, the model was able to do. And then we have a whole bunch of other tasks, like it can even generate audio.

• 1:35:53Speaker 22

  It can outpaint videos where we take an input video and try to paint more contents on the bottom and top. So overall, we evaluated the results and we see that the results are quite competitive with a lot of existing works. In fact, exceeds the most of the works that we tried, which is really cool. And one, a couple of things in particular the model does really well is on prompt following. Because we can train it as like a language model, it's actually easy to scale with existing infrastructure that we had. And also, it does pretty good on motion. You can apply existing image or like text to video results.

• 1:36:35Speaker 22

  And it's like compared to the other works that we tried at the time. It applied much bigger and more motion that looked interesting in the video rather than having something that moves very slightly, like more akin to image animation.

• 1:36:54Speaker 22

  So that's like the overview of the work. If any questions, we'll be glad to answer.

• 1:36:59Speaker 16

  Yeah, if you, so we're talking to an audience here of folks who are AI engineers. They build applications, oftentimes using some of these models as the underlying underpinnings of the AI part. So I'm curious if you would say that the work you've done and the language model approach is a better fit for some use cases versus others? And if there might be other use cases where diffusion models may still be better? Or how would you think about like the trade offs, I suppose?

• 1:37:23Speaker 22

  I think there are a couple of things. If you want to do something that does very good pixel quality, I think diffusion models are still unmatched in this regard. And that's primarily because of the tokenizer. The tokenizer does extreme level of compression. So that's why we're forced to like generate at these pretty small resolutions and need a super resolution model to increase the fidelity. But with a diffusion model, you don't have that restriction. You can do diffusion over these latent tokens that are not as compressed. And as a result, it's a bit easier to get like these high resolution, high level of quality results. However, one thing that diffusion models have a problem with is it takes a very long time to converge,

• 1:38:08Speaker 22

  very long time to train. I think the language model approach is definitely Quite a bit more efficient. Like, we trained it only for a few weeks. And already, we, like, it converged, like, pretty well. And it scales proportionally to existing, like, language modeling approach. So you can easily predict, like, if we keep increasing the model size, as we see here, like, 1 billion model is pretty good, but 8 billion model, we just, like, increase 8 times more parameters and we get much, much better results. I suspect if we just, like, keep increasing the model size, it'll keep improving. So that's also another nice result.

• 1:38:51Speaker 22

  I, diffusion models do have a scaling property, but it's a lot harder to predict, I would say. So I think some of the nice things about language model is a lot more research has been done on the scaling properties.

• 1:39:05Speaker 22

  And there's also, because the tokens are flattened into a 1D sequence, you can do this multimodal representation, whereas diffusion model typically only operates on, like, one modality. There's also maybe, like, you could try to do something like video and audio generation at the same time with a diffusion model. But I don't think it's, you can generate all modalities combined at the same time with diffusion model, just, yeah, at the same level of quality. Like, text diffusion right now is really hard to do and has not had the same level of performance as, like, auto-regressive models. So if you want a general foundation model to do everything all at once, I still would

• 1:39:49Speaker 22

  say language model is at the top right now. But who knows? Like, people are doing research in many different areas. If someone can crack text diffusion, I think you could also create a foundation model that does all these modalities.

• 1:40:03Speaker 16

  And is this work that you've been doing in the context of your role at Luma? And is it work that you plan to continue to kind of push forward in that context?

• 1:40:11Speaker 22

  So I recently left Google to join Luma to do, like, some video generation that I was really excited about. So this is just a work that I worked on while I was at Google. I think there's some continuation of this work possibly in the future, like this general approach. Obviously, Video Poet is still not out, and I think that's just a testament to how fast the field moves right now. It's just like incredibly competitive space. But I do think like this general approach could surface in many different areas in the future. Who knows? Like, right now, language models and diffusion models are battling it out in this battleground. So who's to say which one will win out in the end? Both approaches have been shown to work pretty well.

• 1:40:56Speaker 22

  They have their strengths and weaknesses right now, and more research is going in this space. So right now, I'm really excited about video generation going forward to build out like these more general-purpose models. But at least for this work on Video Poet, I'm also like really excited about like the future prospects of, you know, what's gonna happen next.

• 1:41:18Speaker 16

  Awesome. Thank you so much.

• 1:41:20Speaker 00

  You may have caught that Dan Kondratyuk, the lead author of the Video Poet paper, has left DeepMind to join Luma Labs, which is responsible for the LumaDream machine model that went viral for turning popular memes into videos this year. To tie off our generative video discussions, we will bring in Tali Dekel's invited talk from the text Camera Action. "Frontiers in Controllable Video Generation Workshop" on Saturday on "The Future of Video Generation: Beyond Data and Scale." As a reminder, all talks have public links. So if you want to see the videos she is talking about, click into the show notes.

• 1:41:58Speaker 23

  Hi, everyone. I'm Tali, and it's a great privilege to be here. So today, I'm gonna talk about the great revolution that we are witnessing in generative AI and especially in video generation. And as you know, models in this domain requires a tremendous amount of training data and compute. But I'm hoping to convince you, based on my own experience and work, that I think that the future of video generation goes way beyond just data and scale. It's gonna be a high-level talk that's gonna cover different topics, but I do hope to also dive in into technical details on the more recent works. So again, in the context of this workshop, I think it's redundant to say that,

• 1:42:41Speaker 23

  you know, we are all aware of the fact that the generative AI revolution has been recently expanded to videos and we are now not only able to generate these whoop, what? Mind-blowing still images, but we can also make everything move. And really, I think the past couple of years was shown a dramatic, rapid development in this area. And when we are witnessing this progress, we can start envisioning how movies, movie production and the film industry might look like in the near future and think that we might be able to generate movies completely computationally. So maybe it will look something like that.

• 1:43:27Speaker 23

  We'll ask ChatGPT to help us with the script like this and it will generate the script for us. And then also it will take the script and generate the movie completely computationally like this.

• 1:43:48Speaker 23

  And maybe we'll then ask it to add some special effects like a bullet time effect and it will just do it completely computationally without any real actors or cameras, just using generative AI.

• 1:44:13Speaker 23

  Yeah, you can hear it outside. Okay, sorry. No more audio in this talk. And if you are too young, you probably don't know, but this is not a real generated video. It was taken from The Matrix movie that was produced somewhere in the 80s, 90s. Okay, so I'm sorry to disappoint you, but I think we are very far from this future. And despite all the amazing progress that we are witnessing, state-of-the-art text-to-video models still depict some fundamental failure cases, even models like Sora. So for example, they tend to fail to simulate real physical interactions in the world.

• 1:44:56Speaker 23

  Like this object here is supposed to be a rigid chair. And you can see it is floating in an unrealistic manner in space.

• 1:45:06Speaker 23

  In this example, the treadmill follows the person also in a physically implausible fashion. And also, when we are dealing with more complicated scenes that involve multiple entities, objects tend to unrealistically appear and disappear spontaneously. And this basically tells us that video generation is still not solved.

• 1:45:33Speaker 23

  And furthermore, the costs of scaling up video models and developing these universal foundation models are just huge. You know, a single model training requires about roughly, on average, 200K GPU hours, which translate to almost 280K dollars. And this is just translate to millions and millions of dollars to train such models. And in terms of energy consumption, just to generate half a second of a video amounts to driving roughly four miles on an average car.

• 1:46:12Speaker 23

  And because of these costs, it leads us to the fact that these video foundation models are being sealed in industry and there are also, there are only very few big players in industry that can develop and design such models. So what do we do on the research community in that case? And also, if we go back to our moonshot goal of, you know,

• 1:46:37Speaker 23

  generating films completely computationally, in order to do that, we need explicit fine-grained control. We may want to exactly control the camera position, the character identity, their emotion, their positions, their movements. We may want to control lighting and also sound and speech. And all of these controls are not currently provided by video foundation models.

• 1:47:03Speaker 23

  So my research journey in the realm of videos has actually started on the other side of the spectrum with single video models. And what do I mean by that? I mean that basically we have some neural-based framework that is overfitted to a single test video. So as NERF, for example, is overfitted to a single 3D scene, in this case, we have some neural networks that only observe this test video alone without any additional data. And it turns out that you can do some pretty impressive things with these single video models. So for example, we showed how you could take this really

• 1:47:47Speaker 23

  busy and complex scene, and let's say you want to just focus your attention on a single dynamic object so we can actually remove all the rest of the moving people in this scene, except this girl. And you can notice how not only we remove the people, we also remove the complex deformation that occurs to the trampoline in this case.

• 1:48:12Speaker 23

  Here, this is a video of my son riding his bikes for the first time. And I can take this video and stylize only the background. And you can see that everything moves consistently and physically correct with the original scene. And these works are from 2021, before the big generative AI revolution. And we can also, you know, not only map texture onto rigid objects, we can also map texture to deformable articulated objects. So for example, we can add these flowers to the dress, and they are moving in a physically correct manner as the original video.

• 1:48:55Speaker 23

  And again, these models, the only information they have about the world is just the single video, the input video on the top. Of course, their big disadvantage is that they don't have this rich and powerful prior knowledge about the world.

• 1:49:15Speaker 23

  So just to show the advantage of this in more detail, I think one of the big advantage of this approach is that it allows us to go way beyond just working with raw, huge pixel volumes. We can design sophisticated and more advanced representations for real-world videos. So in layered neural atlases, our key idea, we wanted to support this consistent video editing and the key idea was to basically turn the video or estimate from the video a unified set of canonical images. So given this input video, we estimate two atlas images, like you can see here, one for the background

• 1:49:59Speaker 23

  and one for the foreground, that represents either the entire background or foreground for the entire video. And each pixel position from the original video is being mapped onto these atlas images and this allows to basically reconstruct the original video from this representation. And now, the key advantage of this representation is that it allows to reduce this really difficult task of editing huge pixel volumes of real-world videos to editing a single 2D image. So what you can do, you can just take these images, plug them in into any image editing framework or just load it up in Photoshop and draw some stuff on it

• 1:50:42Speaker 23

  and then use the mapping to map it back to the original video. Sorry, the animation doesn't work. Okay, of course that I'm showing you a discrete set of images, but in practice, everything is being implicitly represented through MLPs and through neural network. So very briefly, each pixel position in the video is fed into these MLPs that maps it into a 2D coordinate in this atlas space. So this is just a 2D coordinate between minus 1 and 1 and you have two such networks for the foreground and the background. And each such position in this 2D unified space is fed into another MLP

• 1:51:27Speaker 23

  that predicts the RGB color of that position. And there is also another small MLP that predicts the visibility of each point, how much it observes from the background versus foreground. And this allows to basically reconstruct the original color of the video to each position and to train this entire framework completely in a self-supervised manner, where the driving loss is a video reconstruction loss. There are other terms in the objective function to make sure that this representation is interpretable, that the structure are being preserved, that the correspondences in the videos are being preserved.

• 1:52:08Speaker 23

  But basically, you can train these things end to end in a self-supervised manner.

• 1:52:16Speaker 23

  Okay.

• 1:52:20Speaker 23

  Here you can see the edit being mapped. Okay, so on one hand side, we have this, again, video foundation models. They require this huge cost to train. They are limited. We don't have access to them in the research community that much. And they provide limited controllability. On the other hand, they can learn this really powerful, amazing spacetime priors about our dynamic world. On the other side of the spectrum, we have the single video models that requires only few GPUs to train. They are accessible and they allow us to be much more flexible and creative in the way we represent video content.

• 1:53:04Speaker 23

  However, they do not have any prior knowledge about the world. Okay. So you can probably guess that the way I think we should go about videos is actually to combine the best of both worlds. And what do I mean by that? So, on the one hand side, we want to have this flexibility and this freedom to represent video content and to gain explicit control over what we are synthesizing. On the other hand, we want to fuse into this representation external knowledge learned from universal models.

• 1:53:45Speaker 23

  And this not only restricted to just video models, we can integrate external information from an ensemble of foundation models that can provide us motion priors, generative priors, generative priors, and semantic priors.

• 1:54:03Speaker 23

  And my first attempt to do so was in Text2Live. So, in Text2Live, we wanted to support text-driven editing. And I think it was, to the best of my knowledge, the first method to demonstrate text-based editing for videos, for real-world videos. Again, this was ECCV22. And the key idea there was to use a pre-trained neural atlas representation of the video as a video renderer. We're gonna have this representation, keep it fixed, and then replace the manual edits that we can perform on the atlas images with automatic text-driven edits described by text.

• 1:54:44Speaker 23

  And to achieve that, we combined this representation with a pre-trained clip model back then that allowed us to gain this for the first time.

• 1:54:55Speaker 23

  And here you can see how we can perform localized and semantic editing to real-world videos without any real generative model. This was just using clip. And again, I think that, you know, performing these localized semantic edits and the type of edits that they showed you for removing dynamic content is still a challenge even to big foundation models that are very powerful.

• 1:55:22Speaker 23

  Again, with all the respect to CLIP and this approach, with the rise of text-to-image models, we wanted to take this approach further and to think of how can we leverage stronger priors about the world. And I think one of the main challenges in pursuing this approach of combining external knowledge to these sophisticated video representations is that most foundation models are basically black boxes to us. We do not understand exactly the priors that they learn and how these priors are internally encoded. So this approach poses this challenge of how to distill learned priors from black boxes.

• 1:56:06Speaker 23

  And basically, one of my research aim is to, an approach, is to dive deep inside those foundation models and find out, like just reveal more, gain better understanding about what they learn and their internal representation. And if we can achieve that, then we can build much more, much better algorithms on top of them.

• 1:56:34Speaker 23

  So with the rise of text-to-image models, diffusion models, like stable diffusion, I was really amazed by the ability of these models to capture these really complicated signals about our visual world. So just viewing these images, we can see that these models can learn priors about composition, about pose, about interactions between objects, appearance, and so on. So I was focusing on this aim of, you know, taking text-to-image models way beyond what they are meant to do, way beyond just generating images from text. And we had a line of works in the lab

• 1:57:16Speaker 23

  that introduced some of the early works in this space. So for example, in Plug and Play, we conditioned the generation not only on text, but also on a reference image. And the output image preserved the semantic layout of the original reference image. In Multi-Diffusion, we extended pre-trained text-to-image models to generate images at arbitrary resolution and also to receive as input region-based text controls, like you can see in these examples. And in the context of videos, I was thinking, how can we take these powerful priors, the text-to-image learn,

• 1:57:59Speaker 23

  and extend them to video synthesis tasks? So in Europe, we introduced Scenscape, that allows not only to generate beautiful scenery, but also to walk through, to generate 3D plausible walkthroughs inside those scenes. And behind those videos, there is actually a real 3D mesh representation of the scene that is being built. And in TokenFlow, we showed how can you not only synthesize static scene, but actually edit real-world dynamic scenes.

• 1:58:33Speaker 23

  And I think, again, many, a huge bulk of work is doing that, like adapting text-to-image models, expanding them in various ways. I think what's kind of like more unique in these works is that we insisted in keeping those text-to-image models fixed and striving to better understand the generation process, the internal representation, to make these black boxes more transparent and utilize our understanding of them. So I want to dive more deeply into some of the works. So let me discuss in more details,

• 1:59:18Speaker 23

  in more detail, TokenFlow.

• 1:59:22Speaker 23

  And again, our key, our goal in this work was to perform this consistent video editing. And we started with this naive baseline of applying plug-and-play or a different method to edit each frame independently. And as you can see, the content is really inconsistent. It's not adjusted at the level of high-frequency flickeriness. The content really changes from one frame to frame. And there is really no reason to believe that the text-to-image model would give us something else.

• 1:59:54Speaker 23

  So we wanted to dive inside the model and understand how these inconsistencies are being represented inside the model. Okay. So in order to do that, we take the original video frame by frame. We use some inversion technique to invert it back to the model. And then we can just extract some features from intermediate layers. And because those features are really high dimensional, we cannot make sense of them. So we use PCA to reduce them into three dimensions and visualize them as videos. So here you can see the original video. And on the right hand side, you can see the PCA reductions of tokens or features

• 2:00:38Speaker 23

  extracted across different levels of the unit.

• 2:00:44Speaker 23

  And what we can easily observe is that this PCA visualization, they depict a shared and consistent representation. We can see that the consistency in RGB and the features resemble, again, a similar consistency in its feature space for this video.

• 2:01:09Speaker 23

  So we wanted to look at this consistency in more fine-grained manner. So in order to do that, we looked on nearest neighbors. You take a feature at a certain position in one frame and just compute its nearest neighbors to all the rest of the frames. And what we saw is that those correspondences, they exhibit this semantic and accurate matching across different frames, as you can see in these examples.

• 2:01:39Speaker 23

  And you can compute this nearest neighbor field densely. So for each, if you are given two frames, you can take each feature in the source frame and compute its nearest neighbor in a target frame. And this will give rise to this dense nearest neighbor field, which we named Token Flow.

• 2:01:59Speaker 23

  So this provides us with semantic and accurate matching, but we wanted to see also to gain more information about what these features hold in terms of information about the frames. And in order to do that, we checked how well we can generate the target frame from the features provided from a source frame. So this has been done by basically taking the source frame and the target frame, extracting their features, computing the token flow, and then just warping the source feature tokens. And now we can intervene in the generation process of a target frame.

• 2:02:44Speaker 23

  We basically do DDM inversion to get the initial latent, but then we swap each feature of the target frame, computes its nearest neighbor from the source frame, and we just swap the features. So we want to check, you know, how the generation of the target frame would be impacted by this swapping. And we observed that the target frame can be synthesized accurately from the source features, which means that those features are interchangeable for the model.

• 2:03:24Speaker 23

  Okay, so what happens now? Again, we applied this per-frame editing, and we saw that the consistency breaks in RGB.

• 2:03:33Speaker 23

  What happens to the features? Here you can see the feature visualization of this per-frame edited video, and you can see that the features depict the same inconsistencies as in RGB. So basically, consistent features gives rise to consistent frames and vice versa. So our key idea in TokenFlow is that in order to achieve consistent editing, we want to achieve consistent features during the generation process. And the way we suggested to do that is by enforcing the original TokenFlow or the original feature matching of the original video on the edited video. So you can see the edited video

• 2:04:17Speaker 23

  and the underlying features of that edited video.

• 2:04:24Speaker 23

  And just to summarize, so this method works as follows. We take the original video, we do the DDM inversion, we extract the features and compute the token flow. And then during the generation, the generation process of the edited video is composed of two stages. In the first stage, we sample some keyframes and jointly edit them with extended attention. This gives basically just rough global coherency between the frames. And then we extract the features of these edited frames and we propagate them using the original token flow of the original video to the rest of the frames. And we repeat this process.

• 2:05:12Speaker 23

  Here you can see some generation results.

• 2:05:18Speaker 23

  And

• 2:05:21Speaker 23

  comparison to several methods. Again, I think since we published this work, generated a great body of follow-up works. You've seen the nice work on editing XD slices today. So these matching and token flow correspondences, they hold between nearby frames. But indeed, when the frames are more distant from each other, those matching, those matches tend to be incorrect. So indeed, our method would break for very complex and motions where these correspondences would be difficult to achieve.

• 2:06:01Speaker 23

  Okay, so I guess I talked about how can we use text-to-image models beyond what they are meant to do. But the main limitation of just using text-to-image models is obvious. It only provides us with 2D information and we don't have any motion priors. And if we really want to model our dynamic world, we need to know something about how objects move, how they tend to move in the real world. We want to know priors about actions and

• 2:06:32Speaker 23

  And that's something that text-to-image model cannot provide us. But again, I remind you all that we are in this amazing world where progress happens really fast. And now we have these powerful video models. And that really motivates their use and their use of and their understanding of motion in various applications. It could be generative task, but I don't think it has to be limited to that.

• 2:07:03Speaker 23

  Okay. So that brings me to the last work that I want to talk about. Space-time features for text-driven motion transfer that was presented at last CVPR. And the motivation there was, again, film industry and the, you know, big efforts that manual work and professional work puts into transferring motion from motion markers and so on to animation using this CGI type of animation. So we wanted in this work to achieve this computationally. So given an input-driving video like this dog jumping to a river,

• 2:07:46Speaker 23

  we want to be able to transfer it to dramatically different objects just using simple text prompts, like you can see here.

• 2:07:56Speaker 23

  And you can see that the big difference between this setting and this task compared to, let's say, what we've done in TokenFlow is that you must enable deviations from the shape of the original objects in order to convey or to fulfill the target edit. In order to transfer the motion of this dog to a dolphin, I must change the shape of the dog dramatically and adapt the fine-grained characteristic of the motion such that it will be plausible, natural with the target object. Maybe the dolphin moves its tail in a certain way and so on. So we really need to distill the essence of the motion from the driving video,

• 2:08:41Speaker 23

  but be flexible enough to allow this adaptation of the content in order to fulfill, to get a naturally looking edit. And for that, we must have a prior about how things are moving in the real world.

• 2:08:57Speaker 23

  So in this work, we used ZeroScope, one of the publicly available text-to-video models. You can see some samples from this model. So it's way, way far from, you know, state-of-the-art text-to-video models that are keep being better and better. But this model still is able to learn valuable information about our dynamic world. Okay, so just in context of this work, we are not defining motion anymore as pixel-level correspondences. Because again, we want to allow this flexibility and deviation from the shape of the object. So in our context for this task, motion is defined as a sequence

• 2:09:42Speaker 23

  of semantic objects, parts, positions. So you can think about an object as being, you know, just a set of the parts that, and how they're general progression throughout the entire video.

• 2:10:00Speaker 23

  And again, in terms of related work, I think none of the existing method allows, is not designed to enable this big deviation in the structure of the objects.

• 2:10:15Speaker 23

  So we follow token flow and took a similar approach and asked ourselves, how space-time information is internally encoded in this text-to-video model? And again, we want to dive deep into the features and understand them better. So in this case, our input is a video, and we can directly invert it into the video model,

• 2:10:39Speaker 23

  again, using off-the-shelf DDM inversion technique, and extract features. And in this case, the features are

• 2:10:47Speaker 23

  four-dimensional. So F is the number of frames, M, and by N is the spatial dimensions, and D is the number of channels.

• 2:10:57Speaker 23

  So here, instead of doing PCA visualizations and so on, we adapted a feature inversion technique. So I guess many of you are familiar with it in the context of understanding classifiers, pre-trained classifiers. It's a classic method. So the general idea is that we have some pre-trained and fixed model. We take our input, we fit it into the model and extract some target features. And in order to understand better what these features encode, now we solve this optimization task where we want to optimize for an image in this case, such that when we'll fit it into the model, it will give rise to the same target features. And in many cases, of course, you need to somehow regularize this optimized image

• 2:11:41Speaker 23

  to avoid adversarial solutions and so on. So in our case, our input is not an image. It's a video. We can fit it into the model, extract features. And now the goal is to optimize for a new video, such that when we'll fit it into the text-to-video model, it will give rise to the same features. Okay? And if we solve this optimization task, so again, you can see the objective at the top and the original video on the left. You can see the feature inversion results from different seeds at the right. And you can see that we can accurately reconstruct the original video in terms of appearance, motion, and so on.

• 2:12:26Speaker 23

  And this is not what we want because we want to allow much more flexibility in both in terms of shape and appearance. So how can we take these spacetime features and build a descriptor out of them that will allow us this flexibility? Our first step in towards removing this pixel-level dependency was to average out or reduce the spatial dimension. So we basically take these features for each frame and just average pull them across the spatial dimension. So for each feature, we have a d-dimensional vector. And so to describe the entire video, we have F by d tensor.

• 2:13:13Speaker 23

  And now we can repeat our feature inversion experiment with those spatially reduced features. And we were really surprised when we got this result to see that even though we averaged out the information across space, you can see from this inversion that we still preserve the pose and accurate movements of the woman in this video while allowing for more flexibility in the structure and appearance. And just in terms of intuition, again, those features are really high dimension, they live in this high dimensional space, so even though we averaged them spatially, this information can still be preserved.

• 2:14:01Speaker 23

  Okay, so in the next step, we said, okay, so let's use these features for editing. We're given some video, the original video, we can extract those spatially mean features from the original videos and just use them as guidance during the generation process of the edited video. So you can see the equation up here, but we basically want to optimize the latent such that when we denoise them with a target text, in this case a camel, we want the resulting features, the spatially reduced mean features, to match those of the original video.

• 2:14:45Speaker 23

  We do that through guidance, through the generation process, and you can see here the result. So indeed, it allows for some flexibility, we can get different deviation in shape and in appearance, but still it looks kind of like a camel that was squished into the shape of the elephant.

• 2:15:09Speaker 23

  Y, así que estas características, aunque las medimos, todavía contienen esta información, demasiada información sobre los objetos originales en el video. Y eso nos llevó, básicamente, a construir la matriz de diferencias de parados SMM. Y esta idea es, básicamente, es inspirada en toda esta línea de trabajos, de la auto-similitud. Que básicamente no queremos encodear los valores absolutos de estas características, pero solo encodear cómo se relacionan entre sí, todas sus relaciones paradosas a través del video. Así que, básicamente, tomamos estas características de 4D para cada frame, y construimos esta matriz de f x f, en la que cada entrada es básicamente

• 2:15:54Speaker 23

  solo la diferencia entre dos características especialmente medidas. Y se puede pensar en ello como encodear algo de movimiento en este espacio semántico de características, porque estamos solo encodando todas sus diferencias paradosas y deltas entre todos los frames. Y ahora queremos, de nuevo, intervenir en el proceso de generación del video objetivo y usar la guía. Pero esta vez queremos animar los videos generados a tener la misma matriz de diferencias de parados SMM. Así que esta será nuestra función objetiva durante el proceso de generación del video editado.

• 2:16:35Speaker 23

  Y ahora se puede ver que podemos obtener un camaleón mucho mejor y todavía preservamos el movimiento en el video original.

• 2:16:49Speaker 23

  Aquí se pueden ver algunos más ejemplos. Y creo que, ya saben, si miran la transferencia del movimiento de este perro a los conejos, entienden que realmente queremos sintetizar los conejos aquí, y necesitan moverse de una manera realista, como los conejos tienden a moverse. Y eso realmente, creo, exemplifica la necesidad de tener un movimiento antes.

• 2:17:13Speaker 23

  Hay algunos más ejemplos con cambios de forma más dramática.

• 2:17:25Speaker 23

  And some more examples on well-known videos.

• 2:17:30Speaker 23

  We also have a way of initializing the initial latent of the video. I'm not going to go into the details of that, but we use a combination of DDM inverted noise and in low frequencies with random noise at the high frequencies. And this allows the method to be more robust and less sensitive to the exact seed that we are using in the optimization.

• 2:17:57Speaker 23

  And again, compared to previous method, they really tend to preserve pixel-level correspondence and they are not able to fulfill the edit in a way that is flexible enough.

• 2:18:16Speaker 23

  So how do we measure success here? In order to measure the fidelity to text, we can use ClipScore, but we wanted to somehow quantify how well we capture the motion of the original video. And again, we want to measure that under these dramatic shape changes, so we can no longer measure just pixel-level similarity between motions. So we suggested a different geometric for that. And we suggested to measure the similarity based on the similarity of two sets of unaligned trajectories. So you can take off-the-shelf tracker and just apply a tracker on the original video and on the edited video, and that provides us with these two sets

• 2:19:01Speaker 23

  of long-range trajectories. And now we can measure their similarity using the Chamfer's distance, where the distance between two tracks, here, we use the just correlation between the tracks. So each trajectory in one set finds its nearest trajectory, a highly correlated trajectory in the other set, and vice versa, and we sum those correlation values.

• 2:19:31Speaker 23

  So here you can see the evaluation of different methods. So on the Y-axis, we have the motion fidelity score. So higher is better. And on the X-axis, we have the clip similarity score. So we want to be on the top right as much as we can. So, and you can see that our method provides the best trade-off between providing good motion fidelity and fulfilling the text. Token flow, which preserve with high fidelity the original motion, gets a better motion fidelity score, but pays in clip score because it cannot fulfill the edit fully.

• 2:20:15Speaker 23

  SD edit on the video model with low noise level, again, is able to preserve the motion with high fidelity, but it cannot deviate much from the original content of the video. And if we use SD edit with high noise level, it's the vice versa. It's the opposite, like we can fulfill the edit, but we can no longer preserve the motion. And again, our method provides the better trade-off between these two ends. Of course, there are, you know, some limitations. So we are still bounded to the priors that can be provided to us from the text-to-video model. So if the target object cannot be fitted in terms of the motion, of the video prior to the motion of the source object,

• 2:20:59Speaker 23

  we will get deviation and this weird motion happening, as in this example. Okay, so just to summarize, I talked about the two ends of video generation, editing and synthesis. The video foundation models on one hand side, the single video models on the other hand. And I hope I managed to convince you that this approach of combining the two is effective and powerful. There are still tons of stuff to do in order to pursue this goal. We still need to understand these huge, big foundation models and devise new, smart representation

• 2:21:43Speaker 23

  in order to fuse this information into them.

• 2:21:48Speaker 23

  And there are lots of open questions on how to do that. I'd like to thank all my students and collaborators from Google and from Weizmann. And I'll continue to work towards breaking new grounds in video analysis and synthesis tasks. And hopefully, in the future, we will be able to generate even such professional effects using computational tools. Thank you.

• 2:22:15Speaker 03

  So you mentioned that the open, like, obviously, open source video models, there's a huge gap in performance compared to what, you know, we can see. What do you think there's still to be done that doesn't really require training?

• 2:22:32Speaker 23

  That would, sorry?

• 2:22:33Speaker 03

  That does not require training a model. So what do you think? For example, in text-to-image models, we saw so many, like, so many papers on different ways of controlling images. What do you think we can do in videos that would be similar?

• 2:22:51Speaker 23

  Yeah, so I think the last work I showed takes a first step in this direction. I think that when you see this generation results, it is evident that these models learn some useful representation about motion, about how things evolve over time. And I think utilizing the internal representation of text-to-video model is still very underexplored. And there's tons of stuff to do there that won't require heavy training in order to adapt them or to leverage them for various downstream tasks. It could be generative tasks, but not only. And I think there is a great potential of, as we all use, you know, pre-trained image

• 2:23:35Speaker 23

  features for various tasks, I think the way to go forward is also to use video features for downstream tasks. And in order to do that, I do think we need to understand these models much better. And I think there are also many open questions about how to gain control over video generation, what will be the correct interface, how intuitively would you would want to even interact with videos? I think it was discussed here at different talks that just using text is not sufficient in order to model our dynamic world and we need to build new tools, new representation,

• 2:24:16Speaker 23

  new intuitive interfaces to interact with dynamic content which is currently not there yet.

• 2:24:23Speaker 24

  Hi, thank you for the interesting talk. So my question is a bit of all of what you just highlighted and more on the like the cost side of universal video models. So like, what would be your thoughts on like, like, since we are in the early stages, do we anticipate like an order of two reduction in the cost? And it could be algorithmic, it could be on the architecture side. As you said, like, how do we control these models might even be the factor that takes us to like this, like the two order of magnitudes further? So what's the future look like compared to where we are today?

• 2:25:05Speaker 23

  I think also it was discussed here in previous talks, but I really think that one missing ingredient in order to push the boundaries of video foundation models is compression. Like, how do you effectively represent or compress information across a video? Right now, I feel that, you know, the early stages of video foundation models are mostly doing the straightforward extensions that we can think about from the image domain and building an effective and video compressor that you can work in its latent space. I think

• 2:25:44Speaker 23

  that will be crucial for pushing the boundaries of video generation order of magnitude more. Yeah, so and I believe we'll get there. It's just a matter of time. Yeah.

• 2:26:00Speaker 24

  Yeah, hopefully. Thank you.

• 2:26:02Speaker 23

  Thank you.

• 2:26:03Speaker 00

  That was the end of part one of this pod on generative video. In part two, we turn to exploring related topics in generative modeling and diffusion that we feel represent the most important work of 2024, that are also helpful building blocks for generative video. First, we have two more DeepMind researchers. You may be observing a pattern in how much work DeepMind is putting into multimodal generative AI. Here is friend of the pod, Sander Dealerman, who works on both DeepMind's VO, video generation model, and Imagen 3. Over the past year, Sander has developed an intuitive interpretation of diffusion.

• 2:26:45Speaker 00

  Where traditionally diffusion models and autoregressive models are viewed as polar opposites, with different hardware utilization and inference paradigms, Sander's perspective of diffusion as spectral autoregression in the frequency domain caught the community's imagination this fall, and for the first time, Sander expands upon this in his workshop.

• 2:27:07Speaker 25

  So I'm going to talk about an intuitive look at how diffusion models work, and specifically in the context of modeling audiovisual data, sort of in the spirit of the theme of the workshop.

• 2:27:22Speaker 25

  So it's roughly structured in four parts. So the first thing I want to do is explain how diffusion works from a geometric perspective, because I think this intuition is really valuable.

• 2:27:34Speaker 25

  And one thing that sort of bothers me about the diffusion literature is that it's, you know, as a beginner, it must be extremely confusing, because there are so many different formalisms, so many, so many different ways of saying the same thing. And I think this geometric perspective is sort of a nice way to tie it all together and link these things together. And then the second section, I'll try to highlight some other perspectives that I think are useful and maybe less well-known. And then, in the third section, I want to talk about diffusion guidance, which is a very powerful tool, that is also very easily explained with this geometric perspective. And then finally, I want to talk a little bit about Imagen 3 and VO, which are the models that I've been working on recently.

• 2:28:17Speaker 25

  So first, let's talk about a geometric perspective on diffusion models. So I don't need to repeat this probably, but we know that diffusion works with iteratively noisy, so we have some data distribution that we're trying to model in the examples. I'll show this will be an image distribution. And we gradually add a bunch of noise, and then we try to remove it. That's diffusion, and diffusion model is in a nutshell.

• 2:28:43Speaker 25

  So I'm going to talk a little bit about this corruption process first. So we first define a way to destroy all the information that is in the data distribution. And so I'm going to take an example here from the training data. I'm going to call that X naught or X zero. The index zero stands for a time step in the corruption process. So we treat this as kind of a temporal process. And at time step zero, we are in the data distribution. And then this process will proceed by adding small increments of Gaussian noise, which I've called delta here. So think of this as a tiny amount of Gaussian noise. And we just do that repeatedly. We add these small increments repeatedly. And then at some time step T in the process, we can look at what our image looks like, and it'll be a noisy image, right? And then if we keep doing that indefinitely,

• 2:29:28Speaker 25

  then eventually that noisy image is going to look like just Gaussian noise. And we're not going to be able to see anything from the original image in there.

• 2:29:39Speaker 25

  A very nice property of doing this with Gaussian noise is that if you have a lot of small increments of Gaussian noise, you can add them together into one larger increment of Gaussian noise. And this allows us to simulate this process much more efficiently. And that's kind of a key idea behind diffusion model training, which is that for any time step T in the process, we can write X T as our clean data X zero plus a scaled version of a standard normal variable. And the scaling factor sigma of T is what we're going to call the noise schedule of the diffusion model.

• 2:30:15Speaker 25

  In practice, we make things slightly more complicated, but also slightly easier to work with by not just adding noise at every step, but also slightly rescaling the input before we do that. So we introduce this extra scale factor, alpha T, which is also dependent on the time step. And then another change that we'll make is we won't run this process indefinitely because we don't have time for that. We're going to stop it at some time step capital T, where basically the what the image that we get is basically indistinguishable from Gaussian noise. But now the interesting part is the backward process, right? How do we run this process in reverse? Because that then allows us to do generative modeling.

• 2:31:00Speaker 25

  And again, this is going to be a gradual process where we add these increments, delta. But now these increments are not just random Gaussian noise. Now these increments actually require us to understand something about the data distribution, to know how to gradually remove this noise. And so I like to represent this geometrically. And before I proceed, I do want to express some words of caution. This is a kind of a dangerous game, what I'm going to do here. Because really, this diffusion process is happening in the input space, right? In this case, in the pixel space. And if we think about image data as a vector space, then the vectors that represent the images are very high dimensional, right? Because you have lots of pixels, each pixel has three color channels. These are very high dimensional vectors.

• 2:31:45Speaker 25

  I am going to represent these as two dimensional vectors because two dimensions is all I have on the screen. This is dangerous because, as we know, it can be risky to draw conclusions from low dimensional observations and generalize them to high dimensions. But in this instance, I think it's actually really quite instructive to look at diffusion in this way.

• 2:32:08Speaker 25

  So what does a diffusion model actually do? We start with some data point x naught. We add noise to it with that formula that I showed you before. Some given amount of noise depending on the time step t. And then we end up at a different point in space xt, which is a noisy version of the image. And what the diffusion model is going to do is it's going to try to predict x naught from that xt. So we are in xt and we try to predict where do we need to go in space to get back to x naught. Now, this is a, this is a very difficult task. And the reason this is a difficult task is because, of course, the noise is obscuring some information that was in, in the original image x naught, and we can't really recover that.

• 2:32:49Speaker 25

  So what we end up predicting is not x naught itself, but rather the expectation of x naught given xt, right? We, we're predicting sort of what are all the possible x naught? What are all the possible images that could have given rise to this particular noisy observation at time step t? And this is not a single image, but rather a sort of region of the input space. And what a diffusion model is going to do is predict the direction that we need to move in to get closer to that region of the input space. And effectively, what we're predicting is a centroid of that region. And if you try to visualize that prediction, if you try to visualize that centroid, it looks like a blurry image. And the reason for that is that this is kind of an average across many possible images x naught.

• 2:33:34Speaker 25

  And the noise is kind of obscuring the high frequency content of these images, but not the low frequency content. So the result that we get is a blurry image. So how does the diffusion sampling process proceed? Well, we just predict that direction that we need to move in, and then we take a small step in that direction. And you can kind of compare this to how we optimize neural networks, right? In optimization, we also predict an update direction, but then we only take a small step because really that prediction is only valid locally.

• 2:34:05Speaker 25

  And then one thing that we do here that we typically don't do in neural network optimization is we add a little bit of noise back. And there are theoretical reasons for doing this that I'm not going to go into. But the intuitive reason for why this might be a good idea is that we're doing a sort of two steps forward, one step back thing, which is going to be more robust to any systematic errors in our predictions of this direction. Because of course, we're doing this repeatedly in a loop and errors might accumulate. Not all sampling algorithms do this, of course, but some do. Okay, and then we just repeat the process. So now we're in a new point of space x t minus 1, which looks like a slightly less noisy version of the image. And we just make a new prediction x naught. And as you can see here, that prediction is going to be slightly different, right?

• 2:34:49Speaker 25

  Because now it's pointing to a smaller region of the input space. Because the noise is obscuring less information. So we can kind of make a better guess as to where we need to move. So we have this new prediction. As I said again, this is kind of reflecting a smaller region of space that we need to move towards. And then the process just kind of repeats. So we add a little bit of noise again. We do this a while longer until eventually we reach time step zero. And then what we should end up with is a sample from our data distribution. We are not, probably not going to end up in the original x naught, right? But we are going to end up in a sample from the data distribution.

• 2:35:29Speaker 25

  So that's kind of this geometric overview of the diffusion process. So everything I've explained so far assumes that a diffusion model predicts x naught, right? It predicts the clean input. Now, if you look in the literature, that's typically not what people are doing. Instead, a very common approach is to predict this quantity epsilon from the formula I showed you before, which is basically just a standard Gaussian noise variable. But it turns out that once you have a trained model, You can always convert a prediction for X naught into a prediction of epsilon and vice versa. And that's because of this linear relationship that we have, right? Xt is given, xt is our input, and we know that xt is linearly related to X naught and epsilon.

• 2:36:14Speaker 25

  So if we have one of these quantities, if we predict one of these quantities, then we can convert that to into a prediction for the other. And people have kind of taken this one step further because you don't just have to predict X naught or epsilon. You could actually predict any linear combination of the two. And that gives rise to things like V prediction and the flow matching target, which is epsilon minus X naught.

• 2:36:38Speaker 25

  For the same reason, predicting X naught is also equivalent to predicting xt minus one. And the reason I bring this up is that this is the kind of approach that is taken in the original de-noising diffusion probabilistic models paper, the DDP. The DDPM paper starts from saying, okay, we're, we have this gradual corruption process. We're going to invert it one step at a time. And then the natural thing is to predict the previous time step from the current time step. But as is shown here, actually, because of these linear relations with, by solving a simple linear system, you can show that this is actually equivalent. This is equivalent when you have a trained model. It's not equivalent during training, which is a little bit tricky. So during training, this choice of prediction target actually affects the relative importance

• 2:37:23Speaker 25

  of the noise levels in the aggregated loss across all noise levels. And that is in turn going to affect the perceptual quality of the outputs. So that's why choosing this prediction target is actually important. But once you have a trained model, all these prediction targets are essentially equivalent.

• 2:37:42Speaker 25

  All right, so in summary, how does the diffusion training process proceed? So we, we take each training example X naught, we sample some random time step t, we corrupt X naught to get xt with this formula that I showed you before. We don't have to run this process one step at a time, we can just do it in one go. And then we use our model to make a prediction for x naught or for epsilon, or however we've decided to parameterize the model. And then to train the model, we just minimize the squared prediction error. And this is just the MSC loss that we all know and love. So this is a very stable training objective, which is nice. The reason we use MSC here, the intuitive reason why this is a good idea, is because really what we want to recover is that expectation from before, right?

• 2:38:25Speaker 25

  We can't predict x naught exactly, but we want to recover the expectation of x naught given x t, and that is precisely the minimizer of the mean squared error.

• 2:38:36Speaker 25

  And then for sampling, at each time step t, we can predict x naught or epsilon from x t with our model, and then just take a small step in the predicted direction to partially denoise x t to get x t minus one. And then, as I said, in some algorithms, we add back a little bit of a noise. In some algorithms, we do not.

• 2:38:56Speaker 25

  Okay, so that's kind of the basis of this geometric perspective. Now I want to talk about a few other perspectives that I think are useful, that are maybe less well known. So one thing I'm going to skip is the sort of score matching perspective. This is also linked to what I just explained, but I think that one is pretty well known nowadays. So I want to talk about a few other perspectives. And one is this way of looking at diffusion models as recurrent neural networks. So if we think of the diffusion sampling loop, we're kind of repeatedly applying this denoiser network that we've trained in sequence. And if you unroll that computational graph, that actually just looks like a much deeper neural network.

• 2:39:38Speaker 25

  And then you could ask, why don't we just train that with backprop like we usually do? Right? And the answer is, of course, it's very, very deep. It's often tens of thousands of layers. You know, if your base diffusion denoiser model is 100 layers and you have 100 time steps, then this is going to be a 10,000-layer neural network. So you can train that with backprop through time. People have done that. That's what you get is actually called a continuous normalizing flow. But you can do another thing, which is to train this with score matching, and then you don't have to backprop through this loop. You can only, you only have to backprop to one step of the denoising. So this gives you a way to look at diffusion models as kind of a deeper recurrent neural network that was trained without backprop through time.

• 2:40:19Speaker 25

  Kind of a hack to train deeper networks, if you will.

• 2:40:25Speaker 25

  This is a perspective that I really like. So one question that I get a lot is like, why do diffusion models actually work so well for image and video? Why did they kind of come in and take over essentially all generative modeling for all modalities except language? And so for images, there's this interesting spectral analysis that we can do that sheds some light on this. So we can take, we can calculate the spectrum of an image. We can kind of summarize that in one dimension. And if you plot this spectrum on a log log plot, what you get is a power law, that you get a straight line and that reflects that there's a kind of power law going on. So the amplitude, or actually rather the power of a particular frequency in the image

• 2:41:07Speaker 25

  is proportional to that frequency raised to some negative power. Usually it's like around minus two and this seems to be some sort of law of nature, right? So you get this negatively sloping line for natural images. If you do the same thing for Gaussian noise, you calculate the spectrum, what you should get is a horizontal line because in Gaussian noise, all frequencies are present in equal measure, right? Now, the interesting thing that happens is when you superimpose these, because that's what we do in diffusion models, right? We add noise to images and you add them together and then you look at the spectrum and you get this hinge shape that you see on the third plot there. And if I increase the noise level, so if I increase the amplitude of the noise, then that hinge sort of shifts position.

• 2:41:51Speaker 25

  And what this is going to do essentially is it's going to obscure more and more of the high frequencies in the signal, but the low frequencies, because they're more powerful, they're kind of kind of jut out above this noise floor and so they're going to be preserved. And so based on this interpretation, I think it is fair to say that diffusion is kind of an approximation of spectral auto regression, right? We're generating images from low frequencies to high frequencies. And so this is true for images, this is true for video, also for audio follows this sort of power law, but obviously not necessarily true for other modalities such as language.

• 2:42:29Speaker 25

  This is not an idea I came up with, so I actually got inspired by this paper from Severy Risanen and his colleagues on generative modeling with inverse heat dissipation, where they do this kind of spectral analysis. And it's really important because the different noise levels actually correspond to different spatial frequencies in the image, in a way. And so that means that when we're reweighting, rebalancing these different noise levels in our training objective, what we're actually doing is we're saying which spatial frequencies matter to us, like which spatial frequencies do we want the model to really understand well? And this actually means that the diffusion, the diffusion loss is actually kind of perceptual loss, right?

• 2:43:09Speaker 25

  Because we're kind of emphasizing the frequencies that the human visual system is sensitive to, and we're de-emphasizing the ones that we are less sensitive to. And I think that's one of the big reasons why diffusion models for images took off so rapidly, even if we didn't necessarily understand this at the time.

• 2:43:29Speaker 25

  All right, so one thing I also want to do a little bit is contrast autoregression and diffusion, because these are sort of the main generative modeling paradigms that are popular today. So we know, we all know probably what autoregression is, you kind of turn everything into a 1D sequence, generate that sequence once at a time. With diffusion, we use this noisy process, this corruption process. So these are just two different ways to do generative modeling, but they're both iterative. They're both using many network invocations to do generation. So they both use this kind of divide and conquer approach to generative modeling. And so for video specifically, there's kind of a continuum almost in some of these choices.

• 2:44:09Speaker 25

  So we could just model video autoregressively, and that would require taking the spatial-temporal volume and dividing it up into tokens, which would be like three-dimensional patches or voxels, and just choosing some order in which to predict these, right? Because we need to turn it into a sequence. Then on the other end of the spectrum, we could just take this entire cube, this entire volume, and just model that with diffusion. But there's kind of a hybrid approach that seems to make a lot of sense for video specifically, which is to treat the temporal dimension ultra-aggressively and you diffusion over the spatial dimensions. That's what I'm showing in the middle here. And all of these approaches have sort of their own advantages and disadvantages. So the ultra-aggressive approach is nice because it would make treating multimodal

• 2:44:54Speaker 25

  models very easy. So if we want to integrate this with large language models, right now, that seems to be the way to go. But of course, these sequences will get very long, and so that means we run the risk of getting problems with error accumulation if we generate very long videos. Then on the other hand, with diffusion, we have kind of robustness against this error accumulation, and we have powerful methods for accelerating sampling, for example, through distillation. I believe also that guidance, while it's not exclusive to diffusion, you can apply it to ultra-aggressive models as well. It does seem to be, at least to me, it does seem to be more effective in the diffusion setting. But of course, working with these very large spatial-temporal cubes, sort of having to

• 2:45:38Speaker 25

  generate this in one go can be quite unwieldy and can create quite a lot of memory pressure. So the hybrid approach could be seen in some sense as best of both worlds, but it also has some advantages and disadvantages. For example, if you want to do distillation, then this hybrid approach where you do temporal ultra-aggression might actually cause issues with error accumulation again. But of course, one nice aspect of the hybrid approach is that we can reuse a lot of stuff that we've done for images, right? Because essentially, this is just an image conditional image generation model, if you will. One more general trend that I want to talk about in generative modeling of perceptual signals is this sort of moving away from measuring likelihood in the input space.

• 2:46:23Speaker 25

  So back in the day when I started working on generative modeling, we had models like Pixel CNN and WaveNet. These were just likelihood-based models in the input space. But they didn't really scale very well to larger inputs because likelihood is actually a very poor perceptual metric. And it's precisely because it's putting way too much emphasis on these high frequencies that are perceptually less relevant. Of course, it works very well for language, as we know. But so the general trend for perceptual data, for audiovisual data, has been that for autoregressive models, you started measuring likelihood not in input space, but in some latent space, right? We first learn latents to kind of make abstraction of a lot of this entropy that is not actually perceptually relevant. Like, you know, the individual blades of grass

• 2:47:07Speaker 25

  in a grassy texture, for example, don't need to be modeled by a likelihood-based model. We just need to be able to kind of paint with a grassy texture, essentially. And then the same thing is kind of implicitly happening in diffusion models in a continuous way, because by reweighting the noise levels, we are kind of also implicitly downweighting these less important frequencies. But of course, with diffusion, we're also nowadays often using a latent space to kind of amplify this effect. And I want to talk a little bit more about that. Why that makes sense? Why that is a good idea? So visual perception, I think, works differently at fine scales and large scales. At very

• 2:47:47Speaker 25

  fine-grained scales, our perception of texture kind of makes abstraction of all these little details. We don't, you know, I can take an image with a grassy, you know, let's say, a dog playing in a field, you know, like sky above, grass below. I can take an image and modify it in Photoshop by shifting that grassy texture one pixel to the left and show it to you again, and you won't be able to see what happened, right? It's too subtle. So that perception is kind of making abstraction of these fine-grained details. And it's not actually necessary to model all these possible variations. We just need to be able to generate one good one, right? And that's precisely what adversarial models give you, right? They don't really bother modeling all the modes of the distribution, but they can give

• 2:48:32Speaker 25

  you a few good ones. So this is just a really good match for fine-grained perception. Whereas at the larger scale, we care a lot more about covering all the possible modes. And so there, it makes sense to use something that's closer to a likelihood-based model or a diffusion-based model.

• 2:48:50Speaker 25

  Right, so the next thing I want to do is talk about diffusion guidance, which is, I call it a cheat code for diffusion models, because it allows them to perform way above their pay grade in a sense.

• 2:49:03Speaker 25

  Guidance allows us to trade off sample quality for diversity. And it just generally makes diffusion models work a lot better. And so I want to revisit this geometric diagram that I was talking about before. So again, we have our clean input sample from the data distribution X naught, and then a noisy version of it at some time step T in the top right corner.

• 2:49:29Speaker 25

  And as before, our diffusion model will predict which direction we need to move in in the input space to move towards the data distribution. But now we're going to do something slightly different. We're going to do classifier guidance, which means we're going to take a classifier that is robust to noisy inputs, and we're going to ask it to classify this noisy image. And we're going to take the gradient of these lodges that we get with respect to the input. And what this is going to give us is a direction in input space that we should move in to make this image be more likely to be classified as that particular class, right? So it's kind of amplifying the aspects of the image that make it adhere to that particular class. And this gives us a different direction in input space. And instead of following the direction we predicted with our diffusion model,

• 2:50:14Speaker 25

  we can actually superimpose these directions, just add them together, and then move in that direction instead. And I want to kind of show you the underlying Bayesian perspective on this as well, which you can get simply by taking this formula for classifier guidance, which is expressed in terms of score functions, like the gradient of the log likelihood. You can actually just undo this gradient operation and this log operation to see what happens in terms of probability. And that's what I'm showing on this slide. So actually, what we're doing is we're taking an unconditional base model, unconditional diffusion model, adding this classifier, PFC given X, and then combining those two to get a conditional model, right? So we can actually turn an unconditional model conditional after training.

• 2:51:01Speaker 25

  But the real power of classifier guidance is unlocked when we introduce this scaling factor, which is called the guidance scale. Right. So we're going to scale this gradient direction that we get from the classifier by some constant gamma. And what this is going to do is just say like, make it look like a rabbit, like really make this image look like a rabbit. I want to get all the characteristics in that image to make it look like a rabbit. So our new update direction is going to be this one. And so we're going to end up in a different point in space. That is kind of following this new direction.

• 2:51:39Speaker 25

  And again, if we kind of look at the Bayesian perspective here by undoing this gradient operation and this log operation. What's happened here is the classifier probability is now raised to this power gamma. And what does it mean when we raise a probability distribution to a power and sort of renormalize it? That's tuning the temperature, right? That's something we do with autorecent models all the time. We're actually just tuning the temperature. But what's interesting about guidance is that the temperature tuning is happening in the output space of a classifier and not in the input space of the generative model. And I personally, I think that's why it's so powerful because we're able to tune temperatures at a kind of high level of abstraction. We're kind of sharpening this classifier distribution.

• 2:52:22Speaker 25

  So next, let's look at the classifier free version of guidance. So I'm kind of doing the same thing here again, like looking at our diffusion model prediction. But now we're actually going to make two predictions. We're going to make an unconditional one and a conditional one. And these are going to be slightly different because obviously the conditioning signal gives us a little bit of information about where in space we might need to move to draw samples from the distribution. The way we can achieve this in practice is by training a conditional generative model and then maybe dropping out the conditioning signal 10% of the time. Right. And that gives us a model that can operate in both conditional and unconditional modes. So we have these two predictions and we can look at the difference vector between the two, which I've called Delta here.

• 2:53:03Speaker 25

  And this difference vector is a direction that we can move in to make samples look more like they belong to this class C. Right? And again, that we can do the same thing that we did in classifier guidance, which is to amplify this difference by some scale factor gamma.

• 2:53:18Speaker 25

  Just to allow us to really hone in on the characteristics of this class C. And then this gives us a new direction which we should move in during diffusion sampling.

• 2:53:31Speaker 25

  And again, as before, the sampling algorithm kind of proceeds as before, so we might optionally add some noise here. All right, now let's look at the Bayesian perspective again.

• 2:53:41Speaker 25

  This is very powerful because you kind of applied Bayes rule twice. And effectively, this vector Delta corresponds to a Bayesian classifier. Right? So this classifier probability that we had before is now replaced by this ratio of P of X given C and P of X. But again, raised to this power gamma, so we're again tuning this temperature. And this is effectively what classifier free guidance is.

• 2:54:08Speaker 25

  And this is a lot less prone to sort of adversarial directions in the input space than classifier guidance would be.

• 2:54:15Speaker 25

  I have these examples, they're quite old in the meantime, so these are from the Glide paper, which was sort of one of the first large scale text to image models from OpenAI. But I really like these because they kind of show what the model looks like without guidance and with guidance, which is rare in modern papers. In modern papers, we only see samples with guidance. But here you can kind of really see just how much of an impact this has. And you can also see the impact, you can see the trade off between diversity and quality, right? You can see the images come out looking much less diverse, but the quality is clearly improving.

• 2:54:49Speaker 25

  Another example from the same paper here with a slightly different prompt. Again, sort of reducing the diversity in favor of making the images just look more, a lot more, a lot nicer overall. And I think nowadays, a lot of these state of the art models that we're seeing, if you were to sample from them without guidance, I think you would be surprised at just how bad they are. Right? These models are really relying on guidance to produce these incredible results that we've been seeing.

• 2:55:15Speaker 25

  So if there's anything you remember for this, the main thing I want you to remember is that classifier guidance is just two applications of Bayes' rule.

• 2:55:24Speaker 25

  Or is it? There's an interesting recent paper by the Nvidia group from Finland where they kind of called this into question a little bit and give some other intuitions about why guidance might actually be working. I won't go into this here, but it's a very good paper. I recommend taking a look at it. It came out last month, so it's very recent.

• 2:55:44Speaker 25

  All right, and then to wrap up my talk, I'm just going to briefly talk a little bit about Imagine 3 and VO, which are the text-to-image and text-to-video models that we've been working on recently. Both were announced at Google I/O in May.

• 2:56:00Speaker 25

  Imagine 3 should be available shortly. VO is obviously a sort of more in real-time model that might take a bit longer, but hopefully you'll be able to play with Imagine 3 soon. And I just have a few samples here from this model. So this is a latent diffusion model, just kind of a change from our previous family of models.

• 2:56:23Speaker 25

  And you can kind of see that, yeah, it does a pretty good job at fine-grained detail, large-scale structure. It's a very nice text-to-image model.

• 2:56:39Speaker 25

  All of these samples are on the DeepMind website on the relevant blog post, and hopefully we'll be able to share some more details about the inner workings soon as well.

• 2:56:51Speaker 25

  And then finally, I also want to talk a little bit about VO, which is our text-to-video model. This is kind of probably looking a lot like what you would expect, so it's again a latent diffusion model. We have a text encoder that includes text prompt input, an optional encoder to condition it on frames with image input. And then the diffusion operates in a latent space, and then we have a decoder that turns this back into pixels at resolutions up to 1080p and relatively long lengths.

• 2:57:28Speaker 25

  And then I have your, I don't know if this is going to play, but yeah, this is kind of a showreel of samples which you may have seen before. Okay, that is supposed to move because it's a video. Okay, here we go.

• 2:57:40Speaker 25

  So this is just kind of a showreel of some samples of the VO model. I don't know if the the quality is quite visible, is quite coming across here, but it's producing a high quality video at 1080p.

• 2:58:01Speaker 25

  All right. So to wrap up, one thing I want to highlight is that pretty much everything I've talked about today is on my blog. So I have a whole series of blog posts on diffusion models and on generative models in general, where I kind of try to build intuition, right? So it's not, it's not necessarily about theory and being mathematically correct. It's about building intuition for these models and how they actually work. And so most of the content from from the slides here is kind of spread across a few of these different blog posts. Okay. That's it for me. So link to my blog, also to my Twitter account and my email address. If you have any comments or suggestions or questions

• 2:58:42Speaker 25

  after the talk, feel free to contact me and I'm happy to take questions now as well. Thank you. I'm curious to hear about where you see the capabilities of these models going.

• 2:58:54Speaker 25

  That's a kind of a vague question.

• 2:58:59Speaker 25

  Yeah, I mean, I so I think bigger and better. I yeah, I think we're kind of, we're kind of early, like I I kind of compare it to what's happened in the in the in language modeling, where we're kind of a bit further along in the scaling process. I would say on the on the video side and on the image side as well, I think we're quite early. So I would expect more big leaps.

• 2:59:23Speaker 04

  I have a question about latent diffusion models. I haven't seen them written down mathematically. Would you mind giving us some intuition? So if the input is fixed, like we're doing diffusion models on X, it makes sense, you can add noise, however much noise you want, you can try to reverse it. For latent, it means you're training a neural network, and from the latent values, you're doing the same process as the network itself is shifting, is training?

• 2:59:48Speaker 25

  So usually, it's a two-stage process. So we're first going to learn some latent space that essentially compresses the input, right? Because one of the issues with generating very large images, very large videos, is that it just takes up a lot of memory. And one of the key advantages of latent diffusion is that you can actually compress a lot of the redundancy out of this and still get a representation that's sort of learnable, right? This is also how it differs from sort of standard compression. You know, you have standard compression algorithms like JPEG and H.264 and whatever. They're really just focused on making things as small as possible. Here, we're kind of trying to do, to control a trade-off between how much we can compress while maintaining output quality

• 3:00:33Speaker 25

  and also how learnable the resulting representation is. Because if you compress too aggressively, that might get difficult. Like if you were to do entropy coding on the latent space or something like that, that might actually make learning more difficult. So it's kind of an interesting twist on the compression problem because you have this trade-off. But it's generally a two-stage process. So you learn the latent space first, and then you freeze that, and then you just train a diffusion model as you always would, except that you're just extracting this feature representation and operating on that.

• 3:01:08Speaker 12

  Thank you. A great talk.

• 3:01:09Speaker 25

  Thanks.

• 3:01:10Speaker 12

  I'd love to hear your thoughts on like current metrics, things that are missing, how we can better evaluate this, particularly for me, video generation, but in general diffusion models.

• 3:01:20Speaker 25

  I have mainly complaints and not many suggestions. It's tough, right? We don't have a lot of great metrics. We do a lot of eyeballing.

• 3:01:33Speaker 25

  For image as well, but especially also for video. It's trickier for image, sorry, it's trickier for video than for image. Because for image is kind of easy to generate, say, 200 samples, put them in a grid, just take a quick glance at them and have a rough idea of what your model is doing. For video, it's a lot harder, because everything is moving, right? So it's much trickier to kind of glance at things. You kind of have to look more at individual samples. And then for audio, it's actually completely impossible, right? Because you just have to listen to them one by one. And this is a very persistent problem that, yeah, I haven't seen any great solutions so far.

• 3:02:11Speaker 25

  Yeah, we use the classical metrics, FID, FED, but we also know they are flawed in various ways and that sometimes we can't trust them. But they're at least useful as canaries, right? They can kind of tell us when something is seriously wrong, at least, so that's helpful. But yeah, definitely a very fruitful space to work in if you want to make an impact, is to figure out how we evaluate these things, especially computationally, without involving humans in the loop. Thanks.

• 3:02:43Speaker 06

  I would like to ask if you think that predicting human evaluation with a model is promising as a direction or not for evaluating these models?

• 3:02:57Speaker 25

  Quite possibly, yeah. I guess it kind of depends on what your human evaluation data looks like. But I think that's a promising direction.

• 3:03:05Speaker 06

  Did you try to scale, to train a model on a lot of human evaluation and see?

• 3:03:11Speaker 25

  And then sort of use it as a proxy, as a reward model in a sense. Yeah, I would say that's a valuable direction to move in. I have one concern with that, which is that, you know, every metric, when it becomes a target, eventually ceases to be a good metric, right? So it'd be very interesting to kind of see how that applies there. And I think we should be careful about that.

• 3:03:32Speaker 06

  Thank you.

• 3:03:33Speaker 26

  Hi, we have seen that some of the diffusion models always produce, or often produce data that's very close to its training data. When you ask it to produce something, do you have any ideas on how they might get more creative or general and further away from the training data?

• 3:03:53Speaker 25

  I think that's probably the easiest way to solve that is to get more data. Like the, you know, if you have an order of magnitude more data than something like that is an order of magnitude less likely to happen. But I think, I think, so I don't deny that this is a problem, but I think we should also, you know, when, when diffusion models kind of rose to prominence, came onto the scene. I think one of the very impressive things was this kind of combinatorial generalization that they exhibit, right? So I think to some extent, there's all, there's already a lot of sort of creativity on the part of these models and sort of combining things in ways that they have, that they don't exist in the, in the training set.

• 3:04:37Speaker 25

  And I would expect with more data that that ability would improve.

• 3:04:42Speaker 27

  Hey, man. Thanks for the great talk and joining us.

• 3:04:47Speaker 27

  With some of the video models that have been released, if for example, you have like, say, water and waves sort of flowing, you can kind of, as a human watching it, see that the laws of physics aren't sort of strictly abided to in the same way that you'd see in real life. What do you think are some promising directions for sort of ensuring future video diffusion models kind of more closely adhere to the physical laws of nature in that kind of sense?

• 3:05:16Speaker 25

  Scale is one. I think a lot of this sort of behavior is emergent and with more data and, and more capacity, the model will learn to do this. But maybe in the shorter term, there's something we can already do to, to improve this, maybe by curating the data, maybe by building in some physical priors into the model, although we do have to heed the bitter lesson here where often it turns out that it's better to just let it learn and, and not try to meddle with that too much, but yeah.

• 3:05:47Speaker 00

  Our second Deep Mind speaker in this section is Ben Poole, who works on inferring 3D structure with 2D priors, which you can see as a key component in upgrading something like Genie 1, which is 2D, to Genie 2, which is 3D. He also introduces the Neural Radiance Field Concept, or NRF, which is now incredibly popular for 3D environment simulation and of course has implications for synthetic data in generative video. Ben combined NERFs with score distillation from diffusion to create Dream Fusion and Re-Confusion. Let's tune into his invited talk at the Structured Probabilistic Inference and Generative Modeling Workshop led by Yoshua Bengio.

• 3:06:30Speaker 28

  Yeah, thank you everyone for being here so early. Thank you so much to the workshop organizers for the invitation to speak. And today I'm going to be sharing some of our work on inferring 3D structure with 2D priors. And so ICML has been really fun, but people keep asking me why I'm working on 3D generation. We've seen some amazing progress in video generative models. And as we scaled up the data and the compute, we often see that the quality improves. And then if you also look at some of the 3D consistency within these video models, they've also improved as we've scaled things up. But you know, the way that we consume content isn't always just staring at a flat screen. We have amazing new AR and VR mixed reality headsets. And the type of content that we want to consume is often interactive. You see some really creative, interesting scene

• 3:07:14Speaker 28

  and you'd love to move around in it and see it from other angles. And it's not just moving around in it. In VR headsets, oftentimes the most fun way of exploring worlds is interacting with them, be it in video games or exploring on a mobile device. And it's unfortunately really challenging to create this kind of 3D content. 3D modeling is really hard. I remember working on some of this in middle school and being immensely frustrated at the inability to create the seemingly simplest objects. And even once you have these 3D models, how do you interact with them and add them to worlds, light them, rig them? It's all an extremely challenging and time-consuming problem. And it's not just about creating things. I think something that I find really frustrating is we've seen the amazing power of AI in a number of different domains. But I'm seeing all of you sitting in front of me. And as a human, I feel like I have this really innate sense

• 3:07:59Speaker 28

  of the 3D structure around me, where objects are. I know my water bottle is here and I can grab it. But it's really challenging for AI systems to have this kind of spatial intelligence. So if we can make more progress in building 3D priors and understanding the 3D world, I think it could really influence the direction that things are going in robotics as well.

• 3:08:16Speaker 28

  Luckily, we've seen amazing progress as well in 3D reconstruction. So here's an example from ZipNURF, which is a powerful method based off of NURF. And you can capture an entire house and turn it into a 3D model that you can move around in and interact with. And the quality of this and the photorealism of this often exceeds even our best video models today.

• 3:08:35Speaker 28

  And how do these methods work? The idea is that we have the space in front of this and we can parameterize it as a 3D volume. And at every point in this X, Y, Z space, we can use a neural network that maps from a point in space to a density and a color. And there's all sorts of different 3D representations that people are exploring these days, but the key idea is that you have a differentiable mapping from somewhere in space to a color or the ability to query different points along a ray.

• 3:08:59Speaker 28

  And the way that we train the parameters of these neural networks that are representing the 3D world is that we can cast a ray into the scene from a known camera and we can evaluate a bunch of points along that ray using our neural network. This gives us a color and density along the ray that we can accumulate to get an RGB color. And the way that we train these neural networks for 3D modeling is that we have gone and collected a bunch of images and we can see how well does the image match the prediction of this neural network.

• 3:09:27Speaker 28

  And I think what people don't realize about NURF is how data hungry they are. So if I want to capture a Lego bulldozer on a table, I can't just take one picture. I have to go out and collect a huge handful of pictures that surround the object and view it from almost all viewpoints. Their ability to generalize to unseen regions is basically nothing. It's really interpolating between known viewpoints. And once you do this, you can get high-quality 3D reconstructions that represent the color and also learn to some extent the 3D geometry depicted by the depth here on the right.

• 3:09:57Speaker 28

  So what happens if I haven't had my coffee this morning and I wake up a little bit early and I only take three photos, but I'm really curious to see what the scene might look like in 3D? Well, here's an example of the state-of-the-art 3D reconstruction methods on a three-view reconstruction. And what you can see is it matches really well at the observed images, but as I deviate away from these images, we get really inaccurate predictions of what the world might look like. And if you think about building a robot that's going to go and grab that Lego bulldozer, that depth map and the 3D geometry looks hugely inaccurate. It's not going to be useful for any of these tasks.

• 3:10:27Speaker 28

  And in general, I think we're at this structured probabilistic modeling workshop. What's the problem that we're trying to solve? Well, we don't have access to the 3D world, or even a lot of ground truth data in the 3D world. We just have the shadows of the 3D world. We have the projections in our eyes, or we take out our camera and just see a two-dimensional image. But we would like to understand what that 3D world is so that we can reason over it. So we're really trying to solve this inference problems of, okay, what's the distribution over what could be there in the 3D world, given some set of observations? And there's often this kind of spectrum that goes from reconstruction, where you've collected a lot of data, you know exactly what should be there, and you want to recreate it in a digital world. So maybe something that's a little looser. Maybe I have a picture, and I just want to hallucinate plausible 3D content.

• 3:11:09Speaker 28

  For what is a 3D scene that could be consistent just with that image? Or maybe I don't have a bulldozer in front of me, but I want to create it for my game or visualize it. And maybe I just want to describe it with text. So we have all these different ways of thinking about observations that we want to condition on, but there's a shared common goal. How do we create this 3D structure, given these partial observations? And we've done some work across the spectrum. So we started off working on text to 3D with Dream Fusion, and then worked on few-v reconstruction with Recon Fusion. And then more recently, we have some work Cat3D that enables us to do everything from text to single image to few-v reconstruction for 3D creation. I'll talk a little bit about each of these projects today.

• 3:11:49Speaker 28

  Okay, so why is 3D hard? I think I got into 3D mostly, not necessarily because I cared about 3D and understanding the 3D world, but it was more, this was a problem that didn't feel like data could just solve. Across the board on, you know, language generation, text generation, and image generation, we've seen that there's an incredible amount of progress just by collecting big data sets. But as we saw before, it's really hard to acquire ground-truth 3D models in the world. It's really expensive, and involves a lot of human effort. But let's say we do this, and we collect a big data set. Now what? How do we represent it? We have all these different 3D representations. We have splats, voxel grids, nurse. You have to pick one of those. And then, once you pick one of those, you have to design an architecture that can scale up as you increase those data sets. But let's say you do this. Here's an example, a bit of an old example these days. Can people hear me okay?

• 3:12:32Speaker 28

  I'm realizing that, okay, great. So, you know, if you have a decent-sized 3D data set, and you train a model on that data set, you can get okay 3D models. But most of our 3D models are just of isolated objects, and it's hard to get the realism to be as high as, for example, the image samples that we get out of state-of-the-art text-image models.

• 3:12:54Speaker 28

  And I think the real problem here is that there's this huge gap. I've been presenting this for a while and I think it is still very true. There's a huge gap between the 3D data that we have access to and the visual world. And I think a lot of that is driven by, everyone here has mobile phones in their pockets with cameras, but not all those cameras have depth sensors. And even if they have depth sensors, when you take a photo, you don't often take a video of the object that encircles it and captures all the different ways that you can imagine viewing it. And so the bet that we made was that, okay, maybe we could find ways, instead of building explicit priors in the 3D space, could we build priors in 2D? And if we have these priors in 2D, now we need to solve a more complicated problem because we can't just do inference over the 3D space if we don't have a prior there. We need to be creative for ways of thinking about using these two-dimensional priors for 3D generation.

• 3:13:40Speaker 28

  And the general inductive bias, or the way that we're going to hack 2D priors into 3D, is that we're going to say, well, what is a good 3D model of the world? As a human, I often don't have the ability of knowing that the 3D world around me is really accurate and precise, but I can view it from different angles. And so the idea is that we're going to take this 3D model that we're trying to learn or do inference over, and we're going to render it from a bunch of novel viewpoints. And what does it mean for that 3D model to be a good 3D model? Well, it just has to look good. And how do we measure how well it looks good? Well, we're going to look at the renderings and we're going to use a 2D prior to score this amount of goodness. And so here we have like a bear playing a guitar. So you might imagine, okay, if maybe from one view it looks good, that might be insufficient for the 3D model. But if every way that I look at that 3D model, it looks good, then maybe it's a good 3D model of a bear.

• 3:14:27Speaker 28

  This opens up a number of questions and problems and research directions for you to solve. What 2D prior, conditioned on what information? How do we actually measure goodness? I think there's been a lot of tremendous work in probabilistic modeling on, what does it mean for an image to look good? And I think we still don't have a really great sense of what that metric is or how to optimize it across all different kinds of probabilistic models. Another big problem is which views? You know, some objects, I can put a camera over here, but then depending on where things are in the scene, it might be really challenging to think about, where do I want to evaluate how good this model is? I don't want to put the camera inside of the object, for example. And then also which 3D representation? You know, nowadays we have a plethora of choices for, you know, you could use flats, you could use nerfs, you could use all these different things. And the 3D representation you use might change on the setting that you care about.

• 3:15:14Speaker 28

  So who here doesn't know about diffusion models? Oh wow, that's great. So the general gist of a diffusion model is that's a way of modeling high dimensional continuous distributions. And we pair a simple destructive process where we take that data and we add more and more noise to it. And eventually, we've degraded all the structure that's present in the initial image in this case. And then what we're learning to do is how do we reverse this process and slowly introduce more structure back into the data. And diffusion models are great if what you care about is sampling. So you've trained on this big dataset, for example, 2D image, and you want to sample 2D images. But in 3D, we don't actually care about sampling 2D images. What we really want to do is back out and infer some kind of 3D structure. And one approach for this is you can think about, well, we're building parameterized

• 3:15:59Speaker 28

  images. There's some parameters of the NERF or a generative model. And we can use those to create an image, and then we'd like to evaluate how good is that image? And so what we're missing here is a loss function that we can use to score these generations or renderings. And if we do have that loss function and it's differentiable, then we can back propagate into the image and then back from the image to the parameters of that generative model. And the idea that we proposed in Dream Fusion was built around this idea of probability density distillation, and so we called it score distillation sampling. And I guess another way of thinking about diffusion models is that they learn a sequence of marginal distributions that start from a clean data point and map to noisier and noisier data distributions. And these noisier distributions are often simpler. They're smoother than the initial data density.

• 3:16:44Speaker 28

  And what we want to do is maybe pick out a single mode of this complicated data distribution. So here you can see P of X is the complex data density defined by the diffusion model. And we just want to infer one mode of that distribution. And the hope is maybe that the kind of the mode might be a good looking sample. And we do this not just at one noise level in the diffusion model, we can average it across all these different modes. And this allows us to learn a loss function that is applicable to any differentiable image representation. And here, what's nice, while we don't have explicit access to the marginal distribution and diffusion models, we do have access to the gradient of its log density, which is all that we need to evaluate this loss function.

• 3:17:25Speaker 28

  So in Dream Fusion, we combined the score distillation loss with a 3D differential 3D representation from NERFs. So if you want to peacock on a surfboard, you start with a randomly ashenali's NERF, and you can iteratively optimize with the score distillation loss. And over time, that builds up a 3D model that looks good from all these novel viewpoints. And at the end of the day, after optimizing this 3D model, you get out, hopefully, a high-quality 3D asset that you can use in different ways. And what's cool is we didn't have to use any 3D data to create these text-to-3D generations. And on top of that, we maybe don't have any 3D data at all for a lot of these categories. So if you collected a 3D dataset, almost all of these kind of text-to-3D generations might be out of distribution. But the more I played with these text-to-3D systems, the more it's like gambling,

• 3:18:09Speaker 28

  where you come up with a text prompt, you hit go, you wait a while, and the results stink. And then you do it again and again and again. And it's not a very fun form of control, and it doesn't allow you to ground these 3D generations in the world. Especially if I take a photo, I don't want to take that photo, describe it with text, and feed it to a text-image model. I would like a better way of grounding it in real scene content. So in some follow-up work on Recon Fusion, we tried to generalize this method from conditioning on text to conditioning on images. And if we go back to our bulldozer example, here's the original 3D reconstruction of this bulldozer model from three images. And if we apply a method that uses a generative prior at these novel views,

• 3:18:49Speaker 28

  you can see that we can accurately recover novel views and decent geometry from just three input images. So how does this work? It's very similar to the earlier work, but we're going to augment a 3D reconstruction pipeline with a model that's not conditioned on text to describe what this novel view should look like, but it's conditioned on images. And so what should a novel view of a scene look like? Well, we often have one or a few images of what that scene is. And so what that novel view looks like should be very informed by what the existing content is and the other 2D pictures that I captured.

• 3:19:22Speaker 28

  So like, what might this image look like? And the idea was to train a new kind of diffusion model that was conditioned on the set of input images and their camera poses. And then, given some novel target pose, we want to predict this novel view. So it's still an image diffusion model. It's only producing one novel view. What should this look like from over here? But now you condition on one or many different inputs that you have in the scene. So it can take our lazy three captures of some image and then turn it into a 3D model. And the architecture that we used to condition on the set of input views was Pixel Nerf, which is an image-based rendering method. And this was inspired by all your work like Nerf Diff and GMBS. And as input, you have a set of input images and their camera poses.

• 3:20:04Speaker 28

  You pass it through the Pixel Nerf to get some rendered features at the target camera pose. And then you combine this as input into a typical text-to-image latent diffusion model, where we replace the text features with now clip embeddings of these different image inputs.

• 3:20:20Speaker 28

  And now, unfortunately, unlike the existing work that we did before, Dream Fusion text-to-image, here we need data that doesn't just have text and image annotations. We need sets of pictures and their camera poses. So we're way more restricted in terms of the kinds of data sets that we can apply in this novel view synthesis setting. Here, we trained on a combination of RealEstate 10K to get some real-world scenes, CO3D and MV ImageNet, which are often orbits around objects but in context, and then a bunch of synthetic renders from 3D models from Optiverse. And if you apply these methods to real-world scenes, you can see that you can get out decent novel view synthesis predictions. But one issue is that these images are predicted independently. We aren't modeling the correlations between views that happen when you have one 3D model.

• 3:21:05Speaker 28

  And so we had to design a procedure that could take these inconsistent 2D predictions and turn them into one consistent 3D model.

• 3:21:14Speaker 28

  So here on the top, you can see the results of the 3D reconstruction and the bottom of the samples. And so this is similar to Dream Fusion where we have these, we don't know exactly what the novel view should look like. So we have to generate a bunch of samples or use an optimization procedure to resolve these difficulties. I think the big problem with all these iterative optimization-based methods for 3D generation is that they're really slow. Dream Fusion takes around half an hour to create a 3D asset. Recon Fusion, it was around an hour. And what could you do in that hour? You probably could have just gone out and taken more pictures of the thing that you were trying to capture. So it doesn't seem like a great practical solution for improving the efficiency and our ability to capture the 3D world. And if you're a robot, you don't want to wait an hour before you move your hand to reconstruct the 3D system.

• 3:21:58Speaker 28

  Another thing that we didn't actually show in Recon Fusion was what happens if I put a single image into the system? And one of the issues with the Recon Fusion work was in areas of uncertainty, where you didn't know what should be in the scene. You often end up with blurring. And this is because those independent image observations would often conflict. And while we use these optimization procedures to resolve them, you often fight back, you're kind of fighting against this aspect of averaging out all these different ideas for what it might look like from this novel view. So here are like some single image results for a hydrant and a bench.

• 3:22:29Speaker 28

  So in our next work, Cat 3D for Create Anything in 3D, the hope was that we could address these problems of hallucinating novel content effectively. And the main idea behind this method was to address this problem of independence. We know that if I have a 3D model, or if we have a video of something, those frames are correlated. And so we would like to model these correlations and not just kind of resolve them post hoc in our 3D extraction procedure. So here are some example samples from Recon Fusion, where we have three input images and then we have these independent output images. And we can resolve them, but it's a very slow process. And the main idea of this work was building on the amazing success of video diffusion models for jointly modeling the correlation between multiple images.

• 3:23:12Speaker 28

  The model that we trained took a set of observed USS inputs. You could have a single image or a set of images. You also have to have their camera poses. And we encode both the image into a latent space and the cameras using a ray representation, which is kind of representing which kind of the corners of the image that you're generating. And then we also have a set of targets. We have where we would like to create outputs. And it's not just one place. We want to create a whole set of image outputs, and we want those outputs to be correlated, such that they could be realized from a single 3D model. So we have the set of observed and unobserved views. We also add a mask to indicate to the video model which of these are observed, which of these are unobserved. And then we get out not just one view, but a whole set of views that we can decode back to an image.

• 3:23:56Speaker 28

  And if we train this model on the same exact dataset that we trained Reconfusion on, we can see that this model is successful at learning correlations across images, and the resulting samples that we get are already pretty consistent. But they're not perfectly consistent, and they don't allow that kind of interactivity that we might want from a real 3D model. So what did we do? All we did is we took a single input image or a set of input images. We generate samples using this multi-view latent diffusion model that gives us a generated set of views. And then we just feed that to a 3D reconstruction pipeline. And there's some additional tricks that you need, like using a robust loss, which allows for some reconciling of these different details across different views. But this whole procedure now just takes a minute instead of an hour.

• 3:24:38Speaker 28

  And here are some examples comparing the Reconfusion results to the Cat3D results. All of these are conditioned on three images, and you can see, not only is it faster, but also if you look especially at the backgrounds, you get way higher quality hallucinations in regions where you actually had uncertainty. What's cool is this works on images and single images, unlike the Reconfusion work. So here's a picture of how we have a very cute golden retriever puppy, and we can take the single picture, and then we can render it and create a 3D model that works from novel views. And here, if you just, for example, had RGB and a depth map and tried to warp, you wouldn't be able to have the same degree of freedom for moving around and visualizing the scene. This is my grandma's dog, Lola. And it doesn't just work on real-world images. You can also use text-to-image models to first cascade a text-to-image generation

• 3:25:22Speaker 28

  with an image-to-3D creation. So here's a factory robot assembling intricate electronic components with precision. Here is a goblin of some kind, some other creatures, and it even works on some small-scale scenes. I think what's really fun about this is I've been really sick of just staring at like 360 spins of objects for the past two years, but now we can turn these into real, interactive 3D models. And I'd encourage everyone to check out the website and play around with this. It feels fundamentally different when you get to interact with something than when you just have a video that's playing right in front of you.

• 3:25:56Speaker 28

  There are several important bits to get this working. I mentioned the robust loss. I think a huge open question is how do you decide where to put the cameras? It should really depend on what content is in the scene. And right now, we have kind of a few discrete sets of camera trajectories that we chose for different scenes, but it'd be great to find ways of learning where to place the camera as well. The way that you do the camera conditioning can impact the quality of the results in this multi-view latent diffusion model. And what's nice, because we kind of have the set-based representation versus the ordered representation in video, we can come up with different and more efficient sampling strategies to create a number of frames in parallel.

• 3:26:29Speaker 28

  So what's left? I think this is maybe an interesting toy, but not useful yet. I think one of the biggest issues is we moved away from just using large-scale text-and-image data to requiring posed, multi-view data. And if you are aware of using some of these state-of-the-art systems for posing, they don't often work when you have a lot of dynamics in the scene. So I think it's still an unsolved problem of how do we actually scale up these methods and get accurate camera poses if we want to train camera-conditioned latent video diffusion models. The recovered geometry is often inaccurate, even though the novel views look good. And as I said, the camera trajectories don't consider the image content. I think one of the biggest issues is that the scene and input are often assumed to be static, and there really is no such thing as like static 3D videos. If you look at a lot of the data sets, as I move through the scene, I cast a shadow

• 3:27:14Speaker 28

  into the scene that changes as I move around it, and that's present in these data sets as well. So we really need to find models that can work with dynamic scenes as well as static scenes. Okay, so what were some takeaways from this work? I think that in the Cat3D work, we found that separating 2D priors from this 3D inference process by first sampling and then reconstructing was a really flexible and efficient framework. Unfortunately, it does require more expensive multi-viewer video models to generate those correlated samples. And something that I've been frustrated by is these optimization-based inference methods like score distillation and variational score distillation. They can handle uncertainty. They allow you to express an uncertain prior from what these novel views should look like, but they're way slower, they're lower quality, and it's more complicated. And I still think there's a big gap between the sample quality you get out when you naively

• 3:27:59Speaker 28

  sample from these models and when you use an optimization-based approach for sampling. So I think there's still room for a lot of innovation in inference methods. And I think the other thing that people don't talk about in the 3D spaces, these 3D models are useless. They often don't have good enough geometry, they don't estimate the material properties. If you turn them into meshes, the topologies are not actually useful. They have baked-in lighting. So if we actually want these to be useful as, for example, assets in a game, there's still so much more work to be done. So with that, thank you very much for your time and happy to take any questions if we have time remaining.

• 3:28:32Speaker 29

  I have a question. There is a recent work from ICML last year, multi-diffusion. They generate panoramas. Would it be possible to combine something like this, some of this approach for like 3D scene? Because it's also about consistency, generating different scenes, and so on and so forth.

• 3:28:55Speaker 28

  Yeah, I think the multi-diffusion work is very cool. It allows you to take a lower dimensional model or a model on a smaller number of pixels and extend them. And there you can think about the way that they resolve differences between, for example, different frames as averaging within the diffusion process. There are some people who have tried this for 3D as well, where you can think about resolving this inconsistency in 3D over the course of the diffusion process, but it's often a little bit more finicky because to update that 3D representation might require multiple steps of optimization where you can't analytically solve for this update and average them together. But yeah, I think it's really cool to think about how to combine different ways of doing conditioning and guidance and sampling and diffusion processes to do some of the, kind

• 3:29:36Speaker 28

  of, enforce more of the consistency at sampling time as opposed to this just let the model do whatever it wants sampling-wise and then do something on top of the samples.

• 3:29:44Speaker 29

  Thank you so much. Does anyone have any questions?

• 3:29:52Speaker 30

  Hi. I think for all these three works, it seems that from the Dream Fusion to Reconfusion to Cat 3D, the more multi-view you are modeling in 2D, the better 3D you get. Is that the conclusion? Maybe you just do everything multi-view like a trainer. You have a model can generate 200 images at once. So you don't need to have any optimization, right?

• 3:30:19Speaker 28

  Yeah.

• 3:30:19Speaker 30

  So would that be the future is that extreme? Or maybe something in the middle?

• 3:30:24Speaker 28

  Yeah, I think what's very frustrating and I think very broken about these models is there's this back and forth. So, you know, how much of the structure do you put into the multi-view prior or the video prior 2D prior? And how much of the structure do you kind of extract afterwards? And it seems very bizarre to me that we put all this effort into training these 2D priors. We train them until they're 3D consistent. And then only afterwards do we touch anything in 3D. And going from like Reconfusion to Cat 3D, we removed 3D structure in the diffusion model and it got better. And so I think these existing methods don't really support the kind of like real time, interactive generation. You know, maybe I want to start capturing the scene and have it fill in the details and iteratively update some 3D structure. And we don't really have methods that do that right now. It feels a little bit broken.

• 3:31:07Speaker 28

  Ideally, you could build one system that gave you the 3D outputs and could learn from solely image based data. But and there's some cool work like you said diffusion, render diffusion that tries to build diffusion models that have that 3D structure inside of it. But so far, the performance of these methods haven't been as much because we don't really know how to scale them and train them on large, you know, bigger data sets like we do for more of these pixels based models. So I'm not sure which way it goes, but I hope we have more hybrids and find ways of incorporating that 3D structure into the 2D models as well.

• 3:31:36Speaker 31

  Hello, thank you for your talk. I think we always see in the literature this effect, particularly in text to 3D generative models, where you have these over saturated colors when you're generating assets. I was wondering if you have any more intuition about what might be causing this?

• 3:31:54Speaker 28

  Yeah, it's a great question. I think my initial intuition was that this just has two things. One is that we have all these tricks for diffusion sampling and also for these distillation approaches that are built around guidance. And so you're going in a direction that matches the text prompt better and moves away from the unconditional prior. And if I have a frog, frogs are often green. And so in the dataset that we had, they might be biased to green things. So putting a green background there might often lead to a higher density mode, but that might not be a good sample. So I think a lot of the issues with this kind of oversaturation and contrast come from how broken and bad the loss functions are, combined with how hacky it is to use classifier guidance for solving these problems. And that's kind of why we've moved to more of these sampling-based approaches, is that

• 3:32:39Speaker 28

  they just work better. You don't have to worry about the artifacts that you get out of optimization. But I think that's like, this is like a gap. I wish that we had a better explanation for why these artifacts appeared. Because you do see them with classifier-free guidance, as you crank it up, that you get over-contrast and oversaturation, but not nearly to the degree that we get in these optimization-based methods like square distillation. It's very frustrating.

• 3:33:01Speaker 00

  Yes, thank you. Believe it or not, there were other research labs than DeepMind represented at ICML. We stay in the generative modeling workshop, but transition to Ricky T. Q. Chin of MetaAI, aka FAIR, who presented the most approachable explanation of the flow-matching technique in generative modeling that we have yet heard.

• 3:33:22Speaker 32

  So I wanted to give a brief talk on flow-matching, or this sort of idea of flow-matching, and applying it to various different kinds of domains, from Euclidean to Riemannian to discrete domains. We recently had a paper put out called discrete flow-matching, which basically uses this sort of flow-matching recipe, I'll call it, as a way to motivate, a way to construct general models over discrete domains. But really, it seems like you can use this recipe, this sort of very abstract notion of a way to build a general model, and apply it to any sort of domain. So let's get started. The goal of this talk is to discuss a few different application domains, but also, I

• 3:34:07Speaker 32

  want to say that in all of these, there is one very simple process to build a general model over these domains, and they share the same sort of underlying principles. And the idea is, I think more people are familiar with the Euclidean space. So here, on the top left, we start with parameterizing some velocity. And with that, if we transport the particles according to that velocity, we also change the distribution of those particles according to some law. And here, we can, yeah, we apply this to material generation. We also applied this in the discrete domain to code generation and text generation just to scale it up and see what happens. So I'll put the end, the last slide here at the very beginning,

• 3:34:49Speaker 32

  which is maybe a quick preview of the flow matching recipe, I'll call it, at least for this talk. I'm not sure if I'll call this later going forwards. But here, we just wanted to find conditional velocities, very simple velocities that's conditioned on x1 that generates x1. So these UTs and these transport formulas on the left, if I start from xt and advance to xt plus h, then if I follow this velocity, then I basically transform particles according to this pt given x1. And in particular, pt at time equal to 1 is going to become a Dirac distribution centered at x1. So if this is the case, that is, I can create these conditional velocities that basically just generate a single data sample,

• 3:35:33Speaker 32

  then basically the learning problem becomes just, I just want to learn the expected velocity. And that's it. So given an xt, this expectation's over x1, which is sample from some data distribution. Turns out, if you learn this expected velocity and follow that with the same transport rule that you have on the top left, that allows us to generate from the data distribution that we trained this expected velocity form. And this relationship is because of something called the continuity equation. And it's actually because of this linearity of the divergence operator as well. And I'll get to these, you know, I'll unravel this explanation in a bit. But just to sort of set the scene, let's start with the Euclidean setting again, which is what most people are familiar with.

• 3:36:16Speaker 32

  So assume we have some samples from a data distribution q of x1. And we're going to construct these conditional probability paths, pt of x, given x1, such that they basically converge to a delta at x1. And if you think about all these probability paths, and we're just going to, we're going to just marginalize that over the q data distribution on the bottom. This marginal probability path is going to generate the data distribution at time 1, starting from some whatever noise distribution that we've set. In particular, time 1 is a data distribution.

• 3:36:47Speaker 32

  Now it turns out that if you just look at the velocities that generate these conditional probability paths, that is, if I were to follow these velocities, then I create samples that are marginally from this Pt given X1. Now, I want to also marginalize the velocity in the case, in the sense that I take the conditional velocity and I take an expectation over P1 given t. So P1 given t is the conditional expectation of X1 given X at this current time point. You can think of this as like a responsibility, right, if you're familiar with Gaussian mixture models. Basically, we weight the conditional velocities by this weighting P1 given t, and that's how we define the marginal velocity. This is the thing that we're going to fit to in that expectation. And it turns out that there's a really simple

• 3:37:33Speaker 32

  sort of explanation for linking this behavior between the marginal probability and the marginal velocity. And to get to there, we need to start thinking about, okay, how do we connect the velocity to the probability of the distribution that we're transforming these samples? And that relationship is from this continuity equation, right? So in particular, this continuity equation says, at a certain point X, the change in probability at that point is related to the negative divergence of this velocity field times that probability, right? So what is the divergence? The divergence is, at that certain point, let's take a small, people usually take a ball or you can take a hypercube around that point, and then we look at all the outflow from that area, subtract the inflow, right?

• 3:38:17Speaker 32

  So how much mass am I losing from this point? Oh, sorry, from this area. You take that area to be infinitesimal, and the divergence is basically the continuous approximation to how much mass am I losing from this current X, right? So the change in probability is going to just be the negative, right? It's the, if I lose mass, the probability goes down.

• 3:38:37Speaker 32

  So that is the basic, very basic relationship between velocity and probability. And if we assume the continuity equation and we basically can find a velocity that generates the conditional probability, then we've, basically, this is the three-line proof for flow matching on Euclidean space, right? So the first line is just, it's by definition, we're defining the marginal PT as just a mixture of conditional PTs, and then we apply the continuity equation, assuming that we have these conditional UTs in hand, that can generate these conditional probability paths. The third line is just, it's an interchange of the divergence operator, it's a linear operator with integral, right? It's because of this exchange that we can basically move the integral inside, and now

• 3:39:20Speaker 32

  we just have this definition for UT, which is the marginal velocity field, right? And we've said, okay, well, this is the form of the continuity equation, and so this marginal velocity field must be generating the marginal probability path. It's a really simple three-line proof, and, you know, we're going to be seeing this, the same proofs for other domains as well in this talk. So, but just to complete the picture, here's the, here's what we did for flow matching. We're going to just directly regress a VT, which is a neural network for parameterizing the velocity, onto the conditional UTs, and this gives the optimal solution, which is the marginal velocity. The way that we proved it is, we basically looked at gradients, and the gradient is the same thing in expectation as the intractable flow matching loss, which directly matches,

• 3:40:05Speaker 32

  regresses onto this marginal UT. Okay, so, I mean, just to show some examples, this is, you know, flow matching applied to text-to-image generation, you have some text, you know, people like to look at these figures, so. All right, all right, it's started getting a little bit more interesting now, okay. So, a lot of people here in the audience like structures, so we're not going to stay in Euclidean space for too long. There's a need to consider non-Euclidean structures, as the entire panelist's discussion was about, so I won't really go into the motivations anymore, I don't think there's a need to do this, there's a lot, there's a lot of different domains that have imposed structure, and we really want to be, you know, modeling that type of structure, maybe explicitly in the general model itself.

• 3:40:48Speaker 32

  In particular, if you think about Ramanian manifolds, that is, manifolds where locally you can basically have a kind of like a first order approximation to the manifold. Locally, it's a Euclidean space, right? So people call this a tangent plane. At every location, there's a tangent plane. And because this tangent plane is just Euclidean, we can also just define vector fields on this Euclidean space. Everything from Euclidean definitions for continuous flow matching, sorry, continuous flow matching flows kind of just naturally extend to the Ramanian manifold setting. And in particular, we can just replace the continuity equations divergence with the Ramanian divergence. I won't go into any of the details on this, but for the sake of,

• 3:41:33Speaker 32

  yeah, anyway, so for the sake of just being complete, let's look at the continuity again. So assuming that we have a UT that satisfies the continuity equation with that Ramanian divergence, I've just replaced that in the second line. So here, the integral is over, it's not the same volume, right? There's a different volume element, and it's a different manifold. There could be boundary conditions. I'm just sweeping all that under the rug in this notation. And the only thing that's important is now I can still exchange this divergence and the integration. And basically, there's still a UT that is the expectation of the conditional UTs. So there's a lot of, you know, complexity in here. It's not clear how to even find a conditional UT. I won't really go into that in too much detail.

• 3:42:18Speaker 32

  So one thing we did at Meta was we basically applied this to material generation. This is an idea that, so you want to generate a crystal or a material, which is represented as an infinite set of atoms that's repeating in every single direction. So the way we represent it on a computer is we represent only a unit cell. And we just assume this unit cell just repeats in all directions. So basically, this unit cell has a periodic boundary condition. That's the manifold we're working with. Now, a material is basically, we call it stable when it can actually be synthesized in the real world. It's the most basic, but the most important property. It's not clear how to even check if a material is actually stable. People usually rely on a database. But we basically applied Ramanian flow matching

• 3:43:00Speaker 32

  to try to generate materials, given a set of stable materials, and see if we could generate something that's novel and also stable at the same time. One thing I'll say about this is that it was actually surprisingly hard to combine manifolds with ecovariance at the same time. So when people, for example, when people use, let's say, a point cloud, right? So you want to impose some sort of translation invariance. The way people do this is you kind of just remove the mean, right? You take the mean of the point cloud, you just subtract it. And then when you define the flow or the diffusion process, you basically take a zero mean noise variable. And so the path basically is always zero mean. There is no zero mean in this periodic boundary condition in this because there's no origin, right? It's a periodic space.

• 3:43:43Speaker 32

  So we actually had to basically project the velocity field to make sure it doesn't actually move the mean, right? So there's a few tricks in here that actually was a bit surprising, but I think it's worth thinking about. Like I said, it's in particular very interesting in structuring. And how do you combine different types of manifolds with different types of ecovariances? It's not actually like an orthogonal combination. Anyway, so yeah, we took a look at this. And a material is represented by three components, a unit cell, which is basically like a conformation, deformation of like a 3D space, 3D coordinates. And then we have these fractional coordinates, which define where the particles are, where the atoms are inside this unit cell. And then we have the atom types for each of these atoms.

• 3:44:26Speaker 32

  So remoniflu matching was great on the continuous variables, which are these unit cells and coordinates. It was basically, you know, we could say state of the art, or at least it proves a little bit over the diffusion baselines. So that's conditional on the atom types. We just tried to find a stable conformation. But if we want to do de novo generation, that is, we want to generate a completely new material from scratch, including the atom types, it was okay. It was not great. I would say it was on par with some of our baseline LOM approaches, but it wasn't a big change. So that was a little bit disappointing, I think. But yeah, so maybe I'll explain a little bit. So here, the atom type we basically represented as a continuous embedding, so embedding into continuous space. And then we just did regular Euclidean flow matching

• 3:45:11Speaker 32

  to try to learn the atom type. And then during sample generation, we would just take a one year's neighbor and say, okay, this is the atom type of my sample. But that didn't work very well. So we wanted to go a bit further and say, okay, well, can we just take what we learned from flow matching, just directly apply it directly to discrete space? In the sense that we don't assume any sort of metrics. We don't assume any sort of like continuous, you know, space. We just have a bunch of different, like, D different possible values for my, for my samples. And, you know, what happens? So before I get into, actually, the, yeah. So, yeah, so the answer is yes. And our work is not the first, definitely not. There's a really interesting work from Campbell et al.

• 3:45:55Speaker 32

  Also presented ICML, general flows on discrete state spaces, which is, is a, basically model a continuous time Markov chain. And it's, it's really nice. A lot of the things that I'm going to go over are sort of also in that paper, just maybe with a slightly different spin on it. So, okay. So the first thing is, what is a velocity? As I said, we're going to just take the expected velocity. But what even is a velocity for transporting a discrete sample x t, right? So in the continuous case, we just added a little offset to the particle itself. In the discrete case, we're going to add a little offset to the, to the probability. Right? So we're going to start from a direct distribution centered at x t. And then we're going to modify that distribution and then sample from it. So as long as we, sorry, this is, this is like a preview.

• 3:46:39Speaker 32

  I'll actually prove that this is correct. I'll justify this while there's sampling it in a few slides. But this is a preview to try to understand what, what velocity is. So, oh, sorry. And people also call this the rate matrix for the continuous time Markov chain paths. So here, the updates are independent per dimension or token, as we say it. And as long as we can find a velocity that basically implies that this new variable, this x t plus h is following the probability path, then this is our definition of a, of a velocity. So again, in the continuous case, here are some visualizations. We basically model for each dimension, each coordinate, some, some change. And then we're going to just take all the changes at the exact same time. Right? So we're just going to move the vector field, sorry, move the particle according to that vector field.

• 3:47:22Speaker 32

  In the discrete space, as again, we have these axes aligned. This is a sort of visualizing the discrete space on a grid like, but you know, it's, it's just a visualization, right? We don't impose any sort of neighboring information. But basically, for each, each coordinate or each token, there is a set of possible values that we can move to. And this u t is essentially a change in the probability mass from x, from x t to some other states.

• 3:47:48Speaker 32

  This will be a bit more clear. So if you look at this stereotis equation for still for a bit longer, right? There are some additional constraints. In the continuous case, the velocity could just be anything. We could just always move like we're assuming you're creating space, no boundaries, nothing. But in this setting, UT needs to satisfy certain constraints. Because we already start with a PMF. The delta xt is itself a PMF. It's just 1 at xt and 0 everywhere else. It's already at 1, so I'm sorry, it's already 0 when the point is not xt. And so the only way to make the right-hand side a valid PMF is to make sure UT is positive or is not negative when xi is not equal to zi. I'm talking about this constraint here. And the other constraint is we need to make sure the normalization constant stays the same, right?

• 3:48:30Speaker 32

  So here, it normalizes the 1, or sums still 1, right? We make sure, we want to make sure this is a valid PMF. And so we need to make sure that this UT sums to 0, right? And so what that implies is basically there's, you know, at xt, if this point is xt, UT needs to be negative. And if it's something not xt, then it needs to be positive. That's, that's it. That's, I'm just saying, there's some additional constraints on this. And so velocity, basically here, is just modeling the transport of probability from one state to another, right? If you're at this current state, you have some positive mass. And I say, okay, well, with probably 20%, I want to move, I want to move 20% of my mass currently to some other state. Then for each particle, I'm going to just going to flip a coin with 20% probability, I'm going to move. 80% probably I'll stay, right? Something like that. So, okay, let's actually derive that.

• 3:49:15Speaker 32

  That was just a sort of preview for what a velocity would be that coincides with the continuous time markup chain equations. So let's start again with the continuity equation, right? But here, we're going to try to define discrete divergence. So divergence is again outflow minus inflow. It's the amount of mass moving outside this domain, outside this node, let's say, this point, right? Minus the amount of mass that's moving in. So in the discrete case, we can basically just assign values to edges on a graph. And this edge will just denote how much mass is moving from one node to another. It's going to be, when we compute divergence at a certain point, we're just going to take all the outflow minus the inflow.

• 3:49:59Speaker 32

  So let's say this v is our flux or current. This is the scalar function that is defined on the edges of the graph. And we want to sum over all the domain, not axis aligned. This z is just over the entire discrete space where each coordinate has d possible values and there's n different discrete variables. Here's where we make an assumption. We're going to assume the single token change graph relates basically that v is only defined, or it's only non-zero when x and z differ by one token, by one coordinate. If it diverges by more than one, it's just zero. So we're explicitly making this assumption that the velocity

• 3:50:41Speaker 32

  or this flux right now can only modify one token at a time. I'm just saying this is slightly different from previous treatments of this where they define these continuity equation or the commograph equation on 1D and then sort of justify why to do it in high dimensions. Here we're going slightly a bit the reverse. We start with the continuity equation defined in high dimensions and we're just going to make an explicit assumption because we don't want to be modeling basically a complete graph over this really large space. All right, so we're going to assume this is a flux again in the continuum setting, so v is going to be p times u, p times the probability. So we're going to make that assumption here as well. And if we do some algebra, we arrive at this equation for

• 3:51:23Speaker 32

  the divergence at a certain point x. So this is our definition for a velocity field. So there's two ways to think about this. The continuity equation is a way to it's a way to build in the relationship between the velocity field and the probability. I'm right now thinking about it as a way to define the velocity. So given the probability, how do we define the velocity field?

• 3:51:49Speaker 32

  So there's two things. So one thing is if we have a UT that satisfies this discrete continuity equation, then the Euler sampling that I just described a few slides ago is now justified. In particular, if we take the first order approximation of PT. So PT itself is going to be just the expectation of an indicator function. And here again, we're going to look at this as an expectation as well. So we're just sampling from PT, and then this is the term inside. If we do a little bit of algebra and move a lot of the terms that are little o of h or higher outside, we arrive at this expression, which is the Euler sampling. So for each coordinate, we're going to just sample independently for that coordinate,

• 3:52:33Speaker 32

  and the rest is just little o of h. So by assuming this continuity equation, we've justified the Euler sampling. And also, if we assume this, then we can also prove that the marginal velocity, this flow matching recipe, also holds. So everything is the same. The first line is the definition of PT. It's the mixture of the conditional PTs. The discrete continuity equation shows up, and then we're going to just exchange what happens inside with this sum over x1. And the sum over x1 gives us the marginal velocity. So again, if we take conditional velocities that satisfy this, we define the marginal velocity.

• 3:53:13Speaker 32

  Then if we have access to the marginal velocity, we can transport using this Euler sampling. That's what we're saying. So how do we actually define? That was a little bit abstract. So let's actually define this discrete path and maybe look at a few special cases that are very strong in practice. So first off, we're going to just say, let's define the marginal PT as a bunch of marginalization over a bunch of conditional PTs. And then each conditional PT is independent in each dimension.

• 3:53:46Speaker 32

  So a special case of our framework, we basically work with like an arbitrary mixture of m different distributions, but let's just consider two different distributions. So condition on x0 and x1, xi only has two possible values. It's either going to be x0 or x1, and it's just a mixture between the two with some probability kappa t. There is another special case, which works really well, which is x0 is going to just be the completely masked state. So given a sequence, and I'm going to just start from the completely masked sequence, and then I slowly unmask each token until it reaches the x1 distribution with some probability kappa t. Now, this is very effective. It's a little bit unsatisfied that it actually is effective,

• 3:54:29Speaker 32

  but it's a very effective probability path, and it's used by many works. It's related to masked language modeling, obviously. So yeah, I just want to bring up these. These works also do a very good job at generalizing this construction a little bit to learn different components. Okay. Yeah, but let's not worry too much about the masked state. Let's work with the mixture of two deltas for now. And one thing that we proved in this paper is that basically there's a conditional velocity, and then if you marginalize it, you get the marginal velocity that is currently highlighted in gray here. And there's two. Okay, it's similar to the continuous case, where you can always add like a divergence-free term to the flux, and there's an infinite number of velocities

• 3:55:12Speaker 32

  for the same probability path. It's very similar here. Basically, we can express the velocity either in terms of a p1 given t or a p0 given t. The first one makes sense when we want to solve things forwards in time. We're going to predict x1, and then we transport forwards in time. The second one makes sense when we want to transport backwards in time, so we predict a p0 given 1 and it just transports backwards. Now, what's interesting here is that both of these velocity fields satisfy that continuity equation. So we can always plug it in. In particular, we can use any convex or any combination of these two multiplied by some coefficient, right? So what we actually do in practice, or what actually works in practice, is we basically take a very large step with the forward time velocity field, and then we take a small step backwards.

• 3:55:57Speaker 32

  So this is kind of similar to predictor-corrector steps, right? So where we take a small step and then you kind of like do some extra computes to sort of change the variables itself, but without changing the marginal probability. So this gives us a little bit of, if you take the mask setting, it just gives a little bit more flexibility in let's not just unmask, which is what the forward process will only do. Once you unmask, you cannot mask again. But if you add in the reverse time velocity, you can also add the ability to remask and then unmask again. So it adds a little bit of a corrector sample.

• 3:56:31Speaker 32

  All right. So another interesting thing is this is very analogous to the continuous case that we've been looking at. So in the continuous case, if you look at the poly path as being a convex combination of x zero and x one, usually people write it in terms of the denoiser or the epsilon prediction. So here again, we also have very similar things. This is the denoiser. This is the epsilon prediction, except now it's predicting the whole distribution rather than just an expectation.

• 3:56:56Speaker 32

  And here are just some examples to show what we tried at scale. We basically trained a 1.7 billion model to try to do some text completion, to try to beat L M's at their own game, which we kind of failed, but we gave it an honest try. So the first thing is, given some doc string, we're going to generate the text, sorry, generate the code. Basically, this is a more trustworthy way of evaluating a large language model. It's not just based on abstract type. There's actually, this code will either run and succeed or it'll fail. The interesting thing is because we have a completely non-autic aggressive model, right, we can do any sort of code infilling. We don't need a condition on the left and then generate the right side.

• 3:57:40Speaker 32

  We can just condition on arbitrary things that we want. So this is one property that's beyond just L M's, but there's no good benchmark for trying to do this. And on the right side is just an illustration of the sampling process. So here it's just pure mask. There's no unmask, there's no corrector stuff. Just looking at the, using a masked p zero and then just unmasking it.

• 3:58:03Speaker 32

  So here's more of a sunny check that we did on Open Web Texts. Learning a language model, in particular, the thing is, equation 9 is the masked Prouloy path, which is what most people do, except we tune the scheduler and we tune the corrector step also as well. And then equation 10 is some combination of a mask distribution, a uniform distribution, and then the delta of x1. And here we see that the equation 10, this mask was used from, you know, from, actually does better than the pure mask setting, but really only maybe at the low NFE setting, right? Or as a high NFE, the mask is OK, right? So the reason for this is, for the mask case, if you just unmask one token at a time, I think it'll be correct.

• 3:58:45Speaker 32

  But if you sometimes unmask two things at a time by parallel, you can end up maybe with incorrect samples, and so you really need to correct that, or maybe allow a uniform Prouloy path where the noisy state includes some other state and the model will learn to correct itself later on.

• 3:59:03Speaker 32

  And again, yeah, these are just numbers for the code generation for the 1.7 billion model. We also tried the discrete flow matching for image generation. So there's no quantized Gaussian, there's no metrics at all. We just took the masks, again, the mask case, and tried it. Yeah, it seems to be slightly worse than K nuance flow matching, but you know, it's almost at 3 FID, which is pretty good. So yeah, so that's the end of the talk. This is like I said, this is the last slide of the talk, I just put it at the very beginning. So as long as we can define velocities that transport particles according to some PT, where PT arrives at x1 exactly, then if we learn the expected or the marginal velocity, then, and then plug that back into the transport equations, we're gonna get the distribution

• 3:59:48Speaker 32

  that we fit this marginal velocity to. And that's it, that's the recipe, and seems to hold for the discrete setting as well. And yeah, here's my research collaborators at Meta. Some of them are also in the audience, so if you have any questions, feel free to ask them as well. Thank you.

• 4:00:07Speaker 00

  Now that we understand the flow matching objective, we turn now to its most famous application this year, in Stable Diffusion 3, presented in the paper, "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis." This paper also won a Best Paper Award at ICML, and here is Patrick Esser, one of the original co-authors of Stable Diffusion under Robin Rombach, accepting it.

• 4:00:33Speaker 33

  Hello, everyone. My name is Patrick Esser. I'm presenting our work, "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis." All of this is the result of a great team of people who are also here and who are all on this slide.

• 4:00:50Speaker 33

  So currently, we observe a big hype about scaling, and it's really tempting to say that we can simply solve all of our problems by throwing enough money at it.

• 4:01:04Speaker 33

  And indeed, we would say the effectiveness of scaling can't be denied. Increasing the model size, the number of training examples, and the overall compute resources that we put into the training consistently improves the model performance. We've first seen this for language models, but we also observe similar trends for image generation, which is what we actually consider in our work.

• 4:01:30Speaker 33

  But of course, scaling isn't free, right? It really massively increases both the development costs, because we have to put all the resources into training,

• 4:01:41Speaker 33

  but also increases the operational costs, because sampling is getting more demanding with the bigger models. So really, to avoid burning money, we constantly have to keep improving the efficiency of both training and sampling. The three key questions that motivated our work were essentially the first. Given that there are currently quite a few different formulations around diffusion models and flow matching variants, which of those are the most effective?

• 4:02:10Speaker 33

  The second question...

• 4:02:13Speaker 33

  I address the question of the architecture, since for the task text-to-image synthesis, which we have in mind, we really have to deal with two different modalities. And it's unclear how to,

• 4:02:26Speaker 33

  which architectural design choices work best here.

• 4:02:30Speaker 33

  Finally, when we talk about scaling, we usually have to measure the progress and we also derive the scaling laws based on simple metrics such as a validation loss. But ultimately, they are really only a proxy for the downstream performance we're interested in, which might be sample quality. And we want to evaluate whether they are an accurate proxy for those properties.

• 4:02:57Speaker 33

  So let's start with flow matching and friends. The common goal of these methods is basically to learn a vector field, which will be parametrized by a deep neural network. And that should generate a probability path between two distributions. In our specific case, we then usually consider one of those distributions to be a simple known distribution, such as a standard normal distribution, and the other one to be a data distribution of images.

• 4:03:29Speaker 33

  The common starting point to learn such a vector field is then to define a so-called forward process, which essentially just defines a trajectory between two samples from the distributions we're looking at.

• 4:03:43Speaker 33

  From this process, we can then derive a tractable regression objective, the so-called conditional flow matching loss, which allows us to recover a vector field that then actually generates a path between the distributions.

• 4:03:58Speaker 33

  The overall paradigm here is fairly general. And for specific choices of the forward process, we can actually recover a wide range of existing formulations and variants, including EDM, DPM, and others.

• 4:04:14Speaker 33

  One of those variants is the rectified flow formulation. And arguably, this is to some degree the simplest choice you can make for the forward process, because it's just a linear interpolation between the two samples. This then also leads to a very clean conditional flow matching loss. And overall, it really makes it very elegant and easy to work with. Also, remember that sampling in this framework then essentially consists of integrating the learned vector field. And because of that, straight paths, as we define them in this forward process here, are actually really desirable, because if they are straight, we could actually integrate them in a single step, which would massively improve our sampling efficiency.

• 4:05:00Speaker 33

  So the conditional flow matching objective actually does not recover perfectly straight paths, even if we define a forward process like this. But at least empirically, we have seen results that compared to the diffusion, to the vector fields you derive from diffusion formulations, they usually have less curvature, which makes them more sample efficient. And they also come with other nice theoretical properties, like a straight-inning effect, which makes them really promising candidates for further improving the sampling efficiency. So overall, this really just makes rectified flows attractive candidates for efficient text-to-image synthesis.

• 4:05:44Speaker 33

  But so far, before the study, they have really mainly been considered in benchmark settings, and it remained a bit unclear how well they would actually perform in practice for more difficult tasks such as text-to-image synthesis.

• 4:05:59Speaker 33

  If we look at the conditional flow matching objective, it always involves the time step distribution over the time steps of the trajectory. And the classical rectified flow formulation really only considers a uniform distribution over the time steps. But since during training, we do a Monte Carlo estimation of this objective, it can really affect the optimization that we perform. And if we look at the loss and also the forward process, how we defined it, then we'll actually really quickly see that at the endpoints of the trajectory, t equals zero and t equals one,

• 4:06:41Speaker 33

  the optimal solution really just involves an estimate of the mean of the two distributions. So we would expect this to be a very simple task in comparison. And because of this, we actually started considering different time step distributions, which put less weight on the endpoints of the trajectory and focus more on the interior. And similarly, this is actually also a big part of the success of diffusion models for modeling images. Because it allows us to control where exactly in that trajectory we're putting in the most weight. And that way, we can actually focus on the parts of the trajectory

• 4:07:22Speaker 33

  where the perceptually relevant aspects of images emerge. So to explore whether we can also benefit from this for rectified flow formulations, we explored various time step distributions that allow us to shift where that focus is.

• 4:07:44Speaker 33

  To then understand which of those formulations is the most efficient, we actually just performed a study across 61 different variants overall. And here we included many existing formulations such as the epsilon prediction with a linear schedule, which is the one used in stable diffusion, for example, a V prediction with linear or cosine schedules. We also include EDM and existing rectified flow formulations. But then besides those, we also included variants, especially of EDM and rectified flow, where we varied the hyperparameters of the time step distributions that are involved. And if we then collect and evaluate those results,

• 4:08:24Speaker 33

  we actually see that the classical rectified flow formulation with a uniform time step distribution does indeed perform very strongly in the regime of sampling with few steps. But if we, for example, one of the strong baselines that emerged from this was an absc linear predict scheme, and compared to that, it actually performs worse when we sample with more steps. And really in contrast, what we saw is that by introducing this particular time step distribution, the logit normal distribution, we end up with a with a variant of rectified flow that actually performs better than all existing variants, both in the regime where we sample with few steps, but also if we sample with many steps.

• 4:09:12Speaker 33

  Then after looking deeper into the generative process, we also considered architectural choices for text-to-image synthesis. Overall, the goal was to focus on transformer-based architectures because of their good scalability properties, but it wasn't directly clear to us how we can best integrate the two different modalities, text and images, which are required for our task. So in one of our ideas, we introduced the MMDIT block, which generally follows the design of a DIT block, but it actually uses two separate weights for each of the two modalities. But then to exchange information between the two modalities, we still use a full joint attention operation.

• 4:09:55Speaker 33

  A similar idea has also been used actually in vision language models. And what we observed from comparisons is really that this this performed the strongest. Some of the comparisons we performed this to a simpler approach where we use a DIT architecture and simply directly concatenate the two modalities. And we also considered UBIT and DIT variants, where instead we use a cross-attention mechanism to incorporate text conditioning, because that had been very successfully applied in unit-based architectures. But overall, we saw that this multimodal design really off the best performance.

• 4:10:38Speaker 33

  So after figuring out efficient formulations and efficient architectures, it is time to scale. To get a clean signal for the progress during the scaling, we evaluate the validation loss at fixed time steps.

• 4:10:54Speaker 33

  Comparing such a metric is really only meaningful if we stay within a single formulation. But if we are in that case, it actually provides a very clean signal and also a very efficient way to evaluate the model and derive scaling loss from it. But ultimately, this validation loss really is only, can only serve as a proxy for performance, because ultimately we are interested in things like human preferences, prompt following, sample quality, etc. And at this point, it was really unclear whether we can just rely on this validation loss being an accurate proxy for those downstream performance measurements. While there have been more work in the, in the language domain,

• 4:11:38Speaker 33

  this was not the case for image domain.

• 4:11:43Speaker 33

  To answer this, we then performed a scaling study and evaluated the correlation between the validation loss and we consider both automatic image evaluation metrics such as GAN-EVAL as well as human preference ratings. Our results then showed that the improvements predicted by the scaling loss and the validation loss actually translate into quality improvements for text-to-image synthesis. We saw similar results also in different modalities, other modalities like video synthesis. And overall, it makes us confident that further scaling will indeed improve content creation capabilities with generative models. So during model development and scaling, we also had a few additional learnings that we share in our work, which I will quickly go over.

• 4:12:30Speaker 33

  One of the things that emerged, one of the problems that emerged as you scale is the, our training instabilities. And here it was really helpful to learn from existing works. One thing we found particularly helpful was QK normalization was stabilized, which stabilized training.

• 4:12:49Speaker 33

  Another point that I quickly want to mention is that we reiterated on the story that scaling improves the performance, but it's also important to note that blindly following this is not, is quickly becoming inefficient. One of those cases is if we again consider time step distribution, we actually have to adjust that to different resolutions. If we don't do that, we lose a lot of performance. And you might say that we could simply scale up the scaling, the base model, but that would, the, the cost you would have to pay for that would be tremendous compared to if you properly fixed the problem. A similar result is related to aligning with human preferences, which gives a very quick,

• 4:13:32Speaker 33

  cheap, but effective boost in preference scores. With this, we then really obtained a high-quality model that worked well across different resolutions, aspect ratios, and had a good prompt understanding and the ability to spell. This was also reflected in human evaluations against other existing models. And with that, so long, and thanks for all the attention.

• 4:13:56Speaker 00

  One of the most underrated applications of diffusion is in speech synthesis. There was also great work at ICML this year on speech, and with the rise of ChatGPT's voice mode, there is a lot of demand in learning about the fundamental problems and techniques. Here, we will simply dip into two oral presentations that we would highlight on speech. Natural Speech 3: Zero Shot Speech Synthesis with Factorized Codex and Diffusion Models by Zhu et al. and Speech Self-supervised Learning Using Diffusion Model Synthetic Data by Gao et al.

• 4:14:33Speaker 34

  Hello, everyone. This is Zetian Zhu from University of Science and Technology of China. Today, I'm very delighted to share with you the exciting progress in the field of Zero Shot TTS, Natural Speech 3: Zero Shot Speech Synthesis with Factorized Codex and Diffusion Models. Let's begin by illustrating the difference between the short TTS and traditional multi-speaker TTS through a practical example. In the scenario of traditional multi-speaker TTS task, the, the user may request the model to read a transcript in the style of certain speaker, let's say speaker number one. This implies that the model should mimic the voice characteristic of the speaker.

• 4:15:18Speaker 34

  Here speaker number one should be added into the training data. However, a significant limitation of this approach is that its inability to expand to unseen speakers. On the contrast, zero-shot TTS offers a more frequent, flexible solution. The user, uh, the user can supply an audio clip as a reference to guide the generation process. Um, for example, if the user submit her voice like this.

• 4:15:47Speaker 35

  Marine engineering proves especially authoritative.

• 4:15:51Speaker 34

  Although this reference speech is short and unseen in the training, our robust zero TTS system can generate speech with a similar, um, convincingly similar outputs.

• 4:16:05Speaker 35

  For the past ten years, Kinsale had gone with me wherever science beckoned.

• 4:16:11Speaker 34

  Zero-shot TTS is now revolutioning the way we think of voice synthesis. Uh, this advanced model uses vast of varied data sets at training, capturing nuance of numerous speakers and a diverse range of acoustic environments. And as a result, the model can utilize the knowledge gained at training to generalize to a certain speaker at inference through a prompt-based generation method. The key to zero-shot TTS lies in the concept of scaling up. Traditional TTS system hierarchically rely on clean data from the recording studio. This data sets typically consist of less than one thousand hours of recorded speech.

• 4:16:54Speaker 34

  Now, zero-shot speech, uh, zero-shot TTS system uh, harness the vastness of inter- internet, use large-scale core data from the web. This approach used a data set uh, with a total duration of exceeding sixty thousand kilo- uh, six thousand hours speech. On the other hand, the scaling of acoustic model is also remarkable. Model is beginning with less than fifty million parameters. And a current zero-shot TTS system has been expanded to uh, three hundred million to one thousand million parameters.

• 4:17:33Speaker 34

  And such scaling up also promotes the transition in data representation. Previous TTS system often rely on human prior based representations such as mouse spectrogram. On the contrast, their short TTS now are using data driven representations such as the representations derived from the codec. Here is an example codec. They use residual vector condesires that are VQ to generate multiple representations in a coarse to fine manner.

• 4:18:11Speaker 34

  And although previous systems achieved great success, they still for short in speech similarity, speech quality and speech processing. This limitation stemmed from the intricate complexity of speech. For example, for a short speech clip, while seemingly very simple, it is a rich capacity of information. It contains timbre, content, velocity, recording environment and so on. This information is important to the overall natureness of the speech. Motivated by this, we signify the importance of factorization.

• 4:18:54Speaker 34

  Since modeling complex, complex information is difficult such as for raw waveform or mouse spectrogram.

• 4:19:02Speaker 34

  Besides, factorization is also non trivial since our VQ structure fails to effectively disentangle information across our VQ levels. Natural Speech Theory apply factorization in both data representation and speech generation. For data representation, we apply a factorized codec which can decompose speech signal into different speech attributes while ensuring high quality reconstruction. For speech generation, we apply a factorized diffusion model. It is a unified diffusion framework to hierarchically generate each speech attribute in each subspace.

• 4:19:44Speaker 34

  For FA Kodak, we consider four speech attributes, that is, um, timbre, prosody, content and acoustic details. We first apply a timbre extractor to obtain a global timbre vector. Then, we apply three of factorized vector quantifiers to represent speech attributes in each subspace. For better disentanglement, we introduce the following techniques, such as information bottleneck, which can limit the representationality capability of each tokens. And supervision to include the intended attributes.

• 4:20:24Speaker 34

  Gritten reversal to remove redundant information and detailed vault to remove unnecessary information from the detail codes. For factorized diffusion models, we apply discrete diffusion in each subspaces to sequentially generate speech attributes in that in an order of duration, prosody, content and detail. The timbre not be not to be predicted since it is of this global vector can be accessible through the prompt audio. In the forward process, we randomly mask out certain tokens in the sequence. And in the reverse process, the model learns to recover tokens gradually under the guidance of context and conditions.

• 4:21:09Speaker 34

  To facilitate in context learning, we prepare the speech attribute prompts to the sequence as a as a prefix. This prompt act as condition and remains unchanged during the diffusion process. And this prompt magnet, uh, this speech attribute prompts. In the scenario of their short TTS, there this speech attribute prompts are derived from the same audio. And as a byproduct, this prompt magnet also offers great controllability since we can select different speech attributes from various sources. Tailor the output speech to meet specific requirements.

• 4:21:57Speaker 34

  And we evaluate uh the zero shot TTS capability on liberal speech and uh emotional TTS data set revenues in terms of similarity and robustness and overall quality. The compelling, the impressive uh results

• 4:22:17Speaker 34

  demonstrate that natural, natural speech tree not only of form strong baseline, but also achieves human level naturalness. We also test reconstruction capability of our FA codec with strong codec baseline on liberal speech test, test set. And it also uh demonstrated that our FA codec can reconstruct speech with high fidelity using this, using this disentangled speech attributes. Here are some demos. And the first row is the three, three second prompt randomly cut off from an entire speech.

• 4:22:58Speaker 34

  And the second row is the uh natural speech three output speech. For case one.

• 4:23:06Speaker 36

  The standard made to hold another oil cup. So.

• 4:23:11Speaker 34

  So this is the prompt and our natural speech tree can generate a similar sentence uh sim uh using this three second prompt.

• 4:23:20Speaker 36

  There was an average cost per lamp for meter operation of 22 cents a year and each meter took care of an average of 17 lambs.

• 4:23:30Speaker 34

  For case two, this is the prompt.

• 4:23:32Speaker 37

  Is it not clear that there is just as much of the pencil left as?

• 4:23:37Speaker 34

  And this is the output, the output results.

• 4:23:41Speaker 37

  There were only four stationers of any consequences in the town and at each homes produced his pencil chips and bid high for a duplicate.

• 4:23:51Speaker 34

  Our natural speech tree can also generate, that can also generate emotional TTS, you know, the airshots manner by prompting an emotional speech. If you prompt the natural, this natural speech tree with a sad audio that like this.

• 4:24:13Speaker 12

  Dogs are sitting by the door.

• 4:24:16Speaker 34

  And natural speech tree can generate a speech, a sad speech like this.

• 4:24:22Speaker 12

  Why feeds the lotus of the water?

• 4:24:26Speaker 34

  If you prompt the model with a calm prompt, or an audio like this.

• 4:24:32Speaker 37

  Dogs are sitting by the door.

• 4:24:35Speaker 34

  And the, the natural speech tree output will like this.

• 4:24:41Speaker 37

  Why feeds the lotus of the water?

• 4:24:44Speaker 34

  And third one, if you prompt a disgust speech like this.

• 4:24:52Speaker 36

  Dogs are sitting by the door.

• 4:24:55Speaker 34

  And the output will sound like this.

• 4:24:57Speaker 36

  Why feeds the lotus of the water?

• 4:25:03Speaker 34

  Our model is also capable of manipulating attributes by manipulating the corresponding speech prompt. And so here is a demo. The first column is the original setting, that is both duration prompt and other prompts are derived from the same audio, that is zero shot TTS scenario.

• 4:25:27Speaker 37

  Had she enjoyed the experience?

• 4:25:29Speaker 34

  And other prompts is the same.

• 4:25:31Speaker 37

  Had she enjoyed the experience?

• 4:25:33Speaker 34

  And the generated speech will sound like this.

• 4:25:36Speaker 37

  The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired.

• 4:25:43Speaker 34

  If we just slow down the duration prompt, the prompt will sound like this.

• 4:25:49Speaker 37

  Had she enjoyed the experience?

• 4:25:52Speaker 34

  And the generated will sound like this.

• 4:25:55Speaker 37

  The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired.

• 4:26:06Speaker 34

  If we just speed up the prompt, it will sound like this.

• 4:26:09Speaker 37

  Had she enjoyed the experience?

• 4:26:11Speaker 34

  And the generated speech will sound like this.

• 4:26:14Speaker 37

  The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired?

• 4:26:20Speaker 34

  Um, we can also replace duration, we can also derive duration prompts from another new audio clip. This will only manipulate duration attribute will and also and will not affect other attributes since other prompts are keep same.

• 4:26:40Speaker 34

  Dogs are sitting right at the door. And the generated speech will like this.

• 4:26:43Speaker 37

  The examination and testimony of the experts enabled the commission to conclude that five shots may have been fired?

• 4:26:50Speaker 34

  Um, uh, if you are interested in our work, please uh scan the QR for more samples. That's all for my presentation. Thank you.

• 4:27:00Speaker 38

  Good afternoon, everyone. My name is Yang Zhang from IBM Research. Today, I'm going to introduce our paper, Speech Self-supervised Learning Using Diffusion Model Synthetic Data, which is a joint collaborative work with UIUC and UCSB. I'm from IBM Research.

• 4:27:19Speaker 38

  So let me start by briefly going over some background about speech self-supervised learning or speech SSL. Speech SSL, just like SSLs in other domains, assumes that we have a large unannotated corpus. We can then use this corpus to portray a speech representation network, which can then be fine-tuned to downstream tasks on a small annotated corpus.

• 4:27:45Speaker 38

  The key to the success of Speech SSL is that we actually need to assume that we have a large unannotated corpus. In most cases, the number of hours of this pre-training data set should be at least 1,000 hours. However, in many cases, obtaining such a large data set is not as easy as it seems. Here, I'm borrowing a figure of a recent research effort for collecting speech data set for over 1,000 languages. The horizontal axis shows the languages and the vertical axis shows the number of hours collected for each language.

• 4:28:25Speaker 38

  As can be observed, for most languages, the number of hours is below 1,000. This means that obtaining a large pre-training data set is simply infeasible for many cases.

• 4:28:40Speaker 38

  So, in cases where the number of pre-training data is limited, it becomes crucial to maximize the information extracted from the limited data set. Therefore, we raise the following research questions: Do the existing SSL techniques extract enough information from the limited pre-training data set? Could we further extract the information that is possibly overlooked by the existing SSL techniques? So, in this paper, we propose DiffS4L, which is a speech self-supervised learning method that augments the limited pre-training data set using a diffusion model. More specifically, assume that we only have a small unannotated data set, say, less than 100 hours.

• 4:29:26Speaker 38

  So what DiffS4L essentially does is that it augments the data set using synthetic data, which is then used to perform the standard pre-training with the standard pre-training techniques.

• 4:29:39Speaker 38

  The entire data augmentation process consists of three steps. In the first step, we use this small data set to pre-train an initial speech representation network. Know that this initial speech representation network is of poor quality because of the limited data set size, but it is sufficient for our purpose.

• 4:30:01Speaker 38

  Once this speech representation network is trained, then for each speech utterance drawn from this small data set, we can obtain its initial speech representation.

• 4:30:14Speaker 38

  As the second step, we feed this initial speech representation together with the speaker embedding, which is also extracted from the original speech, to a diffusion model. We then train this diffusion model to reconstruct the original speech. The diffusion model is also trained only on this unannotated, a small unannotated corpus.

• 4:30:39Speaker 38

  So after this diffusion model is trained, we can then use the diffusion model to generate synthetic data to form this large synthetic dataset. However, rather than directly feeding the initial speech representation and the speaker embedding as is to the diffusion model, we first pass it to a modification module. In this way, we can ask the diffusion model to generate a good variety of speech that is different from the original speech. So the remaining question is how we actually modify these speech representations. So note that speech is a rich information source. It contains many levels of information, including content information,

• 4:31:24Speaker 38

  speaker information, prosody information. Therefore, the synthetic speech should also contain enough variations across all these dimensions. So we designed the following four levels of variations in the synthetic speech. Level one is the original speech itself. Here is an example.

• 4:31:47Speaker 39

  And sharing her house, which was nearby.

• 4:31:50Speaker 38

  So this is the original speech. In the second level, we feed the initial speech representation and the speaker embedding as is to the diffusion model. In this way, the diffusion model would generate something that is almost the same as the original speech. However, since the conditioning does not control everything, the output speech would be slightly different from the original speech, particularly in terms of prosody. So let me play this audio. Please pay attention to the prosody on which.

• 4:32:20Speaker 39

  And sharing her house, which was nearby.

• 4:32:24Speaker 38

  Now, this which becomes a rising tone, whereas in the original speech it was in a dropping tone.

• 4:32:31Speaker 15

  And sharing her house, which was nearby.

• 4:32:35Speaker 38

  So this is the second level. In the third level, we changed the speaker embedding to a different speaker. In this way, the output speech was still under the same content, but with a different speaker's voice. So here is the example.

• 4:32:55Speaker 15

  And sharing her house, which was nearby.

• 4:32:58Speaker 38

  It's now becomes a different male speaker.

• 4:33:02Speaker 38

  Finally, in level four, in addition to changing the speaker embedding, we also partially masked out some of the speech representations. In this way, the diffusion model is forced to fabricate some new content. That's why we call this type of speech "novel content speech".

• 4:33:24Speaker 15

  And now I'm a lord, bear heaven and this die.

• 4:33:28Speaker 38

  So, as you can hear, the output speech would almost like a nonsensical babble. This implies that the diffusion model is unable to fully capture the grammar structure or the word structure in the original language. However, as we will show, even this seemingly nonsensical babble would still help the performance of the pre-training.

• 4:33:51Speaker 38

  To test the performance of DiffS4L, we used this LibriSwitch 960 as a pre-training dataset, which contains 960 hours of speech in English. We considered two different pre-training settings. In the low resource setting, we sampled only 100 hours of real speech. We then augmented into 960 hours of speech in total by adding 430 hours of level 2 plus 3 speech and 430 hours of level 4 speech. In the high resource setting, we used all of the 960 hours of real speech. We then augmented to 2,400 hours of total speech.

• 4:34:36Speaker 38

  We used two standard pre-training techniques for training our initial speech representation as well as the final formal speech pre-training, which is Wave2Vec 2.0 and Hubert.

• 4:34:48Speaker 38

  And we compare four different data augmentation techniques. Number one is no data augmentation at all, Wave2Vec OGG with LM and our proposed DiffS4L.

• 4:34:59Speaker 38

  We test this pre-training model in a number of downstream tasks. In the first task, we try it with the English ASR, or English Automatic Speech Recognition, where there are only 10 hours of labeled data for fine-tuning. The results show that DiffS4L can significantly reduce the error rate for both Wave2V2 and for Hubert. Moreover, DiffS4L can achieve further error reduction when it is combined with WaveLM. Finally, note that the results in these two boxes are both pre-trained on 960 hours of speech. The only difference is that the results in red box

• 4:35:43Speaker 38

  was pre-trained on 960 hours of augmented speech, whereas the results in blue are pre-trained on 960 hours of real speech. So as can be observed, the gap between them is already very small.

• 4:35:58Speaker 38

  To evaluate the performance beyond speech recognition, we chose this superb benchmark, which contains eight different tasks other than ASR.

• 4:36:09Speaker 38

  And the results still shows that DiffS4L is able to achieve the best performance in almost all the tasks for both the low-resource setting and the high-resource setting.

• 4:36:22Speaker 38

  Finally, to test the performance beyond English, we chose 13 more languages, including some of the high-resource languages and some of the low-resource languages. And the results still consistently show that DiffS4L is able to reduce the error rates.

• 4:36:41Speaker 38

  The final experiment I would like to show is to investigate whether all four levels of variations are helpful. To test this, we go back to our low-resource setting, and then we're gonna fix the original speech to 100 hours, and we're gonna fix the total number of speech to 960 hours. But we then vary the proportion between Level 2 + 3 speech and Level 4 speech. And here is the results of English ASR under different dataset compositions, where the leftmost point corresponds to no Level 2 or 3 at all, and the rightmost point corresponds to no Level 4 bubbles at all.

• 4:37:22Speaker 38

  As can be observed, the best performance is achieved somewhere in the middle, where all four levels of speech are present. That means all four levels of speech, including the level four babble, are beneficial to the pre-training.

• 4:37:40Speaker 38

  To summarize our findings, we find that the diffusion model is able to capture the information speech complementary to what SSL learns. Therefore, our proposed DiffS4L can significantly improve the SSL performance in various downstream tasks and languages.

• 4:37:58Speaker 38

  We also find that the synthetic speech with different levels of variations are all conducive to SSL, even the seemingly nonsensical babble. With that, I'll close the talk today. Thank you very much for your attention.

• 4:38:13Speaker 00

  That was the end of part two of this pod. In part one, we explored video generation and world simulation. And in part two, we explored further diffusion and generative modeling methods across NERFs, flow matching, rectified flow transformers, and speech. In part three, we turned to the generative text to video paradigm on its head and check in on the state of computer vision. First, we have the OG Vision Foundation model, Decaf, which this year won the most prestigious Test of Time award, being first presented at ICML 10 years ago. Here is UC Berkeley professor Trevor Darrell, advisor of the Decaf paper and also originator of the CAFE Deep Learning Library,

• 4:38:58Speaker 00

  accepting the award on behalf of the team.

• 4:39:01Speaker 40

  Thank you very much. It's quite an honor to be here, and thank you for the generous introduction. And it's really quite exciting to be able to talk about the impact of the Decaf work and the CAFE work. And that introduction was very generous. I think really the main, the broadest claim we would make is that Decaf democratized access to this class of tools. And that's what led to the transformational change in the field. Of course, as we'll mention and mentioned at the time in the talk, the This builds on the work, of course, of AlexNet and other papers. Look, the paper's decaf, and the title was a mouthful.

• 4:39:44Speaker 40

  A Deep Convolutional Activation Feature for Generic Visual Recognition. And what is an activation feature? I thought I would start off by translating that into 2024 speak. So today, we would probably think of the F in the decaf as being foundation model. Now, I'm not sure we ever needed to define the term foundation model in our field, but since it's been defined and it's broadly used now, I look back and think of the decaf paper as, in fact, perhaps one of the original or broadest foundation models in vision or deep foundation models. And so that's essentially the main retrospective point

• 4:40:27Speaker 40

  as we looked back to see the impact of this work. And we're honored and even pleasantly surprised that it was selected for a Test of Time Award. In one slide, what was the decaf paper? Really, it was maybe one of the simplest papers we've ever published in my group. Essentially, we took the results of AlexNet, showed the effectiveness of this model as a pre-training model in vision, showed that if you took frozen activations, frozen features, and computed activations on those features,

• 4:41:08Speaker 40

  you could have essentially state-of-the-art performance across a wide range of tasks. So this is, in some sense, the OG foundation model in vision. Take the output of convolutional layers, freeze it, train a linear classifier, maybe train even an SVM back in the day on those models, and boom, you get state-of-the-art performance. I think the main things that the decaf paper did that were

• 4:41:39Speaker 40

  You know, impactful and exciting was visualize and show why the model was working. I think that insight maybe is so common today, like we all understand that. But the vision community especially didn't appreciate it. You know, AlexNet was this amazing result, but people thought it was a special case. It's just going to only be for that one task. And the fact that it started to work for everything and that our paper and other papers like it demonstrated that and that it could be used in this pre-training slash fine-tuning way. That was what really revolutionized the community. And the last bullet here, of course, is probably the most important.

• 4:42:22Speaker 40

  I think the thing that made Decaf important, and maybe this was a tipping point in the community. I think, you know, most papers that you would get accepted to ICML or CVPR in 2000, prior to 2014, you would never get a paper that didn't have an algorithm or a model. Or, I mean, even data wasn't enough to get a paper accepted back in the day. And the fact that this paper, of course, was accepted and now recognized for its impact. The reason it had impact was really the open source release and the broad dissemination of the work through that channel. And that's now commonplace, perhaps even dominant in our field today, but it wasn't back prior, you know, prior to 2014.

• 4:43:07Speaker 40

  So that, I'd like to sort of acknowledge how the community has changed and the community standards. Decaf was part of the CAFE ecosystem, which really was a dominant force in deep learning, one of the dominant, preeminent deep learning frameworks between 2013 and 2018. And again, the reason CAFE Was important. It wasn't the first deep learning framework ever.

• 4:43:35Speaker 40

  There were some unique architectures, but really it was the democratization of the models and access to the models. And the emphasis on heterogeneous compute. You could either use a CPU or a GPU. And really, the industry standard code base that worked well in academia and in industry. And it was really the first widely deployed, the platform for the first widely deployed use of Nvidia GPUs. And that certainly has had a lot of impact. We had the first model zoo that we can find. And the timeline is shown here. Decaf came out in 2013, 2014.

• 4:44:20Speaker 40

  Decaf was the frozen, pre-trained, sort of foundation model version of CAFE. CAFE itself was a deep learning framework that eventually was merged into PyTorch that allowed people to train their own models and have had great impact. And the impact is evidenced by the honor the team has. This is one of three Test of Time Awards the CAFE ecosystem has received this summer. And that really is a remarkable observation. The Decaf paper here at ICML, the RCNN paper at CVPR, and the CAFE system paper, which was at ACM Multimedia,

• 4:45:04Speaker 40

  also has had the honor of a Test of Time Award that will be presented later this year. I think Decaf might be the most important of these three papers, which is maybe surprising in retrospect, because at the time, I'm not sure the Decaf paper was viewed as important as these other two papers, or at least as the CAFE system itself. But I'm going to walk through some of the old slides and then tell you a little bit about some of the current observations and thoughts about why and how this work looks sort of from a historical light. And what does the pre-training paradigm mean for the present and the future? So these are the old slides from 2014,

• 4:45:50Speaker 40

  presented at ICML 2014 in China, in Beijing. And this is what the world of computer vision looked like, you know, in the 2010s, early 2010s, 2000s. Really, starting in late 1990s, 1998, 1999, the machine learning revolution took over in computer vision. But for a good decade, it was the pathway seen on this slide with handcrafted features, words like SIFT and HOG and LLC that maybe aren't even, and SURF that may or may not even be known to the community today. But then, and then we had our wonderful CAFE Cat,

• 4:46:33Speaker 40

  which of course, detecting cats on the Internet was the paradigm of its day. But for several decades, and not unnoticed by informed researchers at the time, but yet largely unappreciated by the CVPR and even ICML community, to be honest, was the progress in convolutional representation learning and deep learning, as it was later called, the work of, that goes back to Fukushima and the NeoCognitron, the Romo, Hinton and Williams seminal paper in the PDP book, which I encourage people to go back and look, and of course,

• 4:47:18Speaker 40

  the work of Jan LeCun and then Alex Nett in 2012, finally showing that this paradigm did scale, and this paradigm was going to take over. But I think even in 2012 and 2013, vision people were basically acknowledging this is working for object recognition, but certainly it wouldn't work for other things. It wouldn't work for, you know, fine-grained object recognition. It wouldn't work for complicated transfer learning. It wouldn't work ultimately for segmentation and other things. And as we see, and as the decaf paper

• 4:48:02Speaker 40

  helped convince people, that was not going to be the case. In fact, this paradigm was going to take over the field, and did take over the field. And the decaf paper was Essentially, the simplest foundation style model or pre-training paradigm you could advocate. And very, very simple even in the day, which is, let's just take a frozen AlexNet model, which we're going to provide for you, the user who downloads the code from the Berkeley website, and just take slices of the model and compute activation features. You know, compute the representations

• 4:48:46Speaker 40

  that are formed from the pre-trained AlexNet model and see how it does. And the Decaf paper did this, reported this, and asked a series of questions about the quality of these representations and started maybe the first demonstration that these models are learning something more than they're trained on or more than the literal task that they were trained on. That they are actually learning the latent knowledge that's encoded in those tasks. They're learning semantic hierarchies and things like that. Again, those are concepts that we take as obvious and common sense today, but certainly in the vision community in 2012,

• 4:49:30Speaker 40

  it hadn't yet been accepted. And that because these representations generalize, they work, they actually, sorry, because they capture this latent semantic representation, they generalize to other tasks effectively. And different layers had different performance. So the Decaf paper was one of the first to show visualizations, for example, using T-SNE on these deep-learned representations. And just these visualizations here, that if you looked at higher layers of the Decaf representation or higher layers of the AlexNet representation, that the ability of these models to capture latent semantic features,

• 4:50:13Speaker 40

  which were called superlabels here in the paper, the models were never trained on these superlabels, and yet there they are emerging as a part of the representation.

• 4:50:27Speaker 40

  And again, I think this is the thing that really surprised the vision community, that you didn't explicitly supervise the model on this signal, but it emerged from the representation. And when you would then look to see what was the performance, of course, AlexNet was crushing object recognition, but there was a view that object recognition was now, okay, that's just machine learning. It's not part of computer vision. But computer vision are these other things, fine-grained part recognition or segmentation, domain adaptation and things like that. These representations just started to crush all the tasks, by moving from the prior best feature, which in this era was called SERF,

• 4:51:11Speaker 40

  when looking at the state-of-the-art domain adaptation challenges in computer vision, which was the Office dataset that was released from our lab, actually, back in 2010. You could see the numbers double just by changing the underlying representation for some relatively fancy domain adaptation technique. But even more exciting, you know, from the perspective of this paper, perhaps less exciting from the perspective of the domain adaptation researchers, the baselines went up by a factor of three or four. Just these underlying representations had an ability to transfer that really crushed all the fancy mechanisms that people had been proposing for domain adaptation.

• 4:51:56Speaker 40

  So that was a sobering moment for many researchers. And across a number of different tasks in computer vision, for example, fine-grained recognition, where you want to recognize individual species of birds, and there was a notion that you may want representations that can also localize parts and have some interpretability. The decaf model, out of the box, did better than the prior art, and when integrated into straightforward to techniques for localizing parts, had further improvements in performance.

• 4:52:35Speaker 40

  And last example in the paper that I'm going to highlight here today on scene recognition. Here as well, the model is never trained on these labels for outdoor, indoor, man-made, or natural, and yet, these super labels emerge in the representation when you visualize it with T-SNE. So there are more results, I encourage you to look back at the paper, but I'll just close with, close this part of reviewing the original talk by noting that, I think the main impact here, those observations were important, but the open source dissemination turned out to be the impactful part.

• 4:53:16Speaker 40

  I mean, there were other papers that came out shortly after Decaf, or maybe around the same time, that also were showing transfer performance, but ultimately, Decaf and the CAFE open source release led to wide adoption of these techniques very quickly in the community. And there was this great website and a cute cap that you could look at, of course, back in the day. And at the time, it was just considered remarkable that these techniques that, just a year before, people imagined you needed 10,000 CPUs to try and run deep learning. There was a great paper in the New York Times, you can go find, on how the Google data centers

• 4:53:58Speaker 40

  were taking thousands and thousands of CPUs to run certain deep learning algorithms. And there was just a belief that, like, no way anybody but Google can do that. And so people were just doing whatever they do. And then suddenly, the next year, through Decaf and related efforts for democratization, almost anyone could get most of the performance of these models, at least for inference. And then soon, with CAFE and the advent of GPUs and GPU acceleration of these models, everyone could train a model of this size. And so that's an exciting point, and maybe we're going to have a similar moment in the future. Right now, there's a perception that you can only have tens of thousands of GPUs to

• 4:54:42Speaker 40

  train LLMs. Who knows what the architectures will be in the future, and what the next iteration of a transformative architecture change like CAFE will be.

• 4:54:57Speaker 40

  Foundation model. So Decaf showed back then the surprising effectiveness of transfer using frozen or relatively frozen AlexNet features. It was a pre-training, fine-tuning paradigm. Decaf was the precursor to CAFE, which became the de facto standard for deep learning in academia and industry. But maybe, and why was, why did I say that Decaf paper might be the most important one? I mean, CAFE had a lot of impact, but actually, the way I'm presenting the Decaf paper now as a kind of foundation-ish model, isn't what people were most excited about in 2015 to 2020. In fact, if you wanted to get your paper accepted during that era, you had to put end to end in the paper or in the title somewhere, right?

• 4:55:42Speaker 40

  The cool thing was I could now back propagate all the way from the task. And if you weren't doing that rapidly by around 2016, you weren't considered in fashion. So I don't think activation features, the way they were, you know, explained in the Decaf paper, or foundation models, if we relabel that today, were in vogue for several years. And we were all, and the CAFE system allowed you to now train your own model, get your own GPUs, and do this sort of end-to-end training. But as we know, roughly in the early 2020s, pre-training returns, and actually, in some senses, now the dominant paradigm.

• 4:56:24Speaker 40

  And we see this from BERT and CLIP, and now, you know, a perception that if you keep scaling your data and the model, the underlying representations are just going to get better and better and better, and that's the way to go. We don't think it's the appropriate path to, you know, really fine-tune from scratch for each task. We want to leverage everything we see across the underlying tasks. And maybe I'm oversimplifying what people were thinking in 2016 to 2020, but I think you get the gist. So we see now this pre-training paradigm very dominant in the field.

• 4:57:05Speaker 40

  Decaf was primarily a pre-trained plus fine-tune approach, as are contemporary LLM and LoRa models. But now prompting, pre-trained and then prompt, of course, is dominant in language and vision of language. And in vision, there's very early work along these lines as well. Since I have a test of time talk, I'll plug my own group's work on this, on visual prompting that we had at NEREPs 2022, and large vision models that we had at the CVPR. You can look at these approaches. No language in those models, but they're still foundation-ish models or pre-training plus prompting approaches to vision. And we see the unreasonable effectiveness

• 4:57:49Speaker 40

  of pre-training continuing to this day. Many models that are having a lot of impact. A lot of vision and language models coming out very fast from companies. I wouldn't want to now compete in building the next vision and language model from Berkeley. As I mentioned earlier in the talk, until there's the next revolution of architectures, and maybe we're not going to need 10,000 GPUs and we only need a couple of these, whatever the next great model is. I'm looking forward to seeing what that will be. But even now, I think that's still an open playing field if we consider vision and action, or vision and action and language. And so I'll just maybe close pointing to some of the work in that space. There are many papers coming out

• 4:58:34Speaker 40

  from many different labs right now in this direction. I'll point to two in my lab. One that includes language as it pre-trains a vision and language and action model, and one that doesn't explicitly include language, but also handles humanoid locomotion. If you're interested in the larva paper, we've taken the llama and lava base and added action pre-training into it, where we literally prompt the robot to have a particular control scheme, a particular task, and describe a trace of trajectories that are desired and that can then be performed. And the sort of fundamental approach of

• 4:59:19Speaker 40

  Of action or locomotion as next token prediction is explicitly formalized in our paper, humanoid control as next token prediction. No underlying language model here, but just pre-training on lots and lots and lots of human and humanoid action data, some of which are taken from the wild, some of which are generated in simulation, just straight-up transformer. And the model can then walk around new environments, including San Francisco. With that, I think there's one or two minutes left for maybe a question or a conversation, which I'm happy to engage in. And again, I think the team just wants to very much thank the community and the program chairs for this honor and looking forward to all the research in the future from the community.

• 5:00:04Speaker 40

  Thank you.

• 5:00:05Speaker 41

  Thank you so much. So I think there's time for maybe one question. There's time for one question. So if you have a question, you can come up to one of the microphones there. But then I think because we are basically between the two post sessions, we want to make sure we get over there too. So maybe we'll have one person, if you want to come up to one of the things, we'll give them a second because, you know, people always do it. Thing to do normally as a chair is to ask it yourself, but I won't take that. I won't take that. I'll let someone from the audience do it. So to a certain extent, this sort of notion of prompting, I guess, OK, I'll put it this way.

• 5:00:44Speaker 41

  To what extent do you think fixing features and putting a linear head on top of those features, which we see is very different from prompting in a sense in current mechanisms, to what extent do you think that that's just sort of a side? And prompting kind of is just the way we specify linear heads these days? Or to what extent is language really something fundamentally different when it comes to vision language models that is going to enable another step change the way that deep networks enabled us to step change so many years ago?

• 5:01:23Speaker 40

  I think you asked two different questions. One I took to be the difference between fine-tuning and prompting, and the other I took to be language versus not language, or at least I'll try and answer those two. And I think, and I'll try and do it quickly, I think I could also have added some slides at the end. I think it was an exciting time in the community and many cool papers coming out right now about the mechanisms of in-context learning and prompting, task vectors and function vectors, and how we can interpret and then maybe even patch or extrapolate these models. So I think that's unsettled. I think we're going to see in the coming years, papers that define that paradigm, that are

• 5:02:08Speaker 40

  going to show maybe more formal connections between fine-tuning and prompting in the architecture. That's very hot work right now. And then is language special? I sort of don't think it is. I also think the word "language" is complicated because I don't know whether, I think in the community right now, if I say a language model to the broader, like, the press, they're going to assume there's text in there. I'm not sure if I say the word "language model" to this community, whether you're going to assume that or not. In the past, I didn't always assume that. I thought I could have a language model on a series of tokens that were just vision tokens, and that's why we call that large vision model. A large vision model, I think that's a large vision model that's also a language model.

• 5:02:49Speaker 40

  So if language is just the process of having tokenized something and then predicting it, I think that general paradigm is going to be very high-impact across all areas of intelligence, including those that use text or what we normally call "language," and those that don't, like motor control.

• 5:03:10Speaker 00

  That last audience member question was an incredible segue into our next talk, which is a retrospective on how LLMs and computer vision converged, from Lucas Bia, who was one of the lead authors of the vision transformer paper at ICLR 2020. This would count as yet another deep mind talk on this pod. Prizes for counting how many we have featured this episode. Except that the entire VIT team has just left Google to set up the new OpenAI Zurich office. Let's have a look, eh? Get it? Look at how Lucas views the progress of the VLM field.

• 5:03:45Speaker 42

  I will talk about, yeah, about computer vision in the age of LLMs. And in this case, or in this talk, I will focus on each part. I will focus a lot about the data side of things.

• 5:03:59Speaker 42

  However, yesterday evening, I decided to completely redo my talk. And so I apologize if some parts are not smooth, or if sometimes I'm surprised by my next slide.

• 5:04:10Speaker 42

  So one thing that happened recently in computer vision, or recently, like four or five years ago now, is that suddenly, language has become the API for all vision models and things. And by API, I mean like input-output to the model, like how you communicate with it, basically. In the distant past, what a lot of vision and like classification is the most canonical task, but most tasks look like, is that you pre-train your model on a large database of images labeled with typically classes, because that's what's easy and quick to label. And then maybe there are a lot of classes, so you cover a lot of concepts, but they are not really attached, like it's the class ID number and that's it.

• 5:04:53Speaker 42

  In this way, your visual model learns to understand a lot of things, or at least to classify them into these classes. And then you transfer it to any task of interest, which could be again, classification, but maybe much more focused, like flower classification, or other things. And again, there you had like a labeled dataset, smaller, typically much smaller. And then you fine-tune on that, and then you get your model that you actually care about in the end. Then with the appearance of Clip and almost same time align, things changed. Like they showed how to do pre-training on not class labeled data,

• 5:05:34Speaker 42

  but pairs of images and text that you can find very easily.

• 5:05:41Speaker 42

  And then also not even like use this model, not even fine-tune it, but just... Prompt it, basically, or like, give it a few options in free-form text, and then it tells you which option is the most likely representing the image. And that way, you don't even need to, like, uh... The API, what I mean, is what I mean here, changed. Like, it's not, uh, integers of classes anymore, and it doesn't need to match or anything, and it's just free-form text. So that was very nice. Uh, and in terms of data, it means that it changed. Like, vision data sets, classically, were like this, a list of classes, and then for each class, you go and collect,

• 5:06:23Speaker 42

  uh, typically via image search, a bunch of images, and that's it. So you have this very regular structure or information content. Uh, and even worse, who makes up these classes? Like, is a PhD student just sitting there, "Ah, I'm gonna make a data set about blah blah blah, so let me think about the classes." A fun fact is COCO, which is like, probably the second most widely-used computer vision data set from the time where we did it with classes. Uh, the list... Who knows how the list of classes in COCO was created? Yeah, uh, or very few old-school vision people here. Um, it was the, the senior professor on the project asked his teenage kid, American kid,

• 5:07:07Speaker 42

  like, "What are common objects in your mind?" That's why COCO classes are like, uh, frisbee, football, pizza, and that kind of stuff.

• 5:07:19Speaker 42

  So, so yeah, you can see already, like, how bias, uh, comes into these data sets, right? It's just, it... The model only learns about the things that the person creating the data set, which is usually one random person that just happens to be deciding, uh, thinks of.

• 5:07:40Speaker 42

  But this is what the data now, in modern times, when we learn from image-text combined, looks like. It's like just random collection of images, typically from the web, and text somehow attached to them, like typically alt text, or the title of the page it's from, or things like that. And then you have some random shit like this, that is completely uninformativ, "thumbnail for version as of 21 blah blah blah", and this is legible, yeah. So this is kind of useless supervision signal, but you also get very detailed stuff that you would never come up with if you were to create a list of classes, like this one, "Frankfurt Airport skyline 2017", right? Or "London barge race", or things like that, right? So this completely changes what the models can learn, right?

• 5:08:25Speaker 42

  They are exposed to a lot more noise and useless stuff, but at the same time also to a lot more detail that you would never come up with in the classic way of creating data sets.

• 5:08:38Speaker 42

  Alright, and then a little advertisement. After CLIP came out, or CLIP was the first model doing this, and then a couple of years later, our group made this SIGLIP, which is a variant of CLIP, and is also open model, so you can download it and use it, which is just, after a few years of experience with this, it's significantly better. And the cool thing is, again, like, as with CLIP, but now even better, you can prompt it with free-form text and ask about, become as detailed as you basically want, as you can express with text. Here's a couple of examples. Are these visible-ish? Yeah, here's a couple of cool examples of pictures we took ourselves, so they are not possibly in the training data set and the model doesn't possibly know about them. For example, this one is me and the colleague.

• 5:09:23Speaker 42

  We have bought a coffee-themed t-shirt. Mine, I think, said, "I need coffee", or something, and the colleague is just the molecule of coffee. And then the model fires 100% on the text, "A photo of two guys in need of caffeine", but it fires 4% only, "A photo of two guys in need of water", so this stuff works nowadays. And a classical thing in computer vision, at least until recently, is a pet peeve of mine. People always say, like, "Oh, computer vision models are not robust", "It will recognize a cow, but you put it on the beach, it will fail completely", and not at all. But this has been solved for years. Like here, cow on the beach is 99%, and cow also 36%, but cow in prairie is only 1%.

• 5:10:07Speaker 42

  So this stuff works with CLIP and SIGLIP and models like that.

• 5:10:12Speaker 42

  When we did SIGLIP, we released a separate checkpoint that we trained on all the languages.

• 5:10:20Speaker 42

  And we basically tried to show that, here just from some examples, we didn't really thoroughly evaluate it in the paper. We just released it and tried to show that not only does it learn multiple languages, like from this just web data, images and text, the web is international. So you just learn about all the languages for free essentially, if you use them. So we didn't do anything special like translation or anything, and the model can learn like not just the cow on the beach, but also in other languages like "une vache sur la plage", or "in the Quamstrand", and the other languages I cannot pronounce. But they all say the same thing, right?

• 5:10:57Speaker 42

  And here even we tried to show some cultural, specific things. I think this, this was my Chinese colleague Xiaohua who came up with this, so I think only Chinese people will understand this. This dish in Chinese, it's called "ants on the tree", I think, or "ants on the branch", or something like that. "Ants climbing a tree", exactly.

• 5:11:21Speaker 42

  And so if, here this is just the interesting thing, if you ask this model in English, and like the literal translation of the dish "ants climbing a tree", it just doesn't get it. It just fires on the picture of ants climbing on a tree. And not on this dish, but if you ask, I cannot say this, but one of these is like "ants climbing on a tree" in Chinese. If you ask that, it totally gets that you're not talking about like literal ants climbing on a tree, but the dish, ants, like that. Alright, so...

• 5:11:53Speaker 42

  Ah, yeah. So we released this separate model, multilingual, but only... Back then, we trained only a small version of the model. And recently, we also trained the larger version of the model. And this is new, like, we released this somewhat silently, just in this collab. So now, if you're interested in international siglip or clip-like models, there is a large one available that is pretty good.

• 5:12:20Speaker 42

  But why did we release two separate models? Why not just the one international and that's it? Well, because it turns out that training on English-only data helps a lot. Scores that people, including some of my colleagues, care a lot about, which is just ImageNet zero-shot and a few other English benchmarks. So just here, overview of a broad range of recent clip-style papers or that do clip and look at data. It typically looks like this. If we train on raw data, it's bad. English-only subset of the data, it's better. Some more filtering, it's better and better. And the measurement is done typically in ImageNet zero-shot score. And this is across...

• 5:13:04Speaker 42

  Like, I intentionally don't write which paper because I don't want to blame any individuals. Except here, I can say this is the original clip paper already. It says we get our queries from English Wikipedia, so English-only.

• 5:13:17Speaker 42

  Then, when you see papers using Lyon, they usually use Lyon 2B. But if you look at the citation, the Lyon paper is Lyon 5B. So what is that? Well, Lyon 5B is actually 2B English, 2B non-English, and 1B don't know. So typically, people just use the 2B English, Lyon subset. And then here is another work where we can go through steps of filtering, right? We see first, basic filtering. Like, the first basic filtering is the caption language being English. And then, typical thing is filtering by clip score. So you keep only the data that Clip already understands, which, as we've seen before, as I mentioned, Clip was trained on English-only, is English-only stuff. And then there's more.

• 5:14:02Speaker 42

  Then there is like, keep only the data where in the text...

• 5:14:09Speaker 42

  One of the words in the text is from the ImageNet 21K list of classes. Or even better, not even the text, like, keep only images which are similar to ImageNet images. And the thing is, this more heavy English and ImageNet tailored filtering is what works best. As measured on ImageNet, but also other benchmarks, but which are similar to ImageNet.

• 5:14:36Speaker 42

  Having said all of these negative things, I need to call out one positive paper and not hide its name. InternBL, I like this a lot, like they specifically show, okay, look, we use Lion, the English, but also Lion multilingual and also Chinese dataset, so that was nice. Alright, so this was all about CLIP and about this specific filtering stage, one of the next talks from Angelina will give more details about the effects of this.

• 5:15:06Speaker 42

  But Vision Community, part of Vision Community has moved on and I think all should move on, past CLIP or beyond CLIP and even CGLIP. Because there's some things, no matter how good your data is, how high quality the caption, how descriptive the caption, there is some things the CLIP contrastive loss just doesn't learn. Actually, I just assumed everybody knows how CLIP works, but who knows how CLIP works, the training? Okay, more or less everybody, that's good. So I take this example, you have the image of a cat and a dog and the caption is pretty much, well, it could be more detailed, but it's pretty much perfect, like a cat sitting left of a dog.

• 5:15:50Speaker 42

  Now, these go through the encoders, right, and then they are trying, they are trained to be most similar versus other captions or pairs in the mini-batch.

• 5:16:01Speaker 42

  However, now let's think, what does the model need to learn to perfectly satisfy this objective at training time? It depends on what else is in the batch. If there is no other picture of any cat or any dog in the same mini-batch, the model just needs to learn, for example, cat, and that's good. It matches it to this image, perfect, done. The loss is perfectly satisfied. Or alternatively, it just needs to learn dog, and then it's done. This is the only image that contains a dog. It doesn't need to learn to match any more than that. And models are lazy, just like me. They learn the minimum amount necessary to solve the task.

• 5:16:39Speaker 42

  Now, if there happens to be another picture of a cat in the same mini-batch, if it's not sitting, then the model now only needs to learn either cat sitting or, probably easier, cat and dog. Like, it just matches the word "cat" and "dog" with this image. There is no other image with both a cat and a dog, and it's done. It doesn't need to learn more. You see where I'm going, right? To learn left of what needs to be in the exact same mini-batch, the same thing, like a cat and a dog, but the other way around, with the perfect caption and no other shortcut to match them. This is just not gonna happen. So this is like an inherent disadvantage or limitation of clip-style learning.

• 5:17:23Speaker 42

  And SigLip suffers from that too. So with some colleagues, we set out to, like, find a fundamentally better learning objective that fixes that, and there is a quite simple one that does, which is just captioning. So encode the image, and then pass the image encodings into a decoder that should decode the caption. When you decode, it's just like a language model. Loss is next token prediction. So when, here, you do have a loss that says, "Say left. Don't say right. Don't say above. Don't say below. Don't say any of the other words in your vocabulary. Say left." So the model has to learn this.

• 5:18:05Speaker 42

  And then there is... I'm not gonna go into any more details here, but original clip paper showed this in their Figure 1, that this is so much less efficient to train this way. And in our paper, we also go into a lot of detail in that. It's not much less efficient, actually.

• 5:18:25Speaker 42

  Right, and then we evaluated this. So then we found that actually we are not the only ones thinking about this clip limitation. And there were already multiple benchmarks that specifically measure this clip limitation. The first one we saw was ARO, which stands for Attribute Relation and Order. Like three things that clip models are not really incentivized to learn much. So they designed a benchmark to test exactly that.

• 5:18:52Speaker 42

  And ignore the numbers in the bottom. So when we train a clip style model, we get these numbers. When we train the captioning style model on otherwise exactly the same setup, like both we optimize a lot and they are trained on the same data. This is just so much better. This is worlds better. This is also worlds better than the bottom numbers are a few ideas of how to fix this in clip, which I call mandates. And this is just so much better to train a captioner. Some of these like ordering is just perfectly nice. I actually, no, not this one, next one. But here is an example from the paper, from this ARO paper. I am currently hiding.

• 5:19:38Speaker 42

  So the way the benchmark is constructed is you have an image and two possible captions. And you need to find which one is the correct one and which one is the wrong one. And the captions are designed to specifically differ in either an attribute or a relationship like left off, right off, or ordering. This is example from the paper itself. The horse is eating the grass or the grass is eating the horse. Can you guess without even seeing the picture which one is probably correct? And matches the picture, yeah, right? So this is an issue. You probably, hopefully guessed this way around. So this is an issue with the benchmark. I don't even need to reveal the image, but just for the sake of completeness. And this is just screenshot from the paper.

• 5:20:24Speaker 42

  So we identified this as an issue. And so we also train a blind decoder, just a captioner that never sees the image on our same pre-training dataset of images from the web or image text from the web. And of course that nails the task too. So this is a shortcoming of this first benchmark. But again, we were not the only ones to notice that. Other people. Notice that too and created a new benchmark that is supposed to measure the same things, which is called the Sugar Crap.

• 5:20:57Speaker 42

  And it looks kind of like this. Here is an example. It looks to, it seems to not have these obvious shortcuts. Just one example is here, this picture, and then a yellow tennis racket has a blue tennis ball on it, or a blue tennis racket has a yellow tennis ball on it. Like, both of them are quite plausible. And same with the cake and flowers and things like that. And then it also goes into more details of what things are tested. But the same story here. So here we did, I think we didn't include it in the paper, because the authors of the benchmark already did this blind baseline and showed that it got random accuracy. So we don't need to redo that. But here, same story.

• 5:21:39Speaker 42

  Like, these captioning models are significantly better than the equivalent clip models, or than even the best clip models on almost everything.

• 5:21:50Speaker 42

  Right, so I think this is the future of pre-training models, or something like that, but we should move past script.

• 5:21:58Speaker 42

  Right.

• 5:22:03Speaker 42

  Even more, as if vision models, just like language models, I know getting more and more, or becoming more and more complicated systems. Everything I said before is like, pre-train one model and then we can use it, maybe zero-shot or fine-tuning. But what most models do now is training in multiple stages. And for VLMs, it's basically almost the same as for language models.

• 5:22:29Speaker 42

  From our side, we started this a few years ago with a series of papers and models called Pali. Actually, curious, who knows Pali?

• 5:22:39Speaker 42

  Ok, como medio. Probablemente la gente que hace Vision y Purel y la gente que no.

• 5:22:46Speaker 42

  Así que el modelo de Polly parece así, o al menos fue una serie de papeles y modelos, y esto es de la primera papeza de animación. Así que solo tienes una imagen y texto como entrada y luego texto como salida. Y el texto que es la entrada básicamente es la tarea, como ¿qué quieres? Es a menudo una pregunta que no se responde sobre la imagen o una instrucción como generar un capitulo de esta imagen en romano o cosas así. Y luego esto simplemente va al Transformer y se treina juntos. Entonces, hablaré sobre cómo se treina en un rato. Y luego con este tipo de interfaz de modelo puedes hacer muchas más cosas que solo con el clip o solo con el capitulo o modelo, ¿verdad?

• 5:23:30Speaker 42

  Ahora puedes hacer preguntas y porque es un lenguaje de forma libre y no una lista de clases, puedes hacer preguntas bastante precisas, como puedes preguntar ¿cuántas monedas hay? Y luego dice 12, pero también puedes preguntar ¿cuántas monedas de 1 dólar hay? Y puede decir dos y otras cosas.

• 5:23:48Speaker 42

  Y luego, sí, vamos a saltar eso.

• 5:23:52Speaker 42

  Correcto, pero luego solo hay texto de salida y ok, la gente de la lengua de modelo aquí estará feliz con eso, pero la gente de Vision estará como no, hay mucho más en Vision que texto de salida, ¿verdad? Pero el texto es un poco más universal de lo que podrías pensar. Por ejemplo, una tarea de vision clásica es la detección, crear cajas de límites con coordenadas, esto es muy fácilmente encoded como texto. No, es algo legible. Así que ¿cómo encodes un caja de límites como texto? Bueno, solo como las coordenadas de los dos lados y solo como números enteros, por ejemplo, ¿verdad? No necesita, los números enteros no son píxeles porque entonces es sensible a la tamaño de la imagen, pero como fracciones de la imagen y luego x 1000 para que tengas números enteros.

• 5:24:35Speaker 42

  ¿Verdad? Así que en realidad puedes hacer muchas tareas de vision de computadora clásica con este API de texto también. Aún más, no lo puse en esta diapositiva, pero también puedes crear máscaras de segmentación como salida de texto. ¿Cómo funciona eso? Bueno, puedes entrenar un encoder de máscaras, típicamente VQVIE, que puede compresar una máscara en un código corto de unos pocos tokens de un pequeño vocabulario y luego puede decodar eso. Y luego simplemente concatenas este vocabulario a tu vocabulario de lenguaje y eso es todo.

• 5:25:09Speaker 42

  So it is actually a very universal API. You can do many vision tasks with it. And again, because it's using language and not as in the classic vision segmentation and detection, the list of AT COCO classes, you can be quite precise in what you want, right? Detect the right hand and it only gives the right hand. Detect the left hand, only gives the left hand. Okay. Let's not discuss about what is left hand and right hand.

• 5:25:35Speaker 42

  According to the training data.

• 5:25:39Speaker 42

  Okay. Yeah, one issue is that, like, we had the whole series of like three papers about Pali models showing all of this is possible and this can get better and better and so on. But then times have changed and nowadays people are like, "Oh, nice paper. Where is the model? Give me the model." Otherwise, I will forget about it in a week. So, yeah, this is a good question. And so we did the fourth GEMA model, which is called Pali GEMA, and this one is also open. So you can just go and download it and use it for almost all purposes. We had earlier, like, some licenses say, "Don't use it for evil stuff." We had to use such a license too. So, but you can essentially use it for anything.

• 5:26:27Speaker 42

  What it looks like is pretty similar to the previous one, just slightly different, because now language models are all decoder only. So we use a decoder-only language model, the GEMA 2 billion, and then image encoder.

• 5:26:43Speaker 42

  Yeah. And then, and then let's go to the interesting part, the training.

• 5:26:50Speaker 42

  This slide I copy-pasted from another presentation. Let's just ignore the left-hand side. It's not important for this talk. The pre-training, like, this works in multiple stages. And this is quite similar, I believe, to the language model pre-training also now. So first is the stage zero, which is uni-model pre-training. So the image encoder is pre-trained by itself. We did it with the SIGLIP image encoder. You can use a Kappa image encoder. You could use a DINO image encoder. Anything but a good general image encoder. The language model is trained by itself. Like, in this case, we use GEMA, because we are at Google. You can use LAMA, you can use anything else.

• 5:27:34Speaker 42

  So for this you pay zero cost, because you just download them, like existing ones. Then you do a stage one, we call this is multimodal pre-training. So that's when you stick both of them together. And then you train them on the mixture that looks like image and text in, and then text out. And I will show you the mixture later.

• 5:27:56Speaker 42

  Then in computer vision, it's often important to also for the model to understand higher resolution images. So typically, we train on 2:4 by 2:4 images for traditional reasons, but it's also a sweet spot. Like 2:4 images, you can recognize a lot, but not everything, and it's relatively efficient. But then there is usually a resolution increase stage, which is shorter training at higher resolution, like 448 by 448, for example, because it's more costly, but you can see more details. Like especially if you have images with texts, like pictures of documents or whatnot, then you may really need that. Right, and all of these are basically the pre-training. And then there is another stage, which is a transfer.

• 5:28:41Speaker 42

  So the pre-training tasks, and you will see shortly, are mostly designed to teach the model as many skills as possible and as broad knowledge as possible. It doesn't really, in this stage, you don't really care about the interface being nice, about it understanding user intent well or things like that. Just about putting raw knowledge into the model. And then you have a transfer task stage, which is usually also shorter, where you fine-tune usually the model to what you actually want. And this can be different for different people or companies or projects. And this can include training on mixtures of many things,

• 5:29:25Speaker 42

  like supervised fine-tuning or instruction tuning is part of that too. But it typically doesn't have the goal to give new knowledge to the model, but just to make it focused on the thing you care about. So the pre-training mixture, in this case for Paligera, looked like this. Basically, a bunch of tasks that force the model to learn some things. The one obvious one is like prefix means what is input, like prompt to the model or task description, and then...

• 5:29:59Speaker 42

  So for example, we have a caption and then the language. So caption in Chinese, for example, and then the model needs to predict the caption in Chinese. From the raw data, raw collection of image text from the web, we can just run a language detection on the caption, right? And then we know the language at training time, and then we can put it here. Or for example, if we have pictures that have text in it, and we know about the text that's in the picture, which we can know, for example, with existing OCR systems, then we can ask the model to just read the text that's on the images. So the prompt would just be, "Do OCR." That is one task.

• 5:30:42Speaker 42

  You see, that teaches the model a different skill than describing the image in the caption, right? And then a question answering, including some specific questions which you can generate. For example, if you have an existing pretty good classifier that tells you which kind of classes or objects are in your image, you can run that and then generate synthetic questions like these ones. Like, "How many chairs?", for example. Or, "Is chair in the image?", or things like that. And then there was another paper previously that showed you can also turn them around,

• 5:31:27Speaker 42

  like, generate the question that would give this answer. And this is a different skill set that the model needs to solve this, so it is also good to add to the pre-training. And then we also added a detection and segmentation. The detection labels and segmentation labels are pseudo-labeled, so they come from a good detector model or from a good segmenter model.

• 5:31:53Speaker 42

  Yeah, so this is kind of what the mixture looks like. But this is not really how you want to give the model to the user to use it, right? You don't want the user to first have to type answer EN and then the question of the user.

• 5:32:09Speaker 42

  So that's where then this fine tuning step comes in. And we don't need to go through this whole list. It's just to say like we fine tuned this on a lot of different data sets. It works really well and for fine tuning, you don't need a lot of fine tuning data, because it's mostly about like rewiring the syntax to be aligned with what the task needs.

• 5:32:38Speaker 42

  And then the final step, which from language you also know, but we actually did this at the same time as the RLHF. But in vision is to have a last step of RL tuning of the model to optimize for what you really, really want. Because the supervised fine tuning still usually doesn't optimize for what you really want. Let's see, how can I give an example for that?

• 5:33:04Speaker 42

  Right, let's go back to this example here. If you would do supervised fine tuning on a data set like that for detection, your training objective is to predict each of the tokens precisely one after another, right? But when you do detection, so this task, for example, 298 here, or if you predict 299,

• 5:33:29Speaker 42

  it would be completely wrong to predict 299, right? It's like you predicted the wrong token, so you're wrong, that's it. But in detection, that's not really what we care about. If the box is like one pixel more to the left, that's totally fine, right? What we rather care about is for example to not have one extra box where there should not be any at all, which in terms of tokens would be the same amount of error as getting four of the box of the coordinates off by one pixel. Alright, so what the, what is trained typically in supervised learning is not what you really care about. Was that example kind of clear? Yeah, I hope, okay.

• 5:34:10Speaker 42

  So then, also in vision, what we can do is the last step of RL tuning. So first, and yeah, this was almost exactly the same time as the RLHF paper.

• 5:34:25Speaker 42

  So first, you do the supervised training, or the supervised fine-tuning, or pre-training, because that just works really well, and that does give you a reasonably good model, a reasonably good approximation of what you want.

• 5:34:39Speaker 42

  So that is the maximum likelihood training, which means basically just imitate the training data. So you can also never get better than your training data, or than the best part of your training data with this.

• 5:34:52Speaker 42

  But then, once you have this model that does reasonably good at your task, you can sample from it predictions, and you can now define the reward. And the reward does not need to be differentiable. That's the nice part. You just need to give a number, like is this prediction good or bad? And this can be arbitrarily complicated to get this number. It can even be gotten by asking a human to give a number. Then you have RLHF, for example. Or it can be by going through a very complicated metric. Like the people familiar with detection, they know MAP is the metric that pretty well describes what we want in detection, but it's definitely

• 5:35:33Speaker 42

  not differentiable and it's quite complicated. But you can just compute this, and you give a score to the sample, and then you do RL, which basically means, okay, model, give me like two samples, and then I give a score to both samples, and the one which is better, which scores higher, I say, model, sample this more often, and that one less often. And you keep doing this, and this is the way you can align the model to do exactly the task, or the part of the task that you actually care about. Not, so not just copy what is in the data, which is what the pre-training, the supervised training does.

• 5:36:14Speaker 42

  This was relatively clear in language that you can do this, because in language it's super common now to have models where you can sample from. In computer vision, this used to be completely uncommon. Like, all of the classical computer vision models, like Faster R-CNN, DeepLab, YOLO and so on, are not models you can sample from. So you can not do RL on it. Because you cannot get two samples from the model and say which one was better, which one was worse. It's only recently with this unification of models and this style of models like Pali, and there have been a few others, like Unified IO is another good example, that you can actually have vision models that can sample multiple reasonable solutions, and then you can do RL on top of.

• 5:36:57Speaker 42

  So that's why this only happened recently. Yeah, and here's just a few examples that it works pretty well. We did it for detection. So the left is the base model, and for those who know detection, it gets a COCO MAP of 39, which is okay, but not great. And then you do a little bit of RL tuning

• 5:37:20Speaker 42

  with MAP metric as the score, and then you get much better MAP. And indeed, the detections, you actually catch a lot more things. And 54 is a pretty good COCO MAP. And we also did it with Panoptic segmentation.

• 5:37:37Speaker 42

  And just to demonstrate, you can really, like, you just need to come up with, define clearly what you really want, and then come up with some score of it. We just did this silly example of a colorization model, so grayscale image in, color image out. And it's also generative, so you can sample from it. And then we just arbitrarily define a metric that computes the flashiness of the image, and then we RL tune a bit towards that metric, and then indeed it generates flashier images.

• 5:38:09Speaker 42

  Right, then, uh, one last thing about this, uh, to show that what kind of happens, uh, in the RL tuning. It's really not teaching the model any new things or anything. It's just making it sample more the things that you like, that you score highly, and sample less the things that you don't like, uh, that score badly.

• 5:38:32Speaker 42

  So here, here are a few plots, uh, they are all a little bit hard to digest, but I will try, uh, to work you through. Uh, so,

• 5:38:41Speaker 42

  we have the model before means before RL tuning and after is after RL tuning, and on the y-axis is the reward of the task, uh, whichever it is.

• 5:38:51Speaker 42

  And we get a lot of samples from the model before and a lot of samples from the model after. I think here we got 10,000 samples. And then we just sort them. And what you see here, uh, is that before, you had a lot of samples of low-quality samples, and after RL tuning, you told the model, like, this is literally what RL tuning is, right? This sample is bad, less of that. Um, so you have way, way, way fewer low-reward samples. And you start off with sampling by default many more high-reward ones. However, the raw model before RL tuning also had a very few high-reward samples, like this little green dotted line, right?

• 5:39:33Speaker 42

  So it's not that the RL tuning makes the model better, it just makes it sample these good parts more often, right? The original model was able to be just as good as the RL tuned model, but just very, very rarely. Um, let's see.

• 5:39:56Speaker 42

  Oh yeah, and, uh, this one is just that the, the likelihood of a sample is not enough. Like, you really need to have your score, uh, that you define. Um, so here is, what is it?

• 5:40:12Speaker 42

  Right, here, from left to right, we sample more and more samples. On the left, it's like, just, let's say, two samples here. Then, what the curve shows is the highest reward across these two samples. So, what was the reward of the, one of the two samples which scored the highest?

• 5:40:30Speaker 42

  And we can see the same story again, basically. Oh, no, wait.

• 5:40:35Speaker 42

  Sorry, I misspoke. Let's rewind. Here you have two samples, and here you see the reward of the sample with the highest likelihood. And before, before RL tuning,

• 5:40:53Speaker 42

  that is not really good. The thing is, even if you get many samples before RL tuning, and 10,000 samples or 100 samples, and you pick the one with the highest likelihood, you're not getting better samples in terms of the reward. Because the likelihood is not really aligned with your reward yet, right? So you sample more and more things, but that are not in the high-quality region. And this is what the reward tuning does. It re-weights the likelihood of samples to sample much more high-quality samples.

• 5:41:26Speaker 42

  Alright, and that was too much, alright, too little data in the end. So, this is the end of it. Thank you.

• 5:41:36Speaker 00

  We're going to bring part three to an early end here. Brittany had one more vision-related paper to highlight MLM as a judge. Assessing multimodal LLM as a judge with vision language benchmark, which we appreciated for practical AI engineer use, but unfortunately, we had to cut it for time. You can see their oral presentation in the show notes. Last but not least, we combined parts one, two, and three across world simulation, generative modeling, and vision to check in on the field of reinforcement learning and robotics, which took almost as big of a stage as video generation at ICML this year. For a natural transition from vision to robots, we turned to Ashley Edwards, who was on the Gato and Genie team at Google DeepMind,

• 5:42:21Speaker 00

  but is now at Runway, emphasizing the deep connection between generative video and the world simulation that is essential for diffusion and robotics.

• 5:42:31Speaker 12

  So yeah, today I'm going to be talking about how we can learn actions, policies, rewards, and environments from videos alone. So just as a little bit of a disclaimer, I'm going to be talking about a lot of my prior works, some of which I thought I would never talk about again, others I thought I would never talk about at all. But a lot of them have motivated me to the kind of research that I've been working on these days. So I thought it would be kind of fun just to go back and look at some of the history that led me here.

• 5:43:01Speaker 12

  So I think we've probably seen iterations of this kind of slide throughout this entire conference. But I think we know by now that there's been a lot of progress made in text-to-video generation. And one question we might be asking is, like, how the heck did we get here? I mean, I think just this past year alone, we've seen so many innovations. I hope that many people during this conference will be discussing this, but it won't be me. Instead, I'm going to be talking about how did I end up getting here? My research background is actually in reinforcement learning. But suddenly, I found myself in the controllable video generation space. So this is why I wanted to talk about some of my older works, because I wanted to see, like, how did I end up getting here? Maybe some of the things that I was working on are still relevant today.

• 5:43:48Speaker 12

  So in order to answer this question, I'm going to take us back to the summer of 2016, where I got to spend a summer in Japan. So my main focus here was to actually work on this robot here. So I started off actually as a robotics major. And what I wanted to do here was essentially try to train this robot to learn sign language gestures from videos. And this is when I was really started getting interested in how we can train agents from videos, because coming from a reinforcement learning background, I started to get kind of annoyed with having to always come up with a reward function for training our agents. And every time we had a new environment, we had to come up with a new... Reward function.

• 5:44:30Speaker 12

  And so I was really interested in how we can come up with like a more sort of general way of representing tasks. And that could be done through videos. And so when I arrived at the university, so this was at Waseda University, I realized that the hands on the robot weren't actually working. And so I wasn't actually going to be able to teach it uh uh hand gestures from videos. But this robot was actually like a very expressive robot. I think it was like a comedian actually kind of robot. And so it had a lot of different facial expressions that it can make. And so instead of teaching hand gestures, I decided that, okay, well, fine, I'll try to teach it facial expressions.

• 5:45:08Speaker 12

  So if you think about, if you if you look at what humans look like, they don't look anything like this robot look like. Uh, and so the thing that I was trying to figure out here was how we can actually teach a robot to mimic a uh uh, yeah, this robot in particular, to mimic a facial expression like this when the features look very different. And again, this was like in twenty sixteen. Uh, and so, I mean, we had like a few examples, like one GPU and that sort of thing. So we didn't have a bunch of examples for trying to learn a representation here. Um, and so what I wanted to try to do is figure out how we can put the space, uh, the feature space of the robot to look more like the feature space of of the human. Um, and so one thing that we realized was that if you look at the sort of shape

• 5:45:51Speaker 12

  uh of motion over time coming from from um these facial expressions and in general, any kind of motion, um, there is a bit of a structure. So this here is showing uh something called a motion template, which essentially takes a sequence of frames, uh, sort of concatenates them and averages them over time. So but you can see where the motion has happened as well as when the motion happened in time. Um, so this is what this representation is showing. Um, and the nice thing is that this representation is kind of domain agnostic. Uh, so you can see on the left, for example, you can see the motion of the robot. On the right of that one, you can see the motion of a human. Um, and then again, we saw we had two different tasks, one is smiling, one is surprised. Uh, again, this was like a workshop paper back in the day of, you know, it's like, whatever, I though I thought this was kind of cool.

• 5:46:35Speaker 12

  Um, but it's like the most, like the best kind of results here. Uh, but essentially what you can see is that the shape is similar across these different tasks. Um, and so it like kind of learns how to smile and kind of learns how to make the surprise face, uh, because we're trying to basically mimic the the motion that you see here rather than mimicking the actual features that you would see in a human versus like the robot, uh, if that makes sense. So I guess one other thing about this work was, so we essentially had to sort of hand specify our reward function. We're using hog features to compare the humans' motion template to the, to the robot's motion template. And it was a single task. So we were trying to learn a facial expression from a robot to a human.

• 5:47:19Speaker 12

  But after this, we started getting more interested in how we can learn sort of representations across multiple environments rather than focusing on this single task. And so this is when we started working here. So we're trying to actually learn behaviors from videos. And so in this work, what we did was we essentially got, actually, yeah, we had like a giant data set of publicly available internet videos back in 2017, but it was actually showing a video game playthroughs, mostly consisting of speedruns. But what we wanted to see was if we could try to infer the behaviors that were taking place across these environments. Because you can imagine in these video games, you might see characters moving to the left, moving to the right and that sort of thing.

• 5:48:04Speaker 12

  And so the idea was that if we could infer those behaviors, then we can use them for generating a sort of controller for Asians to say like, when I see this new scene, I want to generate what I want you to do. Again, this is the workshop, so we didn't get to that second part. But we did get to the actual trying to generate these motion templates. So all of these are showing, give an initial scene. Let's generate the motion templates. That's what I can generate, new, new motion templates. Given unseen scenes. And so this is showing some of the results here. And so on the top, you see the, the video game generations coming from, coming from training on that data set. These are unseen environments. You can see it's kind of starting to extract the motion happening across these different scenes. It's probably kind of hard to see, to be honest. But the other interesting thing that we found

• 5:48:49Speaker 12

  was that we could use that same model that had been trained on video games and actually worked really well at segmenting out like animals from unseen environments. And we, we had only trained on video games. But this was like kind of one of the emergent behaviors that you see by predicting motions, things that are going to change over time. You're actually able to sort of extract out these different characters.

• 5:49:16Speaker 12

  So one other interesting thing here was that essentially, instead of trying to predict a single mode, so instead of having your loss being on your next frame generation, we found it was useful to actually try to predict multiple futures. So essentially, what you see here is like, on the left, this is our initial frame, and on the right of that, you see all of the different kinds of generations that are happening. So if you squint enough, you can see, for example, that you can predict moving to the right, or moving to the left, or moving up or down for each of these different scenes. And we found that this was happening consistently. The way that we trained this was essentially to try to take each of these generations

• 5:49:57Speaker 12

  and minimize the loss between one of those, the closest one to the ground truth-generated frame. So we're trying to cluster over our different future predictions. But the interesting thing to take away here was that these different kinds of motions that we're seeing actually represent actions. And so I think the thing that we started to figure out was that actions are kind of a shared representations across these different scenes. And so rather than trying to explicitly represent those through like the motion template that we tried before, we wanted to see if we could actually just try to infer actions alone from the videos.

• 5:50:38Speaker 12

  So that was the motivation behind our work, Ilpo, where essentially, we're going to try to actually learn actions and policies from videos alone. So the way this worked was essentially, so imagine you have an initial frame like this. You might see in your dataset, again, we're going to be trying to learn from videos and train agents to learn to imitate from those alone without actions. But there you might see, for example, a transition that looks like moving to the right, or looks like jumping in the air. And so what we were trying to learn here was something called a latent action, which is just going to be essentially the kind of notion of what caused this transition to occur. So we know that something caused them, we don't actually know the action labels of these, we're going to try to learn them from the data.

• 5:51:21Speaker 12

  And then we're going to have a latent policy that's going to be defined as the likelihood of the expert taking some latent action in any given state.

• 5:51:32Speaker 12

  So essentially, the way that we learn this is imagine in our data set, we see these two sequences here. So let's say the expert moved to the right, for example. What we're going to do is we're going to learn a generative model to again predict each possible next state, given your your initial state here. And essentially, what we're going to try to do again is we're going to try to cluster over all of those potential next frames by looking at the the generation that's closest to the one that was actually shown in the data. So we're going to again see this sort of min loss that says, let me look at all my latent actions. I'm going to find the one that looks closest or the generation that looks closest to the ground truth one. So we're clustering over our future frames here.

• 5:52:15Speaker 12

  And then what we're going to do is try to learn a policy over all of those different transitions that we can see. And so the way that we can do this is let's say, for example, in our data set, we observe that half the time, for example, the expert moves to the right, half the time they jump in the air, or they never stay still. So we're going to try to learn a policy that ends up looking like this. So if you're going to average over all of those future frames, you might see something that looks like that. And so we're going to try to learn a policy that effectively weights all of the different futures coming from our generative model. So that's that if you were to take the expectation under that policy, you would end up having a generation or an average generation that

• 5:52:56Speaker 12

  looks like the expected generation coming from our expert or the expected future coming from our expert generations. And so that's essentially how we can train the policy. So each of these different weightings over the future is actually saying, what is the likelihood that I would take latent action zero in this state, take a latent action one in this state, for example, and we can train it in this way? So yeah, so this is actually showing just after 200 steps of interacting with the environment that our that our model is able to adapt really quickly. And the reason for this is that we're actually learning this policy from the videos before we ever place the agent in the environment. And so we can take some steps from the environment samples and

• 5:53:37Speaker 12

  use those for actually adapting our latent actions to the real world, real ones that you can take in the world.

• 5:53:46Speaker 12

  So one thing to take away from that work is that we can actually represent our actions through the next frame generations that are taking place. Of course, this is assuming that your dynamics are deterministic. Well, let's say that they are. But basically, each of these next frames are representing the kinds of actions that you can take in the world. So we took this idea sort of in a different direction where we could say essentially, let's say we have a reward function. We can now try to learn a value function, an optimal value function, from videos alone, even if you have suboptimal data. So for example, if you have demonstrations like this coming from videos

• 5:54:28Speaker 12

  where the expert isn't really an expert, but they're running into things and doing suboptimal things. But sometimes they run into the right, run into the goal. And so the idea here is that usually in reinforcement learning, you can learn an optimal policy from suboptimal data. But it gets a little bit trickier in videos because you don't have access to actions. So the idea here was to, instead of learning an action, or sorry, a policy over actions, which you would typically see in reinforcement learning. If you do RL, I know this is a video generation sort of environment here. But if, some of you might be familiar with a diagram like this, where essentially you have an agent running around the world. It's taking action and it's trying to maximize its long-term expected reward as a policy over states.

• 5:55:10Speaker 12

  The idea behind this work was instead of having a policy over states, sorry, yeah. Instead of learning a value function over states, you would learn a value function over state, next state pairs. So essentially what we have is this value function. Sorry, I think I messed that part up. You would usually have a value function over state actions. Now we're learning a value function over state, next state pairs. We're learning a policy now over states rather than a policy that's going to tell you which action to take. And then the benefit of that is that you can actually learn this in an optimal way when you have suboptimal data. So this is a lot of different stuff that I'm showing on the screen. But the main takeaway, again, is that we're learning this policy over states, learning a value function that says, what is the value from transitioning from one state to the next,

• 5:55:55Speaker 12

  rather than what is the value of taking an action in a given state? And then we can essentially try to train this policy that's telling us what state do we want to transition to by maximizing our value of moving from one state to the next.

• 5:56:10Speaker 12

  So the other thing that we need to do eventually during when we're actually interacting with the environment is that we're going to have to figure out where the actions come from. So we can also learn an inverse dynamics model. So that's what that's showing there. So again, what we're learning is like given suboptimal data, we can actually learn optimal generations. So this is showing plans coming from our policy over states here that's saying, what state should I move to if I was going to maximize my value? And one interesting thing here to take away is that this is actually basically like a video generation model, like we're trying to generate next frames that tell us how are we maximizing our value? And this is given just random generations, like random rollouts from behaviors, we're

• 5:56:53Speaker 12

  actually able to to generate optimal trajectories.

• 5:56:59Speaker 12

  This also works in reinforcement learning. But yeah, I'll skip over that because we're doing video generation.

• 5:57:06Speaker 12

  But the other thing is, so this required us to actually have a reward function. So one other thing we are interested in is like how we can actually learn from from videos when we don't have a reward function? Can we actually get agents to learn from these from this sort of data? And so I guess one of the things that we can observe is that usually when you have videos, there's a sort of ordering to how like the trajectories are happening. Like typically you would have expert data that's telling you good things to follow. And so what we can do is say, at the end of the video, we're going to say that that's a reward of one. And everything that you if you if you backtrack in time, it gets sort of discounted, just

• 5:57:48Speaker 12

  like you would see in like a reinforcement learning trajectory. And so we can use the sort of idea to learn a value function that tells us how good behaviors are in our videos. And that's essentially what we do. So given a sequence of frames, we can say you get a reward of one at the end, and then we can backtrack that over time, and that's our value function. We can use that for basically training or reinforcement learning agent again, basically replace the sort of bootstrapping step with our learned value function and then essentially try to train your policy in a supervised way here. But you can see essentially, we trained this model over a bunch of different videos of pouring. You can see over time that the values increase. And so this is essentially telling you, you can learn a value function in this way.

• 5:58:34Speaker 12

  You can even use this for training reinforcement learning agents again, because, okay, fine. I have a reinforcement learning background. We do this sometimes. But you can see the agent is able to actually learn, even though it was trained over videos alone.

• 5:58:47Speaker 12

  Okay, so essentially, what we showed is that we can actually learn actions and rewards and policies and videos. From videos. And so I guess what's left, and this is sort of what led me into this sort of controllable video generation regime, where we're now trying to learn environments from videos. And this was the idea behind Genie, where we're going to try to learn a generative, interactive environments from videos alone, that's playable from both humans and AI agents. And so I guess a lot of the previous work that I was doing was really interested in like, how we can use these videos for training the agents themselves. But I was lucky actually to meet people like Jack and Tim from a team who had an open in this background.

• 5:59:30Speaker 12

  And they said essentially, well, we don't only need to learn policies. We can actually learn entire environments and we can place agents within those environments and get them to learn from that. And so this is what led to our Genie work, which we presented here. And so essentially, the idea behind this work was that we can learn three different main things. One was a tokenizer over our videos, so we represented those using a discretized VQ VIE model. We had a latent action model. I think this was probably the most important component, where we could essentially take in sequences of frames and try to infer the changes such that you could predict the future using using that latent action representation.

• 6:00:13Speaker 12

  And then you can plug that into a dynamics model for predicting the future. And this is where the controllability is coming from. It's coming from this latent action model that's telling you how things are going to change over time.

• 6:00:26Speaker 12

  And this is what led to our final results where we essentially found that if you take some like text-generated images, you can plug them into our model and interact with them as if they're a real environment. And again, we were training over a giant data set of platformer games here. So I guess the reason that I actually didn't spend too much time talking about Genie, because I know there's been a few workshop talks already and we talked about it already at the workshop, at the conference. But I was wondering, like, how did I end up getting into this kind of research? And I think the idea is that you can actually use these environments for training agents of the future. And hopefully, we can potentially like learn policies, learn latent policies, learn reward functions in the way

• 6:01:11Speaker 12

  that we discussed before. So yeah, I think that's the main thing that I have. I wanted to also talk about, I mean, point out all my collaborators here. There's been a lot of really great researchers that I've got the opportunity to work with. But yeah, that's all. Thanks. I think I probably have a lot of time for questions.

• 6:01:29Speaker 03

  In more complicated environments, actions alone wouldn't be able to represent all of the dynamics. How do you think we can disentangle actions without supervision in this case?

• 6:01:41Speaker 12

  Oh, without supervision. So I think you can probably, if you have like a notion of reward, for example, or a notion of, or if you can try to like learn a policy, for example, you might be able to extract, like, what are the most likely kind of actions that are going to happen versus the dynamics. But I think it's hard without supervision to disentangle these. Like, in our case, you can probably control the clouds if you wanted to. But I think maybe you can use something like text or that sort of thing to add in additional information. But I think it also scale. Yeah.

• 6:02:15Speaker 03

  So if you wanted to scale, let's say, Genie to real-world videos?

• 6:02:21Speaker 03

  What would be the major architectural and ideological changes to do that?

• 6:02:29Speaker 12

  Yeah, that's a good question. So the GNN model was pretty general. So there wasn't anything in there that said that we were explicitly training on 2D platformer games. We also had experiments where we got it to work on robotics data. So I think probably just scaling the architecture size, the bitter lesson as usual, and adding in more data would hopefully enable it to learn from that. I think that you could probably also change different components of the architecture itself using the current state-of-the-art techniques.

• 6:03:01Speaker 00

  It is surprising, or rather, not at all surprising, how many of the answers to workshop questions are just this one word: scale. We challenge you to go through a day at NUREP's without mentioning the bitter lesson once. As for the audience members' question about action generation and behavior cloning, Brittany was walking the poster sessions and found a possible answer from NYU.

• 6:03:26Speaker 16

  I am here with Sungjae Lee, also known as Jay Lee, to talk about his poster work on the VQ BET model, which is actually one of the spotlight posters being featured here at the ICML conference. The description is "A scalable behavior generation model for efficient multimodal behavior prediction in complex tasks." That is quite insightful, so it would be very helpful to have you maybe explain for us a little bit more what exactly it is that you've worked on here.

• 6:03:54Speaker 43

  Okay, nice to meet you. And actually, our poster is like a... Our poster, our work is started from a question: "How we could use a very powerful LLM-like token prediction framework for behavior generation tasks?" So the main concern about this question is that the action data is in the continuous space. It's not similar with the language that we use, which is really easy to tokenize, right? So what we do is we use VQVAE, vector quantizer, to quantify the continuous action data into a discrete representation

• 6:04:35Speaker 43

  and use that discrete representation as a tokenizer of the LLM-like architecture so that we can predict the behavior based on current observation.

• 6:04:48Speaker 16

  Very, very interesting. And how did you arrive at this area of research? What is kind of the background or the origin story for this project?

• 6:04:55Speaker 43

  Yeah, actually, um, actually, my personal background is more close to reinforcement learning. But after, I mean, after, I think nowadays, we, there are many accessible large action data, and I found that it is really hard to train a behavioral cloning agent using a very traditional way. I mean, it is really hard to train good, good policy with a traditional way with a large data set. So we need a better architecture, which can leverage the LLM-like architecture. So that was our, I mean, the starting point of our research.

• 6:05:38Speaker 16

  And how did you handle the, the data set collection problem? Because I know with a lot of the applications we're seeing on more, the robotic side of things, it seems today that data is a bit of the bottleneck more so than anything else.

• 6:05:50Speaker 43

  Yeah, actually, it is really good question, since getting data set is really expensive. And I mean, it's a really important point in robotics. So most of our environments was, uh, I mean, those kind of environments are open source environments, so you can just download most of the data set. And some of them is the, the data set was collected by VR equipment with humans. And for our real-world experiments, we gathered with a very small manipulation equipments by ourselves with iPhone. So, yeah.

• 6:06:28Speaker 16

  So you bootstrapped the data set in part yourself?

• 6:06:30Speaker 43

  Yes.

• 6:06:31Speaker 16

  And then it looks like you did a bunch of work maybe on the simulation side of things as well?

• 6:06:35Speaker 43

  Yes. Actually, we first validated our framework on simulation. And then, after some, with some consolidated results, we moved on to the real-world experiments. And the strong point of our, of, strong point of our, uh, strong point of our model is that our model is really lightweight, so it does not need large data set. We only need 45 demos for each task in real-world scenarios. So it only takes one or two hours by gathering by humans. So it's not that difficult, yeah.

• 6:07:05Speaker 16

  And can you talk a little bit about the performance results that you've seen with this model since the listeners at home don't have the benefit of the poster in front of them?

• 6:07:13Speaker 43

  Actually, you mean the performance of our model? Yeah, I would say that, you know, there was a very famous diffusion-based models. I would say that our performance is quite similar with those kind of diffusion-based models, but the inference time is really fast, about 20% of the diffusion model. So, you know, inference time is really important in robotics. So, we could say that you could do more than 100 Hz control with GPU and more than 20 hertz on CPU.

• 6:07:43Speaker 16

  Oh, no.

• 6:07:44Speaker 43

  20 Hz on CPU. So, yeah. So, the performance is good enough compared to the diffusion-based policies, but inference time is much better than those kind of baselines.

• 6:07:57Speaker 16

  Got it. And you, I think you mentioned that you published this toward the end of last year. Have you continued to work in this problem area, or how has your research evolved since the publication?

• 6:08:09Speaker 43

  Actually, we believe that the future direction should be scaling up this architecture. I mean, for the more generalizable agents, for example, the agents that could do some tasks based on the language instruction. So, our objective would be scaling up.

• 6:08:28Speaker 16

  Very exciting. And you did this through your work at Seoul National University and then you went over and worked with NIU folks as well?

• 6:08:36Speaker 43

  Yeah, actually, I was a master's degree at Seoul National University. And, I mean, I emailed to the people in NIU and we started collaborating from last summer.

• 6:08:46Speaker 16

  Very exciting. Well, thank you so much for the time walking through this. I really appreciate it.

• 6:08:51Speaker 43

  Thank you.

• 6:08:52Speaker 00

  That was a great spotlight poster from Sungjae Lee, and we also recommend his professor Leroy Pinto's talk on building general-purpose robots, which we link to in the show notes. Brittany had one more robotics paper to highlight. Pivot: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs. But we are skipping it in the interests of time and to not keep adding to our already overflowing Google DeepMind publication counter.

• 6:09:21Speaker 00

  By far one of the biggest names in reinforcement learning and robotics is Professor Chelsea Finn, now founder of the $2 billion startup Physical Intelligence, who gave not one, not two, not three, but four talks at ICML on her lessons on robotics. We are highlighting her keynote here, but we also recommend checking out her colleague Sergey Levine's talk on robotic foundation models.

• 6:09:47Speaker 44

  My name is Chelsea and I do research on both machine learning algorithms as well as on applications of machine learning to robotics. And because I work on both of these two things, I think that robotics has provided a perspective on my machine learning research that's a little bit different than the average machine learning researcher. And today I'd like to share a little bit about that perspective and what that perspective has brought to my machine learning research. So the first thing that I'll mention is that I think that my robotics work, even though it's not necessarily exactly aligned with core machine learning algorithms, it's often indirectly led me to problems that are relevant in applications beyond robotics.

• 6:10:30Speaker 44

  So for example, about 10 years ago, I started working on end-to-end neural network training for robots. This included things like training a robot to put a block into a shape sorting cube, or to use a spatula to lift an object into a bowl. And in both of these cases, we were training a neural network to map from images from the robot's cameras to torques applied to each of the motors of the robot. And we were training neural networks that had an entire 92,000 parameters. And while this might seem not particularly interesting or not particularly new, at the time, this is something that was actually quite different from the typical approach to robotics. And after I started working on training these policies

• 6:11:14Speaker 44

  to control robots with neural networks, I was a bit frustrated by the fact that we had to train a neural network from scratch every time we wanted to train the robot, even though we were typically training the robot to do lots of different tasks rather than just one task. And this led me to be interested in this question of whether robots could learn a new task more quickly by leveraging their previous experience instead of training from scratch.

• 6:11:39Speaker 44

  That led me to work on few-shot learning and meta-learning, which ended up actually having quite a relevant use cases in other applications like in education and in drug discovery. And there's another example of robotics work leading me to relevant problems. In this initial work, the robots were learning policies that were specific to one spatula, or one shaped turning cube, or one environment. And I became very interested in whether we could leverage broad data sets to improve the generalization of robots. And this led me to be thinking about how can we develop machines that can generalize broadly and potentially even be able to generalize beyond their training distribution. And this led me to work on data sets, but also to work on robustness

• 6:12:21Speaker 44

  to distribution shift, which led to a benchmark that we developed called WILDs that actually studies distribution shift in a wide range of real applications and has been used quite widely in the machine learning community.

• 6:12:34Speaker 44

  So from there, in this talk, I'd like to share a little bit about what working on robotics has taught me about machine learning. And to start off, let's talk about a few facts about machine learning in the context of robotics. The first is that machine learning is quite data hungry. And at the same time, we don't have existing data sets on the internet of robots controlling themselves to do different tasks. We don't have the equivalent of Wikipedia for how to control motors to tie shoelaces or to open a water bottle.

• 6:13:12Speaker 44

  Furthermore, we don't have an easy way to interpret or ensure the safety of machine learning policies applied to robots. And this has serious implications when robots have a real possibility of directly harming humans in a physical world. And lastly, compared to other leading approaches to robotics like optimal control, we lack formal guarantees of what a machine learning-based policy would do. And so because of these shortcomings of machine learning in the context of robotics, you might expect me to say that maybe machine learning isn't solving real applications like robotics, and it's fundamentally problematic.

• 6:13:51Speaker 44

  But is that actually true? Let's look at an example. So say that we want a robot to tear off a piece of tape and put it on a box. This may seem like a fairly simple task, but this is actually a task that is incredibly difficult for traditional robotics approaches, because traditional approaches will typically try to model the entire scene, including how the tape will adhere to the canister and to the fingers of the robot, how it will tear when spread across the metal part of the canister, and how to control all 14 of the motors on this robot in order to accomplish the task. It turns out that for this task that is seemingly extremely difficult for traditional approaches, we can actually use

• 6:14:35Speaker 44

  machine learning to address it. So we can develop a teleoperation interface. Specifically, Tony, a student in my lab, developed a teleoperation interface that we call Aloha, that allows you to puppeteer the robot to solve a wide range of different tasks.

• 6:14:51Speaker 44

  And once you develop this teleoperation interface, it means that you can collect data to train a machine learning-based policy to solve a wide range of different tasks, including the really challenging task of tearing off tape and putting it onto a box, as well as other tasks like putting on a shoe.

• 6:15:07Speaker 44

  In this case, it's a machine learning policy that's mapping the images from the robot's cameras to all the 14 joints, and it's doing so with a transformer trained end-to-end on demonstrations collected with teleoperation. And we can use machine learning not just for these fairly complicated tasks, but we can also do it for mobile manipulation. So we can develop a teleoperation interface for an entire mobile robot with two arms, use that to collect data, and again, use a transformer-based architecture to train the robot to do challenging tasks like, on the top, make a piece of shrimp by pouring oil on the pan, putting the shrimp into the pan, flipping the shrimp, and serving it, and on the bottom, putting a pot into a cabinet.

• 6:15:54Speaker 44

  And so again, we're finding that machine learning is able to solve fairly complicated robotics tasks. And beyond these kinds of robots, we can also do something like this for surgical robots. So surgical robots are incredibly difficult to control. This is the DaVinci surgical robot, and we can use machine learning in a fairly robust way to, again, train policies for complicated tasks like tying a knot and picking up a needle and handing it over to the other surgical tool. And finally, we can also do this with full-size humanoid robots, where if we develop a teleoperation interface, which is a little bit harder to do in this case, but we can train a shadowing-based

• 6:16:34Speaker 44

  teleoperation approach, and then use this to train, again, transformer-based policies in this case, to control robots, to do pretty challenging tasks that involve controlling all of the different degrees of freedom, including both the arms and the legs of the robots. And so, going back to my question before of whether machine learning is solving real problems, I do think that machine learning has been making real advances that advance applications and really useful problems in the real world. Supervised learning works really well. We've seen significant advances in architectures, learning algorithms, and optimizers.

• 6:17:18Speaker 44

  And we also have reliable engineering practices for debugging if something isn't working, debugging if a policy isn't working, or if another model is not achieving the performance that we want, and ultimately improving the performance. So, now you might ask, if machine learning is making real advances, why don't we have robots out in everyday environments solving real problems yet? And a lot of people, for that question, will refer you to Moravec's paradox, which states that the things that are most intuitive for humans, like basic motor control, are the things that are often most challenging for machines. And this could explain why robotics is further behind than applications like debugging

• 6:18:03Speaker 44

  complex code or translating between two pieces of text.

• 6:18:08Speaker 44

  But in my work, I've actually found that this isn't perhaps quite the most direct explanation. I think the explanation is actually that the things that lack abundant data are often the things that are most challenging for machines. And this is because things, scenarios that lack abundant data, we're not able to directly apply machine learning and directly try to identify patterns from large amounts of data.

• 6:18:35Speaker 44

  And this can include both data scarce applications as well as just scenarios that are novel that aren't represented well in the training data. So this isn't just things like robotics that don't have a corresponding Wikipedia and so forth. It's also even within applications that do have a lot of data, there's scenarios that they encounter that aren't represented well in the data. And that's exactly where machine learning algorithms often struggle. And as a result, our machines often struggle.

• 6:19:00Speaker 44

  And so perhaps instead of trying to, I don't know, take some sort of approach that tries to combine traditional methods or machine learning or something, I think that actually robotics just needs more of what makes machine learning thrive. Essentially, we need to find more ways to get data for applications like robotics.

• 6:19:22Speaker 44

  And so this is really the core question that I want to talk about today is how can we get good data for a wide range of problems in a cheap and inexpensive way?

• 6:19:35Speaker 44

  So how can we basically handle data scarcity without skimping on data?

• 6:19:41Speaker 44

  And I'll talk about a few different ways to do this. The first is finding ways to augment data with cheap and natural to provide supervision. The second will be to leverage data sources beyond the particular target application. And the third will be to incorporate data from test time in addition to the typical training data set. And I'll spend the most time on this first point because it's a little bit different from some of the ideas that have become more commonplace in machine learning.

• 6:20:12Speaker 44

  Great. So to start out by talking about kind of cheap and natural to provide supervision, let's look at how we currently supervise machines. So we currently will take a training dataset, train a model, evaluate that model. And to evaluate it, we'll actually, ideally, actually look at how it does in a real situation by talking to it or by running a robot and so forth. And inevitably, the model often won't work well in some scenarios. And the best course of action, assuming that you've optimized it well and the architecture is well tuned, is to collect and label more data. And specifically, collecting and labeling more data in the scenarios that is struggling.

• 6:20:56Speaker 44

  And so this would involve going out, getting examples, getting labels for those examples that cover those scenarios that is not working well. And this is really expensive and very human intensive. And if it were cheaper, we would be able to iterate on this cycle more on the model, and we'd probably end up with a stronger model. So that's one shortcoming with kind of a typical supervised learning approach. And the second is that input-output pairs are also a little bit weird in some settings. Say that we wanted a robot to cook a meal. The way to apply supervised learning in this case would be to collect examples of how to move the arms of the robot, how to move the motors as a function of the inputs.

• 6:21:39Speaker 44

  And this is a little bit weird compared to just trying to teach the robot naturally the kinds of things that it should do, like making sure that the water is hot enough before putting pasta in, or setting a timer to make sure that it's been cooked for long enough. Or as another example, say that we want to train a system to make a medical diagnosis. The typical supervised learning way to do this would be to have examples of symptoms and then have kind of examples of the diagnosis as a result. But instead, perhaps the more intuitive way would actually be to teach the machine about what sorts of, kind of, what, how diseases actually manifest in humans and patients.

• 6:22:21Speaker 44

  And so this is kind of bringing us to the idea that perhaps we might be able to train machine learning models in a more data-efficient way if we were able to incorporate natural-to-provide supervision. And so now, one thing you might think about here is, instead of providing labels, what if we use human feedback? Reinforcement learning from human feedback has been quite successful where, instead of providing input-output pairs, we'll look at an input-output pairs, a set of them, and say, "This diagnosis is better than this one," or "This pasta tastes better than this pasta." And this can require a lot less supervision because you don't actually have to write out or actually provide the exact motor torques.

• 6:23:05Speaker 44

  But it still requires many labeled examples, many examples of an outcome and what is preferred. And so is it possible to give machines far less supervision but still allow them to improve? So we're going to look at this both in a robotics example as well as in a more standard image classification example. Let's start with the robotics example. So we're going to be looking at long horizon by manual tasks. The goal, for example, might be to put all the objects into the bag. And it's really expensive to collect demonstrations that cover all of the possible scenarios that the robot might end up in. And the form of natural supervision that we're going to be considering here is just verbally telling the robot how it might handle

• 6:23:48Speaker 44

  or how it might improve in situations rather than trying to collect a ton of demonstrations for the scenarios that it's struggling in. And so specifically, say the robot is going about the task, and it's struggling on this part of the task of putting the sponge in the bag. What we'd like to be able to do is, we'd like to be able to tell the robot at this part, "You should use the sponge to open the bag wider," because right now, the bag is not kind of open very widely. And ideally, it'd be able to use this, just this verbal snippet of text to both improve on the fly, to be able to figure out how to solve the task in that scenario, as well as how to then take that data and actually improve the policy

• 6:24:32Speaker 44

  and improve its ability to handle new situations like that in the future. So we'd like to be able to use this high-level language supervision both on the fly and for future improvement.

• 6:24:48Speaker 44

  So how do we do this? If we want a robot to be able to improve from high-level language corrections, we need a way to connect what the robot is doing with language. And so to do this, we're going to train a hierarchical policy, a high-level policy and a low-level policy, where language is the interface between those two policies. And so, more specifically, we'll take the observation, this will be fed into a high-level policy that then predicts language corresponding to a skill, like pick up the sponge or put the Sharpie into the bag. And then this language command will be fed into a low-level instruction-following policy

• 6:25:28Speaker 44

  that takes as input the robot's observations and outputs how to move the motor commands. This kind of hierarchical approach is not new, it's actually been done in a wide variety of prior works, and so it's not what we're introducing here. The key insight of what we're going to do here is that we can actually update the high-level policy only with language supervision, because its output space is language, it's a kind of a skill that the robot should do next. And because of this, if the low-level policy can follow a wide range of instructions, then we can actually improve this full system just by updating the high-level policy and just by giving it language feedback. Specifically, we can do something like the DAGger algorithm, the dataset aggregation

• 6:26:13Speaker 44

  algorithm, on the high-level policy and freeze the low-level policy. So specifically, what this is going to look like is we'll intervene, we'll tell the robot what we want it to do, like, in this case, maybe it should kind of rotate the tape in order to put it into the bag. And this intervention, this language command, will override the high-level policy, and that intervention will be fed into the low-level policy instead of what the high-level policy is predicting. And then, that will allow it to, on the fly, be able to leverage these interventions. And we'll also aggregate these interventions into a dataset and use this to update our high-level policy, so it actually also learns how to improve from these corrections in the

• 6:26:57Speaker 44

  future. And so, we're freezing the low-level policy and updating the high-level policy by supervising it just on the language corrections that the human is providing.

• 6:27:09Speaker 44

  We gave this a fun name, "Yell at your robot", or "Yay, robot", because you can articulate your corrections or frustrations with the robot to help it improve. And what can this do? So let's look at some videos of fully autonomous policies on the robot. And we'll start just with the base policy before doing any language corrections. And so this policy is trying to put the objects into the bag, and it'll make mistakes. Like, in this case, instead of putting the Sharpie into the bag, it put it underneath the bag. And it struggles to be able to recover from that. It'll also make mistakes. And so it'll also make other mistakes. So here it's trying to pick up the Sharpie. And the high-level policy output is shown here on the top left. And we're actually finding that the high-level policy

• 6:27:54Speaker 44

  isn't ever issuing corrections like "go lower", or maybe "rotate the gripper" in this case. It just keeps on telling the policy to try to pick up the Sharpie. Now, after we fine-tune on language corrections, we find that it's able to autonomously correct for mistakes. So here it's making the same mistake as before by putting the Sharpie under the bag, and then it's trying to self-correct. It then makes a mistake again, and then it's self-correcting again to try to move towards the camera, go higher, and then put the Sharpie into the bag. And by self-correcting, it's able to solve that part of the task successfully. It also learns to self-correct for grasping, where it'll self-correct to move to the right after it made a mistake of grasping too far to the left. And when trying to put the sponge into the bag,

• 6:28:39Speaker 44

  we'll also see it just change strategies completely. So here it's trying to, in some ways, kind of shove the sponge into the bag, and it's doing so unsuccessfully. And now the high-level policy is going to tell it to instead try to release the sponge and sort of poke it into the bag instead. And this helps it get it into the bag more successfully. And as a result of the robot's ability to self-correct from just this language supervision, we find that the robot is better overall at doing long-horizon tasks. And so this video is pretty long because the task is quite challenging, so I won't play all of it. But we get a sense that despite this task being quite challenging and having all sorts of scenarios that we don't necessarily have demonstration data for,

• 6:29:24Speaker 44

  we find that by leveraging this very cheap language supervision, the robot is able to perform the task a lot more successfully, even though this task is quite long.

• 6:29:39Speaker 44

  Cool, then there's one more thing I wanted to highlight from the system, which is that we can't, um, instead of just correcting after the robot has made a mistake, we can also actually proactively correct the robot when we think it might make a mistake in the future. And so this is a different task that we trained the robot to do, which is to make trail mix. Uh, the grad students were quite happy about all the trail mix that ended up in the lab as a result of this. And, um, we see that right here, um, I paused the video. The robot is, it looks like it's actually about to accidentally pour a whole bunch of peanuts onto the table because the scoop is behind the bag instead of inside the bag. And right here, because we kind of noticed that it looks like it might be about to make a mistake,

• 6:30:23Speaker 44

  we can intervene and instead of telling it to continue by moving the scoop into the bag and presumably then trying to pour into the bag, we can interrupt the robot and correct it and tell it to move the left arm to the left, go higher, move the scoop into the bag, and then allow it to continue autonomously to pour into the bag. Um, and so this is an example of how in real time we're able to improve the performance by proactively preventing the robot from making a mistake. And after fine tuning, we find that it also learns this sort of proactive corrective behavior where it notices that, um, in this case with cranberries, it was about to make a mistake there. It didn't successfully get it, get the scoop into the bag.

• 6:31:02Speaker 44

  Um, and it then corrects itself to move the scoop into the bag successfully. Cool. So those are a number of qualitative examples. Quantitatively, we also see a large gain in performance just from verbal corrections. So the dark orange bar here shows the success rate on average after fine tuning on just language data, whereas the gray bar shows the policy before language corrections. We see a 20% improvement in performance. Um, and this closes a lot of the gap to this light orange bar, which is the performance if we use human corrections on the fly to override the high level policy.

• 6:31:41Speaker 44

  Lastly, it's worth mentioning that the performance of this still has room, like, there's still a lot of room for improvement for even when we're using kind of oracle high-level human corrections. And so this suggests that the low-level policies have room for improvement.

• 6:31:57Speaker 44

  So, to summarize, you can productively yell at your robot to help it actually accomplish tasks, but more importantly, the robot can improve just with language feedback without demonstrations by fine-tuning this high-level policy. And this is a lot more data-efficient. It's a lot more data-efficient to simply tell it to pick up the sponge or move to the right than to actually collect demonstrations with teleoperation. And then, of course, this approach relies on a performant instruction following policy, and so you're not completely out of the woods in terms of having to collect some low-level data on the robot.

• 6:32:36Speaker 44

  Great, so this is an example of how we can use natural supervision to augment data and get much better performance in a very cheap way. Can we do something similar for other machine learning systems beyond robotics? So, say that we want to perform a classification task, an image classification task, based on the species of the bird, and we train a model to do this, and here I'm going to be visualizing the predictions that the model is getting right and the predictions that the model is getting wrong.

• 6:33:09Speaker 44

  And if we contrast the correct predictions from the incorrect predictions, one thing we might notice is that a lot of the incorrect predictions, there's a little bit of a pattern, which is that a lot of the incorrect predictions, not all of them, but a lot of them, have trees in these examples. There's a lot fewer trees on the predictions on the left. And so it'd be nice if we could just verbally tell the model to pay less attention to trees. So just like how we told the robot, kind of corrections like go lower in these situations or take a different strategy in these other situations, if we could simply verbally tell the model to correct its behavior here, we'd be able to correct the model far more efficiently than collecting additional images and labels for those images.

• 6:33:56Speaker 44

  And so we tried to develop an interface that would allow humans to verbally correct machines in that way, where we first train an initial model. We allow people, including non-experts, to describe failures using a natural language of that model and then correct for those model failures just by using the language feedback.

• 6:34:20Speaker 44

  So how this works is first we'll present the correct predictions and the incorrect predictions, just like I showed before. And so in like a didactic example, where we're trying to classify squares versus ovals, where the model might be paying attention to color when it shouldn't be. This would look something like this, where you would contrast the examples on the left and right and then try to describe verbally how the model is making a mistake by paying it too much attention to the color red or color blue.

• 6:34:48Speaker 44

  Or in the Waterbirds days, that perhaps the model is paying attention, too much attention to the trees in the background. And then, once we visualize these model failures, we'll then ask a person to describe the model failures and then also help them understand whether that description is something that the model can actually understand and use to improve itself. And so we developed a web interface to allow users to look at these examples. We're using CLIP in the background to help understand if the model is actually able to connect that verbal concept to the images that are in its dataset. We can then compute the similarity between the text prompt and each image to figure out whether or not that text prompt is separating the correct examples from the incorrect examples. And if it is separating those examples,

• 6:35:33Speaker 44

  then we can then use that to improve the model. And so if it gets a high error score, then we can then directly start to use that for training. If the user finds that they aren't able to describe something that the model understands and can separate these concepts, then the user can iterate on their description to then try to describe it in a way that the model can interpret.

• 6:35:55Speaker 44

  And so then, once we have this text feedback, we'll take a very simple approach that was presented in a previous work called DFR, where we're simply just going to balance the data across these different groups. So, for example, if we're finding that the model is paying too much attention to trees, then we're going to balance the images that have trees in them with the images that don't have trees in them. And this will decorrelate the data such that it no longer is incentivized to pay attention to trees.

• 6:36:23Speaker 44

  And then, once we decorrelate the data, we'll then kind of retrain or fine-tune on that decorrelated data to get a model that stops paying attention to that piece of feedback. And then, if desired, you can, in principle, also iterate on that process to then identify any new model errors that popped up.

• 6:36:42Speaker 44

  Great, so in our experiments, we tried to identify, first, if non-experts could actually identify and describe model errors in a way that led to improved robustness of the model. And, second, whether or not we could scale this sort of approach even to very large data sets, be able to cheaply provide supervision that is allowed to, that can identify model failures in these large-scale situations. And so, in the first case, we recruited 26 participants on a crowd platform that had very minimal qualifications. They're native English speakers and so forth, and so likely people that are not machine learning practitioners. And we had them interact with waterbirds and celeb.

• 6:37:23Speaker 44

  And what I'm showing here is I'm showing all of the different verbal descriptions that each participant identified for describing model failures. And the black line is showing Yunho, the lead student researcher on this project, his performance, or the error score that he was able to get for his reference phrases. And what we can see here is that, in a lot of the cases, the human, non-expert participants are able to identify model failures fairly accurately compared to Yunho. In these examples, they're able to identify the correct concept underlying the model failure, but they might have some suboptimal wording compared to the wording that Yunho used. There's a number of examples where they find, basically, the same phrase that Yunho used.

• 6:38:07Speaker 44

  And then, there's also examples where, actually, the non-experts provided better descriptions of model failures than Yunho's reference prompt. Then, lastly, there's also a few cases, specifically four cases, where participants struggled to identify the correct model failure.

• 6:38:24Speaker 44

  And then, once we have these descriptions, then the question is, how cheap is this supervision? And can we use this cheap supervision to provide, to actually improve the model? So first, we found that, on average, these non-experts were providing 2 to 3 minutes to give feedback to the model. So this is pretty fast, and a lot faster than collecting additional labeled data.

• 6:38:48Speaker 44

  And second, we found that if we use their descriptions to rebalance the data and retrain, we were able to get a model performance shown in yellow that is a lot more robust than simply training on the original dataset or zero-shot prompting approaches. And so we see, in this case specifically, a 7 to 10% improvement over training on the initial dataset, just with 2 to 3 minutes of additional supervision from a non-expert.

• 6:39:18Speaker 44

  And then, beyond these somewhat simple datasets, what about datasets like ImageNet? We didn't run a user study on this specific thing, but we found that using this interface, Yunho is able to fairly quickly identify model failures on ImageNet. And so, for example, here are some examples, all from the same class, actually from the sliding door class, where the model is doing very well on these images and very poorly on these images. And as you might notice, these examples on the right have a high similarity with cars, and a lower similarity with cars on the left. And so, this is, it's kind of struggling to classify sliding doors if they're sliding doors on cars.

• 6:40:03Speaker 44

  And he's able to identify model failures on 31 different classes in ImageNet, and able to do so in a relatively short period of time. And with data reweighting, he was able to improve the performance of the model on the minority split of the data while preserving overall performance.

• 6:40:22Speaker 44

  And so with this sort of approach, we found that we're able to give verbal feedback based on an initial trained model. And because it's based on the model that's already been trained, similar to the robotic setting, it's actually easier to target model failures efficiently rather than simply trying to out of the blue guess the kinds of supervision that the model might need.

• 6:40:43Speaker 44

  And second, the verbal feedback that we're giving, it's no longer data point level. It's actually at a more global concept level. And this means that the verbal feedback is especially cheap because with a single sentence or a single phrase, we're able to actually address a broader class of model failures rather than providing individual examples.

• 6:41:04Speaker 44

  And then importantly, I'll also mention that in this most recent work, we're only identifying and correcting one kind of model failure. And this means that the scope of the work is quite limited. But it'd be really exciting to see if we could use this kind of high-level verbal feedback for other kinds of model failures and develop a more general approach for improving models just with verbal feedback.

• 6:41:27Speaker 44

  And so kind of the takeaway from both the Yay Robot work and the Clarify interface is that natural supervision, like language supervision, if the model can use it well, that supervision can be far cheaper and sometimes even more informative than collecting a large number of labeled examples. And so it's a useful tool to have when we don't have a great deal of initial training data.

• 6:41:53Speaker 44

  Great. And so now, another example of how we might try to, basically another example of some data that's essentially out there, but we just need algorithms to be able to use it well, is data from other sources beyond the target application. And so specifically, one natural thing to do to improve generalization for a particular application is to leverage internet data, to like leverage models trained on text and images.

• 6:42:23Speaker 44

  One very common way to do this is just to use, for example, an encoder pre-trained on ImageNet. And we find that, at least in robotics applications, it does improve performance somewhat compared to just training from scratch. And especially, we can do well on tasks and scenarios that are seen in the training dataset. But when evaluating on generalization to unseen objects, backgrounds, and environments, there's still a really substantial gap compared to the things that it saw during training.

• 6:42:56Speaker 44

  And yet, the internet has really vast training data. And so we expect that maybe we could do better than this. Specifically, maybe if we could more closely connect the pre-trained model with the downstream task, we might be able to more effectively leverage all of the rich knowledge that exists in internet data.

• 6:43:16Speaker 44

  So specifically, what we're going to do is we're going to take a visual model. Instead of taking a model trained just on ImageNet classification, we'll take a model trained for visual question answering. And we can formulate the downstream task, specifically the robotic control problem, as a visual questioning answering problem. And so instead of having it output continuous values, we're going to frame it as a question. What should the robot do to do a task, like to pick up the chips or to move a bottle upright? And then we'll likewise also frame the output of the model as a series of tokens similar to the output of a VQA task. And these tokens will correspond to different language actions,

• 6:44:00Speaker 44

  like how to translate and rotate the gripper of the robot. And if we formulate essentially this downstream task just like the tasks that are seen during pre-training, perhaps it will be able to leverage the pre-training data more effectively and understand how to generalize robotics tasks similar to how it generalizes these VQA tasks.

• 6:44:21Speaker 44

  And so once we have this data, we'll use the same architecture, specifically a pre-trained vision language model. And you can either fine-tune it just on the robot VQA tasks or a combination of the robot tasks and the existing internet VQA data that the vision language model was pre-trained on. And so it'll output these language tokens that'll then be converted into robot actions to be run on the robot.

• 6:44:49Speaker 44

  So essentially, we're posing robotic control as a visual question answering problem and defining tokens corresponding to robotic actions. And we'll refer to this kind of fine-tuned model as no longer a vision language model, but a vision language action model in the sense that we're now having actions, kind of, some of the tokens are representing actions. So now, if we go back to this example of how the, if we're just using a pre-trained ImageNet encoder, how well that does, we find that models that use this sort of vision language action recipe, we find that they're actually able to generalize far better than the model that is pre-trained just on ImageNet classification. And so essentially, by connecting the pre-trained model and the downstream

• 6:45:34Speaker 44

  task, we're able to get gains in generalization. Now, what does this look like for more recent state-of-the-art models? So we can also compare state-of-the-art models that use standard pre-training or no pre-training to recent vision language action models like RT2X and OpenVLA. And we'll, we'll be doing this on evaluations that focus on generalization. And what we find is on two different robot platforms, the vision language action models shown in red and green do substantially better on average than the models that don't use this vision language model pre-training and don't use this formulation that formulates the downstream task very

• 6:46:17Speaker 44

  similarly to the pre-trained task. And so again, we're kind of, even with these state-of-the-art models, we again kind of see this trend that generalization improves significantly if we connect the pre-trained model with the downstream task. And so going back to trying to handle data scarcity without skipping on data, we can leverage data that already exists, data from the internet that's easy to get, and we can leverage it much more effectively if we connect the pre-trained model with the downstream task.

• 6:46:51Speaker 44

  Great, and then lastly, I want to talk about incorporating data from test time.

• 6:46:56Speaker 44

  So specifically, thinking about whether if we are in a new situation that's not represented well in our training data, can we adapt on the fly? And I think this is a really important problem, because when machine learning systems are faced with the real world, there's a vast number of objects, vast number of configurations and scenarios that these machine learning models will be faced with. And I don't think we can even hope to anticipate every possible scenario that these machine learning models are faced with. And because we can't anticipate it, then maybe instead we can just adapt after the fact, when we see more data from that situation.

• 6:47:36Speaker 44

  So, for example, say that we're trying to open a door, maybe this is a new door that we haven't seen before.

• 6:47:42Speaker 44

  If we're trying to do this, we might make a mistake and might need to retry. And it turns out that this is a video of a human opening this door, and it was quite subtle, but the human actually did make a mistake and adapt very quickly. So let's replay the video, and specifically, we see that the human puts the key into the door, actually puts it in the wrong place, right here, and then continues by taking the key back and then putting it in the correct place. And so even humans are making mistakes and adapting, and so even humans, which in many ways are sometimes even a gold standard compared to machine learning, if even humans are adapting, can we develop machines that can adapt in a similar way? So let's look at this in the context of robotics problem.

• 6:48:27Speaker 44

  This is a scenario that's unseen to the robot, and so the robot's here, its goal is to get to over here, and if it's trying to approach this problem and it makes a mistake, can it actually retry? So the robot only gets this first-person observation right here. Without any context, if it hasn't actually attempted the task, maybe from this observation it'll try to crawl under and see where it gets from there. And then maybe if it tried to crawl and then realized that it was very close to an obstacle, then maybe it should try a different strategy. And so with this context, with this previous history of what it's tried in the past, maybe it should try with the same current observation to do something different, like turning left or turning right.

• 6:49:15Speaker 44

  And so this is exactly what we'll do. We'll take these recent attempts and we'll combine them with a model that's known to be fairly good at adapting from recent attempts. Specifically, in this case, we'll use a vision language model. We'll pass these recent attempts and the robot observation into the model. We'll then have this selective skill for the robot to do, and then output actions.

• 6:49:39Speaker 44

  And ideally, the vision language model should leverage what the robot has tried before and pick appropriate skills after it's made some mistakes.

• 6:49:49Speaker 44

  And so if we do this, we find that exactly on the scenario before, which is unseen from the robot, if we don't use history and don't allow it to adapt from its mistakes, it often makes the same mistake over and over again. Whereas if we do, using context learning is able to try something different and adapt on the fly based on what it has seen in this test environment. And likewise, here's another setting, an outdoor setting. This is actually quite challenging because there's this step that is quite unstable in front of the robot. And at this point in the video, the robot actually can't even see that its back legs are stuck on the step. And so it's trying to walk forwards, and if it doesn't have history, it doesn't know that walking is being unsuccessful in this scenario.

• 6:50:32Speaker 44

  But with history, it's able to figure out that it should go backwards and instead try to climb over the step instead of just trying to walk over it.

• 6:50:40Speaker 44

  And we also see quantitatively that allowing, like leveraging test time information, leveraging these images that the robot sees at test time, improves the robot performance by more than 50%, both in terms of success rate and in terms of the time it takes to complete a test scenario.

• 6:50:58Speaker 44

  Cool. So the takeaway is that in context learning greatly improves the adaptability of the robot, and in turn, this improves its resilience and performance in unseen situations. And there's also limitations and future work with this, as with any research and like all the research that I presented, which is that in this case, it's unclear necessarily the best way to ground language to the low-level locomotion policies. And also, in many cases, we might not want to use a language abstraction as the way to retry and as the way to connect with vision language models, and so there might be interesting ways to expand on that.

• 6:51:35Speaker 44

  Great, so for this last part, we found that incorporating data and information from test time can make up for lack of representative training data. And all of these are examples that I covered in the talk are examples where more data is out there, more data is either out there, it already exists, or it's pretty easy to get. And we just need algorithms that can leverage things like natural supervision, pre-trained models, and test time data in order to effectively handle these new situations, or these situations that aren't covered well by the training data. Now, I also mentioned that along these three directions, there's also, I think, exciting directions for future work. I talked about one way to leverage cheap, natural language supervision.

• 6:52:20Speaker 44

  But I think that in the future, maybe we can operationalize entirely new learning regimes that leverage natural supervision in a general-purpose way. Moreover, I showed how we can connect pre-trained models with downstream tasks by making the downstream task look more like the pre-training problem. But maybe in the future, we could actually change pre-training in a way that makes it easier to connect with all sorts of downstream tasks. And then lastly, I showed how we can adapt at test time in a robotic scenario to make up for lack of representative training data. But there's all sorts of examples and applications in machine learning where we're interfacing at the end of the day with a human or with some other environment. And so can we also allow machines in non-robotics examples to adapt on the fly and retry when

• 6:53:05Speaker 44

  they're interacting with a person or interacting with some other environment, like a web environment? Great. And so then the last thing I'll also mention is that these are, I discussed a number of different kind of creative ideas for leveraging different sources of data and different sources of supervision. There's also this question of what if we also have broader training data? I think that all of these are quite interesting, even when you have broader training data. And we've seen from the regime of large language models that there's a lot of things that are quite exciting to try and do when you also have a large training data set.

• 6:53:42Speaker 44

  In the context of robotics, we've also been starting to study this problem. And so back in March of this year, I co-founded a company to help actually try to see what happens when you do try to scale up data and models in the context of robotics, to try to tackle a broad range of real-world use cases and robot platforms. And some initial results are here, where we find that we can do actually pretty cool tasks, even with data that hasn't been, data that was collected since March of this year.

• 6:54:15Speaker 44

  And then the last thing that I'll mention is that we, I talked a lot about finding new forms of data, like natural supervision or data at test time. And these are things that are actually quite widely applicable and make the overall problem easier. But a lot of our machine learning benchmarks actually aren't necessarily designed for these kinds of ideas or these kinds of algorithms that leverage different forms of supervision or data. And so it may actually be the case that in some scenarios, benchmarks might actually be harder than the problems that they're trying to represent,

• 6:54:48Speaker 44

  because they don't necessarily allow for you to use other forms of supervision or data. And so perhaps by understanding the context surrounding different real applications that we're trying to study, we might find new and interesting ways to find data or new and interesting problem standings and also make more progress as a whole. Great, so I'll leave you with that. I'd like to mention that all the work that I presented was done with a really fantastic set of collaborators. I'd especially like to highlight the students that led the work that I presented. Yunho led the Clarify work. Lucy led the Yay Robot work. Annie, Alec, Andy and Govind led the test time adaptation work. And Mujin, Carl and Sid led the Open VLA project. And happy to take questions.

• 6:55:32Speaker 00

  The last thing we want to highlight in this epic seven-hour coverage of ICML 2024 is the new position paper track that encourages researchers to step back from individual papers to make arguments relevant to their entire field. Here is Yanghao Park arguing that automatic environment shaping is the next frontier in RL, which we think has been the implicit argument we have been developing through the papers and talks we have been exploring this episode.

• 6:56:00Speaker 18

  Hello, everyone. Thank you for being here. My name is Yanghao Park, and I'm excited to present our position, automatic environment shaping is the next frontier in RL. This is joint work with my colleague Gabe and Paul Gett from the improbable AI Group at MIT. To give you some context before we start, me and Gabe both come from a robotics background. And as a grad student working on robotics, I always dream about a magical box that can automatically create a robotic controller for me by simply specifying the robot environment and task I want. And I call this magical box Automatic Behavior Generator. And before we move on, I want to emphasize the word automatic here. It means that this box should only be powered by time and compute, not by human effort.

• 6:56:45Speaker 18

  This magical box, if realized, will serve as a core tool, enabling robots to autonomously generate behaviors on the fly, even after its deployment to people's houses. But I want to ask you all, do you think we're being a bit overly ambitious? Is our dream, this magical box, too good to be true? Well, if you think about it, this is what reinforcement learning is promising us in some sense. Reinforcement learning in theory is a generative automated optimal control solver that can produce working controllers for any MDP setting. However, from a practical viewpoint, who's trying to use RL as a tool to train robots, this claim is not necessarily true.

• 6:57:29Speaker 18

  Although RL itself does not require human effort during its training process, we want to point out that there is a very heuristic, labor-intensive process that are required to make RL work in practice. And that is what we call environment shaping. When an RL algorithm fails to find a solution in practical scenarios, between the choice of fixing the RL algorithm and shaping the environment to make it work, practitioners typically tend to choose the latter. The core problem of such practice is that it heavily relies on human effort. Domain knowledge for the task, intuition, and sometimes a bit of luck is crucial to get things right.

• 6:58:09Speaker 18

  A very well-studied example of environment shaping that you might all already know about is the reward shaping problem. We all know that RL agents love to hack the reward when they can, so engineers typically go through the process of shaping the reward to prevent it. In fact, I would say that this is the biggest reason why some people in our community hate RL so much. And I completely understand the process of reward shaping, it's definitely not fun to do. Unfortunately, I want to point out today that reward is not the only thing that we should we usually shape. Robotics engineers carefully shape nearly every component of the environment to make RL work in practice.

• 6:58:50Speaker 18

  And again, the only optimizer that is currently known to work the best for this problem is graduate student descent, a process entirely relying on human effort. All those being said, what am I arguing today? First, I argue that the company should start prioritizing research to automate the heuristic process of environment shaping. At the same time, we also need better RL algorithms that doesn't require heuristic environment shaping in the first place. And to do that, I argue that we should be benchmarking our RL algorithms on unshaped environments without any task-specific heuristics included. To better back up our argument, from now on, I'll try to give you some examples of

• 6:59:31Speaker 18

  the heavy heuristics that are involved in popular robotics RL environments and show how crucial they are to make RL work. As an exemplary environment to analyze, we chose Isaac Jim M's, one of the modern benchmark environment containing diverse robotics tasks. Let's first talk about action space shaping. In the context of robotics, action space shaping is a process of choosing how to convert the action, predicted by the policy, to an actual command that can be sent to the motor. An unshaped action space will thus look very simple. We are just letting the policy to directly predict feasible motor commands. However, most RL environments apply a bunch of task-specific heuristics to shape the policy outputs before it gets passed into the motor.

• 7:00:17Speaker 18

  This example code you just saw, for instance, applies diverse scaling, clamping, moving average filters, and PD controller at the end to finally convert the policy outputs to motor commands. The problem of this kind of shaping process is that it not only is very task-specific, but it also introduces a bunch of extra knobs and hyperparameters to tune. Unfortunately, this action space shaping is a necessary evil for RL algorithms. We have tested that PPO, for instance, completely fails to solve these tasks if we remove such shaping.

• 7:00:51Speaker 18

  And our findings are similar for observation space as well. Observation space shaping is basically a feature engineering problem, selecting the relevant states from what's available from the simulation to create an observation for the policy. For instance, for the task of opening the door using a manipulator, an unshaped observation space would be a simple concatenation of every raw simulation state that are available. However, typical RL environments go far beyond this simple concatenation. They introduce multiple hand-engineered task-specific terms, and they often convert certain states with unique properties like rotations to a different representation that are known to be better for neural networks to process.

• 7:01:32Speaker 18

  And such processes are also very crucial to make RL algorithms work in practice. We can break the RL by just removing those handcrafted terms from the observation.

• 7:01:42Speaker 18

  Although I'm skipping the other examples of environment shaping due to time constraint, you can take a look at our paper for more comprehensive examples.

• 7:01:50Speaker 18

  Now that we learned about the details of environment shaping and how it affects the RL performance, let's talk about how we can automate this environment shaping process. Automating environment shaping is a challenging problem for many reasons. One of the major problems is that there is no compact way of parameterizing the vastly diverse ways of doing environment shaping. If we assume a fixed functional form for everything, we can try extracting the coefficients and do some classical hyperparameter optimization on top of it. But this is a very limiting way of representing these shaping functions. Therefore, people have recently started to think about a more flexible way of representing these shaping operators. One of them is to use Python code itself as a way of representing these functions.

• 7:02:34Speaker 18

  This allows us to view environment shaping as a code optimization problem using large language models. This paper called Yurka is a good example of using large engine models as a sampling based optimizer to automate the rear shaping process.

• 7:02:50Speaker 18

  So we have conducted some experiments to see whether the proposed automation method using L L M's can be extended to other shaping components. And as you can see over here, models like G P T four was able to successfully shape action and observation space with similar to human performance. However, interestingly, when we asked G P T to shape multiple components jointly at the same time, the performance dropped dramatically. And this can be a critical problem since our experimental findings suggest that optimizing individual components one by one in a sequential manner often leads us to locally optimal performance. All that being said, I believe we still have a long way to go to fully automate the process of environment shaping. Now that we have discussed all the aspects of environment shaping, let's discuss about path forward.

• 7:03:36Speaker 18

  Recall that I was advocating for the research focused on either automating the environment shaping or developing better R L algorithms. To support both direction of research, we have created codebase, which basically contains a collection of unshaped robotics environments that people can test their R L algorithms on with nice little A P I's and tools to facilitate the research of automating environment shaping. And before I wrap up my talk, I want to discuss about possible counter arguments that people might have against ours. Going back to the beginning of my talk, I shared about the dream I have, creating a magical box that can automatically generate closed loop controllers for robots. And then I kind of implied that reinforcement learning will be powering this magical box

• 7:04:21Speaker 18

  in the future. However, I think some people might disagree with this, especially considering the research and popularity of doing manual data collection and retention learning. Some people might think that our dream of magical box will be realized not by automating R L, but by training some huge foundation model that consumes all these data sets collected by all these companies.

• 7:04:46Speaker 18

  However, I still believe in the power of RL as a tool to generate robust, generalizable, and especially superhuman behaviors that cannot be easily achieved with imitation learning. And also, the behaviors generated by RL pipelines can also be used to train those foundation models as well. Therefore, I argue that making RL easier to use will enable a virtuous data cycle for training better embodied intelligence. And with that, I would like to wrap up my talk today and I'm happy to engage in exciting discussions about our position. Thank you.

• 7:05:21Speaker 00

  And that's a wrap for ICML 2024 Part 1, our coverage on generative video world sim, diffusion, vision, reinforcement learning, and robotics. We're busy preparing for Latent Space Live at NAREEP's 2024 in Vancouver, so grab your tickets at lu.mus/lslive and see you there.

• 7:06:43Speaker 45

  Don't wanna overthink it baby It's just three You're by It's a fact

• 7:06:53Speaker 45

  Baby you got me ice and fire

• 7:06:59Speaker 45

  You're giving me wind and rain You're some kinda butterfly Baby you got me here now You whip up my appetite Don't leave me here high and dry Oh

• 7:07:20Speaker 45

  But I don't wanna jinx it baby Just not yet Oh yeah, so I'm glad Oh

• 7:07:32Speaker 45

  But I don't wanna miss you baby So I stand up and speak my mind Oh