Skip to content

Building Visual Intelligence — Andreas Blattmann & Dustin Podell, Black Forest Labs

Latent Space: The AI Engineer Podcast

1h 38m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode features Andreas Blattmann and Dustin Podell from Black Forest Labs discussing the journey of visual intelligence, from the invention of latent diffusion and stable diffusion to SDXL, stable video diffusion, SD3, and the latest Flux models. The guests explain how their research expanded from image generation into video, audio, and robotic action prediction, detailing the technical shifts from traditional diffusion to flow matching and transformer architectures. They also share stories from their early days, the creation of Black Forest Labs, and their vision for robotics and single-shot learning by 2027.

  • Data quality and filtering are critical components in training frontier models, often underappreciated compared to algorithm tweaks. 35:38
  • Flow matching simplifies the mathematical trajectory between noise and data, making fast sampling and scaling significantly easier. 1:08:36
  • Training on general video pre-training provides strong implicit 3D and physics priors for downstream tasks like robotics. 53:14
  • Targeted dataset curation and short training runs can unlock powerful multi-reference image editing capabilities without complex prompting hacks. 59:32

Chapters

  1. 0:00
    Introduction and Podcast Overview

    The hosts introduce the episode and set the stage for a discussion on visual intelligence and robotics.

  2. 0:41
    Founding of Black Forest Labs

    Andreas and Dustin discuss how they left Stability AI and founded BFL, including their early ties and motivations.

  3. 2:56
    Future Directions and Closing Remarks

    The episode concludes with reflections on upcoming projects, the broader AI landscape, and a teaser for future episodes.

  4. 3:56
    Funding Milestones and Revenue

    The hosts reveal BFL’s fundraising achievements and revenue milestones, highlighting the company’s rapid growth.

  5. 4:30
    Closing Remarks and Future Optimism

    The hosts wrap up the episode, expressing excitement for upcoming robotics developments and thanking the guests.

  6. 17:21
    Technical Evolution: From SDXL to Flux

    Andreas explains the progression from SDXL to newer models like Flux, detailing architectural shifts and multimodal capabilities.

  7. 36:10
    Video Generation and 3D Understanding

    The discussion focuses on BFL’s video diffusion work, data importance, and how simple video training can yield 3D world models.

  8. 40:00
    Robotics and Visual Intelligence Vision

    The team outlines their ambitions for robotics, visual intelligence, and closing the loop between observation and action.

  9. 43:30
    Diffusion Trajectory Optimization

    Discussion of how to focus compute on the most informative parts of the diffusion trajectory to improve training efficiency.

  10. 45:22
    Text in Images and Data Quality

    Exploration of the importance of captions and data quality for learning text in images, and the role of better models and data.

  11. 47:08
    Founding Black Forest Labs

    Recounting the founding story of Black Forest Labs, the early team, and the launch of Flux One.

  12. 53:37
    From Image Generation to Robotics

    Transition from image generation to robotics, explaining the vision of a general model that can handle video, audio, and action prediction.

Highlights

  • 6:44

    Naming Black Forest Labsstory

    The founders chose to base their new research lab in Freiburg at the foot of the Black Forest mountain range, where they were born.

  • 29:34

    The Untold SDXL Refiner Lorestory

    The guest reveals that the SDXL refiner was actually an unreleased, separate diffusion architecture that struggled with structure but excelled at high-frequency details.

  • 36:45

    From Video to 3D Understandingdata

    Training a video model on plain 2D videos and fine-tuning for only one thousand steps on 3D data produced a robust 3D model, demonstrating implicit world understanding.

  • 47:22

    Founding Black Forest Labsstory

    After a founding offsite in France where the team hacked together for a week, they kicked off training for Flux One, releasing a model that blew everyone away three months later.

  • 57:46

    Building Flux Contextstrategy

    The image editing capability started as a side project and presentation at a cabin in France, proving that a pre-trained model could learn general image editing with minimal training data.

  • 1:14:20

    Diffusion vs Autoregressive Data Efficiencyinsight

    The guest explains that while autoregressive models are more data efficient at training time, diffusion models allow for much faster sampling through step distillation.

  • 1:21:14

    Expanding to Robotics and Actionsstrategy

    By predicting robot state space parameters alongside video generation, a single multi-modal model can generate actions for real-world robots and computer games.

Quotes

  • “The data is one of the most important levers that you have as one who wants to train these models.”
    Unnamed speaker · 35:42
  • “We raised, all in all, four hundred fifty million dollars of funding so far.”
    Unnamed speaker · 19:47
  • “The refiner was actually a full diffusion model that the whole point of it was to take essentially what was not a rough image.”
    Unnamed speaker · 29:34
  • “So one, one central part of the paper was, where should we spend our compute?”
    Unnamed speaker · 44:35
  • “Simple, scalable is, is almost always the most effective, I would say, al- always the most effective.”
    Unnamed speaker · 46:11
  • “It took eight times less data to basically train on it.”
    Unnamed speaker · 1:25:05
  • “It's around like four hours of fine tuning, you need at least.”
    Unnamed speaker · 1:26:12
  • “A video upscaler, we, we talked about the temporal consistency.”
    Unnamed speaker · 1:28:18
  • “Twenty twenty seven will be the year of robotics.”
    Unnamed speaker · 1:30:06
  • “It's not AI. It's just Whisperflow.”
    Unnamed speaker · 1:35:02
  • Deepgram 50:47–52:30

About this episode

What can a model learn about the world by watching video and listening to it? Black Forest Labs trained FLUX 3 on images, video and audio together. Andreas Blattmann and Dustin Podell join swyx to explain why. A moving camera reveals the far side of an...

Transcript

0:00
1:38:23

20,816 words · 4 speakers

Topics

  • stable diffusion
  • SDXL
  • latent diffusion
  • Black Forest Labs
  • BFL
  • Flux Context
  • SD3
  • Latent Space
  • Germany
  • Flux Three

Page updated · transcript and summary are machine-generated