Runway details latent diffusion for cheaper image synthesis
A Runway research page presents latent diffusion models, which run diffusion in a pretrained autoencoder's compressed latent space to cut the heavy compute cost of pixel-space diffusion while keeping image quality competitive.
Runway presents the paper "High-Resolution Image Synthesis with Latent Diffusion Models" by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer. The paper targets a known cost problem: diffusion models that work directly in pixel space often take hundreds of GPU days to train, and inference is slow because it needs many sequential evaluations.
The authors train diffusion models in the latent space of a pretrained autoencoder. A mild compression stage removes imperceptible detail, and the diffusion model then learns the semantic structure. The authors say this reaches a "near-optimal point" between reducing complexity and spatial downsampling. In ImageNet experiments run on a single NVIDIA A100 with a fixed compute budget, pixel-based LDM-1 trained far more slowly than models with downsampling factors of 4 to 16. Heavy compression at LDM-32 lowered sample quality. LDM-4 gave the strongest balance of sampling speed and FID.
Cross-attention layers let the models take general conditioning inputs, including text prompts and bounding boxes. The paper's text-to-image model was pretrained on LAION and fine-tuned on Conceptual Captions. The authors report competitive results on unconditional generation, inpainting, super-resolution, layout-to-image and semantic synthesis. A model trained at 256² resolution can also generate larger images such as 512×1024 by sampling convolutionally.
In the paper's 64→256 super-resolution comparison, the authors say SR3 produced more coherent fine structures, while LDM-SR rendered more realistic textures. The page does not include a publication date or code release details.