Runway explains how it built Gen-2
Runway has described how it moved from latent diffusion research to its Gen-2 text-to-video model, framing video generation as next-frame prediction aimed at eventually producing a feature-length film.
Runway's research team explained the path from its Latent Diffusion Models work, published in April 2022, to the release of Gen-2 in June 2023. The account is a Q&A, and the post does not name the person answering.
According to Runway, work on the Gen models began in earnest in September 2022, a month after Stable Diffusion was released. At that point CogVideo was the state of the art, and Runway's first goal was to beat it using a latent diffusion architecture.
The main obstacle was temporal consistency. Generating video one frame at a time with an image model causes flicker and gives no control over motion. So Runway started with a simpler problem. Gen-1 took an input video as conditioning to set the structure of its output. A few months later, Gen-2 dropped that requirement and generated video directly from text. A later update added image-to-video generation from any starting frame.
Runway compares Gen-2 to language models. In its view, a model trained to predict the next frame has to build a detailed representation of the visual world, much as next-token prediction does for text. The company's stated long-term goal is to let people build a two-hour film scene by scene, which means solving fidelity within a single shot first, then consistency across shots, then narrative.
Runway says it follows "incremental deployment." Gen-2 ran on Discord for about two months before it reached the web and mobile apps. Recent work has focused on control: starting frames, camera direction and the amount of motion in a clip. The post does not give benchmarks, model sizes or training details.