Runway proposes two-stream keyframe propagation model
Runway researchers describe a two-stream video model that pairs global attention with local deformable aggregation to carry fine textures from keyframes, reporting 44% and 26% gains in FID and LPIPS for video inpainting.
Runway has published research on a model for propagating edits from keyframes across a video. The work targets editing tasks such as rotoscoping and object removal.
The researchers note that transformers and other attention-based methods, which pool features globally, already do well at carrying object masks from keyframes through a whole clip. They have a harder time reproducing high-frequency detail such as textures. Runway's hypothesis is that global attention is inherently biased toward low-frequency features.
To get around this, the team splits the problem into two streams. Low-frequency features interact globally, which keeps the model robust in hard cases such as large camera motion, where explicit alignment breaks down. High-frequency features interact locally through deformable feature aggregation. That local stream also uses information from the global stream to learn to detect and fix errors in the deformation field.
Runway evaluated the approach on inpainting. According to the team, it improves feature propagation within a single frame, as image inpainting requires, and also from keyframes to target frames. For video inpainting, the researchers report improvements of 44% in FID and 26% in LPIPS.
The page does not name the baselines, datasets or model sizes behind those figures. It also does not say whether or when the technique will appear in Runway's products. The post includes video results but no further quantitative detail.