Runway's Soundify matches sound effects to video
Runway researchers have introduced Soundify, a system that pairs video with studio-quality sound effects by using CLIP as a zero-shot detector, without training on audio-visual correspondence or generating audio.
Runway Research has described Soundify, a system that matches sound effects to video footage. It is aimed at one of the most manual parts of video editing.
The work began with interviews with 10 professional video editors. The editors described adding effects and ambient sound to footage as tedious and time-consuming. They named three main bottlenecks:
- finding suitable sounds
- aligning those sounds precisely to the video
- tuning parameters such as pan and gain frame by frame
Soundify takes a different approach from most earlier research. According to the researchers, prior work has mostly either learned audio-visual correspondence from large datasets or synthesized audio from scratch. Soundify instead draws on labeled, studio-quality sound effects libraries. It extends OpenAI's CLIP, a neural network known for zero-shot image classification, into what the authors call a "zero-shot detector." They say this produces high-quality results without resource-intensive correspondence learning or audio generation.
The authors are David Chuan-En Lin and Nikolas Martelaro of one affiliated institution, and Anastasis Germanidis, Cristóbal Valenzuela and Yining Shi of a second. Germanidis and Valenzuela are Runway co-founders.
The post shows example visualizations and results. The material provided does not include quantitative evaluation figures, details on how Soundify handles pan and gain, or any plans to ship the system in Runway's products.