Most people can't spot Runway Gen-4.5 video
In a 1,043-person study it ran itself, Runway found that only 9.5% of participants could reliably tell Gen-4.5 clips from real footage, and the company says detection alone can no longer be relied on for trust.
Runway published research on Thursday, January 22, testing whether people can tell real video from clips made by its Gen-4.5 model. The study, called "The Turing Reel," came out the same week the company released image-to-video for Gen-4.5.
Each of the 1,043 participants watched 20 five-second clips in random order, 10 real and 10 generated, and judged whether each one was real. Runway says it generated every clip once, with no editing, regeneration or post-processing.
Only 99 participants (9.5%) got at least 15 of 20 right, the threshold Runway used for statistical significance (p < 0.05, binomial test). Overall accuracy was 57.1%, only a little better than chance. Accuracy was 58.0% on real clips and 56.1% on generated ones.
Results differed by subject. Videos of people, including faces, hands and actions, were easier to identify, at 58–65%. Animals and architecture scored 45–47%, below chance, meaning participants more often took generated clips for real ones.
The setup was narrow. Real clips came from the stock library Filmpac in five categories. Runway took the first frame of each real clip and fed it to Gen-4.5 on default settings. Clips were matched in resolution, and viewers could watch each one for at most 10 seconds. The study tested only Runway's own model and was run by the company, not checked by an outside group.
Runway concludes that "detection is an inadequate strategy for trust and verification." It notes that all of its outputs carry C2PA provenance metadata but says that standard "is not infallible." The company calls for stronger standards, new media literacy and updated editorial practices. It has also launched a public site where anyone can take the test.