Kimi's PerceptionBench: no frontier model hits 60%

Moonshot AI's Kimi team has released PerceptionBench, a benchmark that tests visual perception on its own, without reasoning or outside knowledge mixed in. The team says every question can be answered just by looking at the image.

The team did not start with a fixed list of skills. It studied where frontier models failed on 42 existing benchmarks, traced each failure to the first step that went wrong, and grouped the causes into 10 "atomic" perceptual capabilities: visual relation, counting, attribute, depth & 3D, localization, comparison, fine-grained recognition, context integration, OCR and hallucination.

The released set has 3,000 verified questions, drawn from an in-house pool of more than 17,000. About 60% were broken down from real model failures and 40% were newly written. Each category has between 255 and 330 questions, and difficulty is spread across levels. The team says the source benchmarks overlap little in which perception errors they catch (mean pairwise weighted Jaccard of 0.20), so no single existing benchmark covers perception fully.

Across the sixteen frontier multimodal models tested, none reached 60% accuracy. Hallucination about what is in the image was the weakest capability on average. The team also says models with nearly the same overall score can differ sharply in what they actually perceive.

The blog post does not list which models were tested or their individual scores. The dataset and evaluation code are on GitHub.