Moonshot's WorldVQA tests multimodal models' visual knowledge

Moonshot AI's Kimi team has released WorldVQA, a benchmark that tests whether multimodal large language models can correctly identify the specific entity in an image, rather than guessing from visual patterns.

The dataset has 3,500 image-question pairs. Each question has a single answer that can be checked. About 64% of the questions are in English and 36% are in Chinese. They cover nine categories, including geography, culture, entertainment, notable people, objects, transportation, brands, nature and sports. Example questions ask the model to name a bird species (Chestnut Shortwing), a phone model (iPhone 17 Pro) or a video-game character (Bayle the Dread). The data is split into "head" (common) and "tail" (rare) knowledge so researchers can see how accuracy falls as subjects get more obscure. Difficulty is graded as 31% easy, 41% medium and 28% hard. The team says every item went through several rounds of human checking.

According to the Kimi team, even state-of-the-art models often score below 50% on long-tail items. The team also measured calibration, meaning how well each model's stated confidence matches its actual accuracy. Every model it tested was overconfident, and most put their confidence in the 90–100% range. Moonshot's own Kimi-K2.5 had the best calibration, with an expected calibration error of 37.9% and a slope of 0.550. A perfectly calibrated model would score 0 and 1.0.

The blog post does not give per-model accuracy figures or name the other models it tested. The authors created the benchmark and their own model leads it, so independent replication will matter. The dataset, evaluation scripts and an arXiv paper are publicly available.