Moonshot open-sources Kimi Vendor Verifier
Moonshot has released a tool that checks whether third-party providers serve its open-weight Kimi models correctly, because the company found that scores for the same model differ widely depending on who hosts it.
Moonshot AI announced the open-source Kimi Vendor Verifier (KVV) together with its Kimi K2.6 model. KVV is a test suite for checking whether an inference provider's deployment of Kimi matches the official API.
The company says the work started after users reported unusual benchmark scores following the K2 Thinking release. Many of those cases came down to incorrect decoding parameters. Moonshot responded by requiring Temperature=1.0 and TopP=0.95 in Thinking mode on its own API, and by checking that thinking content is passed back correctly. A LiveBenchmark evaluation then showed a large gap between third-party APIs and the official one. After testing many infrastructure providers, Moonshot says the gap is widespread.
KVV first runs a pre-verification step that confirms parameter constraints are enforced. It then runs these benchmarks:
- OCRBench: a five-minute smoke test for multimodal pipelines
- MMMU Pro: checks how vision inputs are preprocessed
- AIME2025: a long-output stress test meant to catch KV cache bugs and quantization degradation
- K2VV ToolCall: measures how consistently tools are triggered (as an F1 score) and JSON Schema accuracy
- SWE-Bench: agentic coding, used internally but not open-sourced because it depends on a sandbox
Official Kimi API reference scores are 91.0 on OCRBench, 98.4 avg@32 on AIME2025 and 78.8 on MMMU Pro Vision.
A full run took about 15 hours sequentially on two NVIDIA H20 8-GPU servers. Moonshot says it works with the vLLM, SGLang and KTransformers communities to fix root causes and gives vendors early access to models so they can test before release. It also plans to keep a public leaderboard of vendor results. The post does not say which vendors have been tested or when the leaderboard will launch.