Moonshot AI benchmark shows top models fail basic vision

Moonshot AI, the developer behind the Chinese assistant Kimi, has released PerceptionBench to isolate and evaluate the visual perception of multimodal language models. The benchmark strips away logical reasoning and external knowledge, focusing purely on ten distinct visual skills, such as depth perception, counting, and localization. Out of an internal pool of 17,000 verified questions, the researchers published 3,000 tasks. These include seemingly simple challenges like identifying symbols on a clock face or counting flowers inside a red box.

The evaluation of 16 frontier models revealed that none could break the 60 percent accuracy threshold. GPT-5.6 Sol led the group with a score of 59.7 percent, followed closely by Kimi K3 at 58.5 percent, Claude Fable 5 at 57.2 percent, Gemini 3.1 Pro at 56.2 percent, and GPT-5.5 at 55.8 percent. Open-source alternatives lagged further behind, with Qwen3.5-397B-A17B scoring 47.5 percent and GLM-4.6V hitting just 32.5 percent. Notably, models struggled heavily with hallucinating nonexistent objects; GPT-5.6 Sol scored only 26.9 percent in this category, while the smaller Gemini 3.5 Flash led the sub-test at 50.6 percent.

These findings align with previous research on visual limitations. In the WorldVQA benchmark, Gemini 3 Pro scored only 47.4 percent, while on the BabyVision benchmark, it scored 49.7 percent compared to a human baseline of 94.1 percent. Moonshot AI's team suggests that these failures stem from a "verbalization bottleneck" where visual data loses its fidelity during the translation into language.

For AI practitioners, this benchmark shifts the diagnostic focus for multimodal errors. Many failures previously attributed to flawed reasoning actually occur during the initial image-reading phase. By using the open-source PerceptionBench dataset and evaluation code, developers can pinpoint exactly where their models' visual processing breaks down, allowing them to address fundamental perception errors before attempting to debug complex reasoning pipelines.