PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the dissonance between benchmark saturation and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions that constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 5% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.

Artificial intelligence systems that understand images are often reported to perform very well on public benchmarks, but in real use they can still make obvious mistakes, such as missing important objects, confusing actions, or describing details that are not actually present. This paper introduces PerceptionRubrics, a new way to evaluate how carefully these systems perceive images. Instead of giving one broad similarity score to an image description, our method breaks each image into many clear, checkable rubrics. Some requirements cover essential facts that a good answer must get right, while others target common mistakes that models tend to make. We build a benchmark of 1,038 information-rich images and more than 12,000 detailed evaluation criteria. These criteria are created from carefully reviewed reference descriptions and are designed to reflect what people actually care about when judging whether an image description is reliable. Our scoring method strongly penalizes missing or incorrect essential facts, rather than allowing such errors to be hidden by an otherwise fluent description. By testing many leading multimodal large language models, we find that current systems can still be brittle on dense and detailed images, even when they perform well on existing benchmarks. PerceptionRubrics provides a more precise and human-aligned way to reveal these weaknesses, helping researchers better measure progress toward trustworthy AI systems that truly understand what they see.