Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also train Unison-Judge, an evaluation model well aligned with human judgments to achieve reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Unison will be publicly available at https://github.com/FudanCVL/Unison.

UMMs are increasingly designed to both understand information and generate images. However, current benchmarks only measure these abilities separately, missing how well they actually work together. To bridge this gap, we developed Unison, a unified benchmark with 2,169 tasks specifically designed to evaluate the synergy between comprehension and generation in UMMs. Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. It provides both unified and decoupled tracks for understanding and generation. We also propose Unison-Judge to achieve automatic evaluation.