Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution

The rapid development of multimodal large language models (MLLMs) has created an urgent demand for more reliable and robust evaluation protocols, however, existing static benchmarks are prone to data contamination and performance saturation, which can result in inflated or misleading evaluation results. To address these limitations, we first introduce a graph formulation to represent both static and dynamic visual question answering (VQA) samples. Building upon this formulation, we propose Knowledge-Enhanced Benchmark Evolution (KBE), a dynamic multimodal evaluation framework that first analyzes the original static benchmark, then expands it by integrating multimodal knowledge, transforming the static benchmark into a controllable, dynamic evolving version. Crucially, KBE can both reconstruct questions by Re-selecting visual information in the original image and expand existing questions with external textual knowledge. By explicitly controlling the degree of question exploration, KBE enables difficulty-controllable evaluation across a wide range of model capabilities. Extensive experimental results demonstrate that KBE effectively mitigates data contamination and benchmark saturation, while providing a more comprehensive and flexible assessment of MLLM performance.

Traditional evaluation methods may suffer from issues such as inaccurate evaluation caused by models memorizing leaked benchmark data, or insufficient discriminative power due to overly simplistic test samples. To address these limitations, we generate new test samples by understanding and extending the original evaluation data. This enables us to dynamically construct different versions of evaluation benchmarks under various settings for dynamic assessment. Specifically, we represent the key information of a VQA problem using graphs and employ the KBE-DME framework to automatically select and expand relevant graph components, thereby achieving dynamic modifications of the benchmark. Furthermore, we control the relative difficulty of the dynamic benchmark through the number of edges in the graph structure. Experimental results on the dynamic benchmark show that model performance consistently declines as benchmark difficulty increases, while different models exhibit varying degrees of performance degradation. This provides a new perspective for evaluating models beyond traditional static assessment paradigms.