MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field. The code and dataset are available at https://github.com/baochenfu/MMKU-Bench.

As real-world knowledge continues to evolve, the knowledge acquired by multimodal AI models during training can gradually become outdated. For example, a model may memorize a person’s job title, but such information may change over time. Existing research mainly focuses on enabling models to learn previously unknown knowledge, while paying much less attention to updating knowledge that models have already learned but that later changes. In addition, current evaluations are typically conducted within a single modality, such as text only, lacking a systematic analysis of knowledge consistency across images and text. To address these issues, we introduce MMKU-Bench, a large-scale benchmark for evaluating multimodal knowledge updating capabilities. The benchmark contains more than 25,000 knowledge instances and 49,000 images, covering both knowledge updating and unknown knowledge learning scenarios. It enables systematic comparisons of different methods across multiple types of knowledge. Based on this benchmark, we evaluate several representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE) methods. Experimental results show that SFT and RLHF are prone to forgetting previously learned capabilities when learning new knowledge, while KE methods better preserve existing capabilities but still exhibit clear limitations in continual knowledge updating.