MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

Multimodal large language models (MLLMs) are trained on massive multimodal data, making data unlearning increasingly important as data owners may request the removal of specific content. In practice, these requests often arrive sequentially over time, giving rise to the challenging problem of MLLM Lifelong Unlearning. However, most existing benchmarks are limited in scale and scope, failing to capture the complexities of MLLM lifelong unlearning. To fill this gap, we introduce the MLUBench, a large-scale and comprehensive benchmark featuring 127 entities across 9 classes under lifelong unlearning requests. We perform extensive experiments using MLUBench and reveal that existing unlearning methods suffer from severe, cumulative degradation. More critically, we further identify the unique challenge of this problem: unlike in unimodal models, MLLM lifelong unlearning is constrained by the need to preserve multimodal alignment. Continually unlearning from one modality could degrade the entire model. To alleviate this challenge, we propose LUMoE, an effective and efficient method. Experiments demonstrate that LUMoE significantly mitigates the degradation problem faced by baselines. The source code and the MLUBench dataset are open-sourced in this URL.

MLLMs learn from both images and text. Sometimes, people ask to remove certain data — like private photos. These requests come one after another, creating a problem called MLLM lifelong unlearning. We built the MLUBench, a large dataset to investigate MLLMs' behavior in the MLLM lifelong unlearning setting. We found that existing methods cause the model’s performance to steadily get worse. Worse, forgetting images can also harm text understanding, because the two are linked. We then developed LUMoE, a new method that helps the AI forget unwanted data while keeping its overall ability. Our code and data are publicly available.