Alignment-aware Data Selection for Unlearning in Contrastive Vision-Language Models

Recent advances in contrastive vision-language models have increased the need to selectively remove knowledge of specific data, drawing attention to machine unlearning. In this paper, we observe that unlearning performance in contrastive VLMs largely depends on the composition of the forget set. Based on this insight, we propose ALISE, a data selection framework that measures each forget sample’s alignment with both the retain set and the full forget set, and selects samples accordingly. Extensive experiments across diverse downstream applications demonstrate that ALISE facilitates removing target knowledge in contrastive VLMs while preserving model utility.