Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories

Data-efficient learning aims to eliminate redundancy in large training datasets by training models on smaller subsets of the most informative examples. While data selection has been extensively explored for vision models and large language models (LLMs), it remains underexplored for Large Vision-Language Models (LVLMs). Notably, none of existing methods can outperform random selection at different subset sizes. In this work, we propose the first principled method for data-efficient instruction tuning of LVLMs. We prove that examples with similar cross-modal attention matrices during instruction tuning have similar gradients. Thus, they influence model parameters in a similar manner and convey the same information to the model during training. Building on this insight, we propose XMAS, which clusters examples based on the trajectories of the top singular values of their attention matrices obtained from fine-tuning a small proxy LVLM. By sampling a balanced subset from these clusters, XMAS effectively removes redundancy in large-scale LVLM training data. Extensive experiments across 4 target models, 2 proxy models, and 2 datasets show that XMAS consistently outperforms 10 baseline methods. Moreover, XMAS can discard 50% of the LLaVA-665k dataset and 85% of the Vision-Flan dataset while fully preserving performance of LLaVA-1.5-7B on 10 downstream benchmarks and speeding up its training by 1.2×. This is 30% more data reduction compared to the best baseline for LLaVA-665k. The project’s website can be found at https://bigml-cs-ucla.github.io/XMAS-project-page/.

Training AI systems that can understand both images and text often requires hundreds of thousands of examples, making the process expensive, slow, and energy-intensive. But large datasets often contain many examples that teach the model nearly the same lesson. This raises a practical question: can we train these systems just as well using much less data? We propose XMAS, a method for finding a smaller, more useful training set. Instead of judging examples only by their content, XMAS briefly trains a smaller model and watches how each example helps the model connect visual information with language. Examples that guide learning in similar ways are grouped together, and XMAS keeps a balanced set of representative examples from these groups. Across several image-and-text AI models and datasets, XMAS consistently selects better training data than existing approaches. It can remove half of one large training dataset and 85% of another while preserving the performance of training on all the data. This makes training faster and shows that better data selection, not just more data, can make powerful AI systems more efficient.