Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

Recent advancements in Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision-language tasks, yet the quadratic computational complexity arising from the vast number of visual tokens incurs significant memory and latency bottlenecks. While visual token reduction (VTR) strategies have been explored to mitigate this burden, existing methods overlook the positional and attentional consistency between the full and reduced sequences, resulting in a distorted representation. To this end, we propose RESTORE, a novel VTR framework that rectifies the positional and attentional distortions while maintaining efficiency. Specifically, we present a simple yet effective calibration method that restores lost visual attention by augmenting attention weights based on relative distances. We also introduce a distinctive anchor selection for token merging to mitigate information loss during feature averaging. Experimental results on multiple benchmarks demonstrate that our method consistently improves the accuracy of various reduction methods, achieving state-of-the-art performance while maintaining computational efficiency.

Recent AI systems that jointly understand images and text have achieved remarkable results, yet processing an image as thousands of small pieces, which are called "tokens", demands considerable time and memory. To ease this burden, existing approaches discard or combine many of these tokens, but they overlook a subtle side effect. Once the tokens are reduced, the model's focus drifts from the image toward the text, partly because it neglects image tokens that lie far from the text tokens. This loss of visual focus can lead the model to overlook what the image actually shows. To address this, we propose a method that restores the model's focus on the surviving image tokens while preserving efficiency. Specifically, we rebalance the model's focus so that distant image tokens regain their proper weight. We also select tokens that represent their neighbors when combining them, rather than choosing them randomly. Experiments across many models and benchmarks demonstrate that our method consistently recovers accuracy while remaining efficient. Ultimately, our work is a step toward running powerful image and text AI efficiently on small devices such as phones, not only on large servers.