CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large Vision-Language Models

In large vision-language models, visual tokens typically constitute the majority of input tokens, leading to substantial computational overhead. To address this, recent studies have explored pruning redundant or less informative visual tokens for image understanding tasks. However, these methods struggle with pixel grounding tasks, where token importance is highly contingent on the input text. Through an in-depth analysis of CLIP, we observe that visual tokens within referent regions often exhibit low similarity to their textual representation. Motivated by this insight, we introduce LiteLVLM, a training-free, text-guided token pruning strategy for efficient pixel grounding inference. By reversing the ranking of CLIP's visual-text similarity, LiteLVLM effectively retains visual tokens covering the referent regions, while recovering context tokens to enable clear foreground-background separation. Extensive experiments demonstrate that LiteLVLM significantly outperforms existing methods by over 5% across diverse token budgets. Without any training or fine-tuning, LiteLVLM maintains 90% of the original performance with a 22% speedup and a 2.3$\times$ memory reduction. Our code is available at https://github.com/sejong-rcv/LiteLVLM.

How long did you wait for ChatGPT or Gemini to answer a question about an uploaded image? Or, if you have ever run these models on your laptop, how much memory did they take up? Such large vision-language models (LVLMs) can understand images alongside text, but this becomes slow and computationally expensive because they process hundreds or even thousands of visual tokens (small pieces of an image). Our work explores ways to accelerate LVLM inference by removing unnecessary visual tokens while preserving important ones. Our finding is contrary to common intuition. We found that visual tokens belonging to the object are surprisingly less similar to the input text – akin to the image pieces within a black bear being barely related to the text “a black bear.” Using this insight, we develop an efficient technique called LiteLVLM to segment text-described objects at the pixel level. This brings you faster, lighter, and easier-to-use LVLMs, even if your device is not the latest one.