SPLIT-VLM: Salience-Guided Partitioning towards Local Coverage for Importance-Aware Token Dropping in Vision-Language Models

Large-scale vision–language models (VLMs) excel at multimodal reasoning, yet efficiency collapses when vision tokens—often orders of magnitude more than text—dominate compute and memory. Prior token-reduction strategies typically trade off salience (which is prone to position bias and incurs extra computation) against diversity (which can under-cover salient regions and is sensitive to hyperparameters). We present SPLIT, a theoretically grounded framework that jointly preserves salience and diversity while aggressively eliminating redundancy. SPLIT (i) estimates token importance via temporal shifts of hidden states across layers—eschewing attention scores and their biases; (ii) assigns adaptive region-level budgets to guarantee localized coverage; and (iii) selects tokens using a diversity score that prioritizes distinctive, non-redundant representations. Our analysis shows that adaptive budgeting yields tighter coverage guarantees than uniform allocation, and our selection rule maintains diversity without costly tuning. Empirically, SPLIT consistently outperforms state-of-the-art on image and video understanding benchmarks. On image understanding with LLaVA-1.5-7B, SPLIT preserves over 99\% accuracy with 192 vision tokens and about 92.8\% with only 64 tokens, demonstrating robust performance under severe token budgets. These results indicate that SPLIT delivers scalable, attention-score-free token reduction that makes multimodal reasoning substantially more efficient without sacrificing accuracy.

Vision-language AI models process images by breaking them into hundreds or thousands of small patches. In practice, these visual patches often make up over 80\% of what the model processes, even though most of them carry redundant information — much like reading every word of a document when skimming the key sentences would suffice. We developed SPLIT, a method that intelligently selects only the most informative image patches before feeding them to the model. Rather than relying on attention scores, which tend to be biased toward certain image regions, SPLIT measures how much each patch changes as it passes through the vision encoder. It then divides the image into regions and ensures each area retains enough patches to preserve local detail, while also favoring patches that are visually distinct from one another. The result is that SPLIT can discard nearly 90\% of image patches while preserving the model's original performance, making powerful vision-language AI substantially more practical to deploy in real-world applications.