Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models

Vision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information, breaking spatial structure or collapsing diversity. We propose SpecFlow, a training-free framework that shifts the paradigm from destructive pruning to conservative condensation, strictly enforcing spatial coverage and statistical conservation to ensure stability. Treating visual tokens as nodes in a $k$NN graph, SpecFlow (i) computes a stable importance field via spectral heat flow to preserve structural coherence, (ii) allocates budgets via adaptive spatial partitioning to guarantee coverage, and (iii) aggregates discarded information into coreset sinks to maintain statistical conservation. The method is plug-and-play, requires no fine-tuning, and is compatible with FlashAttention. Experiments confirm that our SpecFlow outperforms SOTA methods across tasks, VLM architectures, and pruning ratios. Notably, LLaVA-1.5 with SpecFlow retains 95.6% of original performance despite pruning 88.9% of visual tokens, offering an exceptional efficiency-accuracy balance. Code is available at https://github.com/Lzy-dot/SpecFlow.

AI systems that understand both images and text are powerful but slow, because they break each image into hundreds of small pieces and process them all. A natural fix is to keep only the most important pieces, but deciding which to drop is tricky: naive choices often leave "holes" inside objects or discard useful background, hurting the model's answers. We introduce SpecFlow, a method that makes such AI systems much faster without any retraining. Instead of judging each piece on its own, SpecFlow lets importance spread between visually similar pieces, like heat flowing through a material, so meaningful regions are recognized as a whole. It then keeps a balanced selection across the whole image, and compactly summarizes the discarded pieces into two small "summary" tokens so context is not lost. Across many image and video tasks, SpecFlow keeps about 95% of the original accuracy while removing nearly 90% of the visual pieces, running notably faster and using less memory. This helps bring powerful multimodal AI to everyday devices and reduces its energy cost.