TGV-KV: Text-Grounded KV Eviction for Vision-Language Models

Vision-Language Models (VLMs) inherit the auto-regressive generation paradigm and cache the keys and values (KV) of all previous tokens to accelerate inference, resulting in memory consumption that scales linearly with context length. This issue is particularly pronounced in VLMs due to substantial redundancy in the visual modality. Although KV cache eviction approaches can effectively reduce inference memory, they often incur significant performance degradation in VLMs, as most are designed for language models and overlook the inherent gap between text and vision. By systematically analyzing the modality gap in VLMs in this work, we argue that the importance of visual information should be grounded in textual guidance and accordingly propose a Text-Grounded KV Eviction method for VLMs (TGV-KV). TGV-KV comprises three submodules: (1) Text-Vision Budgeting (TVB) assigns budget to each layer based on the mutual information interaction. (2) Text-Weighted Ranking (TWR) assesses the priority of text and ranks vision importance based on weighted text-image attention. (3) Text-Prioritised Retention (TPR) policy strategically preserves text KV to avoid acute information loss. We evaluate TGV-KV across five models with different sizes and architectures, showing that TGV-KV preserves 99.2% full-KV accuracy on the VizWiz-VQA task with LLaVA-NeXT and boosts end-to-end throughput by 52.6% with an extreme retention budget of 5%. Code Link.

Vision-language AI systems can answer questions about images and videos, but they often need a large amount of computer memory when generating responses. This is because they keep a “memory notebook” of many previous image and text pieces so they do not have to recompute them later. For images and videos, this notebook can become very large, even though many visual pieces are repetitive or less relevant to the user’s question. Our work introduces TGV-KV, a method that decides which parts of this notebook should be kept and which can be safely removed. The key idea is simple: the text question should guide what visual information matters. Instead of treating image and text pieces equally, TGV-KV protects important text information and keeps the visual details most related to it. Across several vision-language models and tasks, TGV-KV greatly reduces memory use while preserving most of the original accuracy. This makes powerful multimodal AI systems faster and easier to deploy on limited hardware.