Semantic Impact–Driven Visual Scheduling in Vision-Language Models

Vision-Language Models (VLMs) suffer from high inference latency due to long visual sequences. To enable efficient, on-demand utilization of visual information, we argue that visual necessity should be assessed by its semantic impact on the output distribution, rather than inferred from intermediate interaction signals such as attention weights. We propose a training-free framework based on token embedding subspace decomposition, which we term a prediction-conditioned Semantic Lens. Specifically, at fixed decoding intervals, we perform QR decomposition on the Top-K candidate token embeddings to construct an orthogonal semantic basis. We then introduce Semantic IImpact–Driven Visual Scheduling (SIVS), which measures how visual inputs impact model predictions by projecting visual-induced hidden-state variations onto this semantic lens. SIVS provides a geometrically grounded, impact-driven criterion for dynamic visual KV scheduling. Empirical results demonstrate that SIVS achieves ~87% visual KV compression while maintaining over 99% of model performance.

Vision-language models can answer questions about images, but they often process hundreds or thousands of image pieces while generating every word of an answer. This makes them slower and more expensive to use, especially when many image details are not actually needed for the current prediction. In this work, we study how to keep only the visual information that truly matters for the model’s answer. Our method asks a simple question during generation: does a piece of the image meaningfully change what the model is about to say? Instead of relying only on where the model “looks,” we measure whether visual information changes the model’s likely next words. Based on this signal, our method periodically keeps the most useful visual information and removes redundant parts from the model’s memory. Experiments show that this approach can remove most visual memory while preserving almost all of the model’s performance. This can make vision-language models faster, cheaper, and easier to deploy in real-world applications that require image understanding.