Skip-It? Theoretical Conditions for Layer Skipping in Vision–Language Models

Vision–language models achieve incredible performance across a wide range of tasks, but their large size makes inference costly. Recent work has shown that multimodal processing contains significant redundancies, making it possible to skip certain layers with minimal performance loss. Yet current pruning techniques remain ad-hoc, relying on heuristics or hyperparameter sweeps rather than principled criteria for determining when layer skipping is beneficial. In this paper, we propose a unified framework that characterizes the redundancy conditions under which pruning can enhance efficiency without sacrificing performance. Central to our approach are experimentally verifiable and interpretable notions of redundancy that can be evaluated without requiring downstream task performance as a metric. Applying this framework, we corroborate prior findings that both early and late vision tokens are redundant across models, and we validate our conditions by showing they align with actual performance degradation. Beyond these empirical results, our framework provides a theoretically grounded understanding of redundancy in VLMs and unifies many of the ideas behind modern layer-skipping techniques.

AI models that can answer questions about images have become genuinely useful in everyday life, from identifying objects to helping doctors spot abnormalities in medical scans. But these models are slow and expensive to run, and speeding them up without breaking them is an open challenge. One promising approach is to skip certain layers of the model during processing, since not every layer contributes equally to getting the right answer. Researchers have proposed various ways to do this, but these have largely been trial-and-error, with no clear explanation of when or why skipping a layer is safe. Our work fills that gap. We develop a framework that identifies the conditions under which a layer can be skipped without hurting performance, and crucially, these conditions are concrete and checkable without needing to test the model on downstream tasks. We find that layers processing image information tend to be the most skippable at the very beginning and very end of the network, a pattern that holds consistently across different models. More broadly, our framework gives the field a shared, theoretically grounded understanding of redundancy, turning what was a patchwork of heuristics into something principled. As these models continue to grow in size and cost, knowing which layers we can safely skip becomes increasingly important.