Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain

Large Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens. This approach improves visual grounding and mitigates language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and tokens.

AI systems that answer questions about images can give fluent responses while relying more on text patterns than on the picture. This can make them miss visual details or describe things that are not actually there. We introduce Visual Information Gain, a simple score that measures how much seeing the image helps the model predict the right answer, and use it to focus training on image-dependent examples and answer words. This helps models use visual evidence more faithfully, make fewer incorrect visual claims, and learn from less training information.