Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation

Large Vision-Language Models (LVLMs) extend large language models with visual understanding, but remain vulnerable to hallucination, where outputs are fluent yet inconsistent with images. Recent studies link this issue to language bias—the tendency of LVLMs to over-rely on text while neglecting visual inputs. Yet most analyses remain empirical without uncovering its underlying cause. In this paper, we provide a systematic study of language bias and identify its root in modality misalignment during training. Our analysis shows that both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) often prioritize textual improvements, which may cause LVLMs to overly lean toward language modeling rather than balanced multimodal understanding. To address this, we propose two simple yet effective methods: Language Bias Regularization (LBR), which mitigates language bias through regularization during instruction tuning, and Language Bias Penalty (LBP), which penalizes language bias in the DPO training process. Extensive experiments across diverse models and benchmarks demonstrate the effectiveness of our approach. LBR consistently improves performance on over ten general benchmarks, while LBP significantly reduces hallucination and improves trustworthiness. Together, these methods not only mitigate language bias but also advance the overall alignment of LVLMs, all without introducing any additional data or auxiliary models. Our code is publicly available at https://github.com/lab-klc/HAVAE.

Large vision-language models can understand both images and text, enabling applications such as visual question answering and image-based assistants. However, these models often “hallucinate” — they generate responses that sound convincing but do not match the image content. We find that a major reason for this problem is that these models tend to rely too heavily on language patterns while paying insufficient attention to visual information. In this work, we systematically study why this happens and show that common training methods can unintentionally encourage models to focus more on text generation than on accurate image understanding. To address this issue, we introduce two simple training methods that reduce this language over-reliance during different stages of model training. These methods help models better balance visual and textual information without requiring extra data or additional models. Experiments on a wide range of benchmarks and model architectures show that our approach consistently improves overall performance and significantly reduces hallucinations. By making vision-language models more reliable and trustworthy, our work helps advance safer and more accurate multimodal AI systems.