On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs

Reinforcement learning (RL) fine-tuning has become a key technique for enhancing large language models (LLMs) on reasoning-intensive tasks, motivating its extension to vision language models (VLMs). While RL-tuned VLMs improve on visual reasoning benchmarks, they remain vulnerable to weak visual grounding, hallucinations, and over-reliance on textual cues. We show that simple, controlled textual perturbations—misleading captions or incorrect chain-of-thought (CoT) traces—cause substantial drops in robustness, and that these effects are more pronounced when CoT consistency is taken into account across open-source multimodal reasoning models. In contrast, closed models exhibit similar failure modes but maintain markedly greater robustness and reasoning consistency, suggesting that the gap reflects a shortcoming in current RL fine-tuning methods rather than an inherent limitation of the task. To better understand these vulnerabilities, we further analyze RL fine-tuning dynamics and uncover an accuracy–faithfulness trade-off: fine-tuning raises benchmark accuracy, but can simultaneously erode the reliability of the accompanying CoT and its robustness to contextual shifts. Although adversarial augmentation improves robustness, it does not by itself prevent faithfulness drift. Incorporating a faithfulness-aware reward can restore alignment between answers and reasoning, but when paired with augmentation, training risks collapsing onto shortcut strategies. Together, these findings highlight the limitations of accuracy-only evaluations and motivate training and assessment protocols that jointly emphasize correctness, robustness, and consistency in visually grounded reasoning.

Modern multimodal AI models can be trained to reason before answering questions about images, but they remain vulnerable to misleading language. We test this by presenting models with an image alongside a misleading hint: either a caption that gives incorrect context about the image, or an initial line of reasoning that is wrong. A visually grounded model should be able to identify and disregard this misleading information by relying on the image itself. Our results show that open-source models often fail to do so. Their accuracy drops substantially, and even when they produce the correct final answer, their written reasoning often conflicts with that answer, indicating weak consistency between reasoning and response. Closed-source models are considerably more robust, suggesting that this weakness is not inevitable, but instead reflects limitations in how current open-source models are trained. By examining the training process, we find a trade-off: improving benchmark accuracy can make a model’s explanations less consistent and more susceptible to misleading context. Simple interventions, such as training on adversarial examples or rewarding agreement between reasoning and answers, help in some cases. However, combining these interventions can backfire, as models may learn to exploit the reward rather than genuinely improve their visual reasoning. Overall, our findings suggest that reliable visual reasoning requires models to be trained and evaluated not only for correctness, but also for robustness to misleading language and consistency between their reasoning and final answers.