Context-Aware Reasoner: Enhancing Contextual Reasoning in Multimodal Large Language Models

Multimodal large language models (MLLMs) have demonstrated remarkable reasoning capabilities over internalized knowledge. However, current research overlooks contextual reasoning, the ability to reason based on the relevant information present in the context. To investigate this issue, we construct the Visual Contextual Reasoning Benchmark (ContextReasonV-Bench), and our analysis reveals two predominant failure modes: context neglect, where models rely on pre-trained knowledge instead of contextual information, and superficial pattern matching, where models exploit shallow correlations rather than genuine patterns. To address these limitations, we propose a two-stage approach that progressively establishes and reinforces contextual pattern acquisition. The first stage establishes an "analyze-then-solve" reasoning paradigm through supervised fine-tuning (SFT). We then employ a context-aware reinforcement learning (RL) framework that integrates context-aware reward modeling with hierarchical advantage estimation to encourage the model to identify genuine contextual patterns. This approach yields Context-Aware Reasoner (CAR), a model that achieves 38.14\% accuracy on ContextReasonV-Bench, a 22.09\% improvement over the base model, and generalizes effectively to unseen tasks.

Can AI models use information given to them, or do they simply rely on what they already know? We found that most multimodal AI models, systems that process both images and text, struggle with this seemingly simple challenge. Like a student who ignores a reference sheet during an exam and answers purely from memory, these models tend to overlook provided context, or latch onto surface-level cues instead of reasoning through the material carefully. To study this problem systematically, we built ContextReasonV-Bench, a new benchmark specifically designed to test contextual reasoning. Our experiments confirmed two recurring failure patterns across existing models: ignoring the provided context entirely, and exploiting shallow shortcuts rather than genuine understanding. We then developed CAR (Context-Aware Reasoner), trained in two stages to first learn the strategy of analyzing context before answering, then reinforce this through a reward system that explicitly encourages genuine contextual reasoning. CAR improves accuracy by over 22% compared to the base model and generalizes well to tasks it has never seen before.