Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization

Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO) have significantly advanced the reasoning capabilities of large language models. Extending these methods to multimodal settings, however, faces a critical challenge: the instability of std-based normalization, which is easily distorted by extreme samples with nearly positive or negative rewards. Unlike pure-text LLMs, multimodal models are particularly sensitive to such distortions, as both perceptual and reasoning errors influence their responses. To address this, we characterize each sample by its difficulty, defined through perceptual complexity (measured via visual entropy) and reasoning uncertainty (captured by model confidence). Building on this characterization, we propose difficulty-aware group normalization (Durian), which re-groups samples by difficulty levels and shares the std within each group. Our approach preserves GRPO's intra-group distinctions while eliminating sensitivity to extreme cases, yielding significant performance gains across multiple multimodal reasoning benchmarks.

Modern AI systems are increasingly trained to "think step by step," including when they have to look at an image and solve a problem about it. A popular way to teach such a model is to let it produce several attempts at the same question and then reward the better attempts more than the worse ones, so that the model gradually learns which kinds of reasoning work. However, sometimes, all the attempts on a question turn out to be either all correct or all wrong, because the model can fail both at seeing what is in the image and at the reasoning that follows. When that happens, the usual way of comparing attempts becomes unstable, and a few unusual questions end up dominating training, drowning out the signal from the rest of the data. We then propose Durian, instead of comparing attempts only within one question, Durian first sorts questions by how hard they are along two intuitive axes — how visually complex the image is, and how confident the model itself appears in its own answer — and then shares the comparison scale within each difficulty group. This gives a more stable training signal and prevents a handful of extreme questions from steering the whole training. Across five widely used benchmarks for visual reasoning, math-with-diagrams, and visual hallucination, Durian consistently improves a strong open-source vision-language model by 11.3\% on average, while adding essentially no extra training cost. More broadly, our results suggest that how a model's rewards are normalized can matter as much as which rewards are collected — a perspective we hope will encourage more robust training methods for multimodal AI.