Training Data Efficiency in Multimodal Process Reward Models

Multimodal Process Reward Models (MPRMs) are central to step-level supervision for visual reasoning in MLLMs. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, incurring substantial training cost. This paper studies the data efficiency for MPRM training. Our preliminary experiments reveal that MPRM training quickly saturates under random subsampling of the training data, indicating substantial redundancy within existing MC-annotated corpora. To explain this, we formalize a theoretical framework and reveal that informative gradient updates depend on two factors: label mixtures of positive/negative steps and label reliability (average MC scores of positive steps). Guided by these insights, we propose the Balanced-Information Score (BIS), which prioritizes both mixture and reliability based on existing MC signals at the rollout level, without incurring any additional cost. Across two backbones (InternVL2.5-8B and Qwen2.5-VL-7B) on VisualProcessBench, BIS-selected subsets consistently match and even surpass the full-data performance at small fractions. Notably, the BIS subset reaches full-data performance using only 10% of the training data, improving over random subsampling by a relative 4.1%.

Many AI systems solve complex problems by producing step-by-step reasoning. To improve this process, researchers train models to judge whether each intermediate step in a solution is correct, but this often requires large and expensive annotated datasets. In this paper, we show that much of this training data is redundant: using carefully selected examples can achieve similar performance to using the full dataset. We propose a simple data selection method that keeps examples with a useful mix of correct and incorrect reasoning steps while avoiding unreliable labels. Experiments on multimodal reasoning benchmarks show that this approach can match full-data performance with as little as 10% of the training data, reducing the cost of training process reward models.