DC-W2S: Dual-Consensus Weak-to-Strong Training for Reliable Process Reward Modeling in Biological Reasoning

In scientific reasoning tasks, the veracity of the reasoning process is as critical as the final outcome. While Process Reward Models (PRMs) offer a solution to the coarse-grained supervision problems inherent in Outcome Reward Models (ORMs), their deployment is hindered by the prohibitive cost of obtaining expert-verified step-wise labels. This paper addresses the challenge of training reliable PRMs using abundant but noisy "weak" supervision. We argue that existing Weak-to-Strong Generalization (W2SG) theories lack prescriptive guidelines for selecting high-quality training signals from noisy data. To bridge this gap, we introduce the Dual-Consensus Weak-to-Strong (DC-W2S) framework. By intersecting Self-Consensus (SC) metrics among weak supervisors with Neighborhood-Consensus (NC) metrics in the embedding space, we stratify supervision signals into distinct reliability regimes. We then employ a curriculum of instance-level balanced sampling and label-level reliability-aware masking to guide the training process. We demonstrate that DC-W2S enables the training of robust PRMs for complex reasoning without exhaustive expert annotation, proving that strategic data curation is more effective than indiscriminate training on large-scale noisy datasets.

When AI solves complex science problems, showing the correct steps is just as important as the final answer. To teach AI to evaluate its own step-by-step work, we usually need human experts to grade every single step, which is incredibly expensive. We could use cheaper, less accurate AI to do the grading, but this "noisy" data often confuses the learning process. To solve this, we created a new method that acts like a smart filter for noisy data. Our system finds the most reliable grading signals by checking for agreement in two ways: first, do multiple imperfect graders agree with each other? Second, do similar problems receive similar grades? By carefully selecting and learning from only the trustworthy parts of this data, we successfully trained AI to evaluate complex reasoning without needing exhaustive human grading. Ultimately, our work shows that when teaching AI, smartly filtering for high-quality examples is much more effective than simply feeding it massive amounts of messy data.