Adversarial Training for Process Reward Models

Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs (APRM), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$), while $R$ concurrently learns to detect them. This interaction yields progressively harder negatives for $R$, improving it's robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, APRM improves solver accuracy by $+3.4$ percentage points (pp) over the strongest PRM baseline. APRM achieves gains of $+5.3$ pp on out-of-distribution tasks.

AI systems can now work through hard reasoning problems, like competition math, one step at a time. But they still make mistakes that look convincing while being wrong, which is risky in areas like medicine, finance, or science. One way to catch these is to train a separate "checker" model that reviews each step and flags the bad ones. The trouble is that building a good checker usually needs large amounts of hand-labeled examples of mistakes, which is expensive to produce, and a checker trained on a fixed set of mistakes tends to miss new or subtler ones. As the problem-solving AI improves, its errors get harder to spot and the checker falls behind. We train the checker by pitting it against an opponent. One AI model learns to slip plausible-but-wrong steps past the checker, while the checker learns to catch them. As the opponent invents harder, more subtle mistakes, the checker is forced to keep improving, so the two get better together, with no hand-labeled mistakes needed. To decide whether a generated step is genuinely wrong, we use a fast set of automatic mathematical checks rather than a person or a large AI judge. Across several math and science reasoning benchmarks, a checker trained this way makes the problem-solver more accurate, with the biggest gains on problem types it was not trained on, showing that it generalizes to new kinds of errors.