ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4\% in Avg@16 and 7.0\% in Pass@128. Code is available at https://github.com/1229095296/ResRL.git.

AI assistants are getting better at solving hard problems, but the way they are often trained can make them too narrow. When training strongly rewards only one successful answer, the model may become less willing to try other useful paths. This hurts hard math, coding, and tool-use tasks, where several attempts or partial ideas may be needed. This paper introduces ResRL, a training method that helps models learn from wrong answers more carefully. Instead of treating every part of a wrong answer as harmful, ResRL compares it with successful answers and avoids punishing parts that are still useful, such as a reasonable intermediate step or a correct partial plan. It focuses correction on the parts that are more likely to be truly wrong. In tests covering math, programming, digital tool use, and multi-step tasks, ResRL improves accuracy while keeping the model’s answers varied.