Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning

The core challenge of Test-Time Reinforcement Learning (TTRL) lies in estimating rewards without access to ground-truth supervision. Existing TTRL methods predominantly rely on majority voting to generate pseudo-labels, under the assumption that the most frequent answer among sampled trajectories is correct. However, we observe that this assumption frequently breaks down in complex reasoning tasks, where correct solutions often constitute a logical minority. As a result, rare yet correct trajectories are systematically undervalued by majority-voting-based approaches. To address this limitation, we propose Self-Reflective Test-Time Reinforcement Learning (SR-TTRL), a novel framework that leverages self-reflective verification to produce high-fidelity pseudo-labels. Specifically, given multiple sampled trajectories for a problem, SR-TTRL first groups trajectories according to their final answers and selects one representative from each group to form a candidate pool. Each candidate trajectory is then summarized to preserve its core reasoning steps while reducing verbosity. Finally, the model performs self-reflection over the candidate pool, critically evaluating and selecting the most plausible trajectory as the pseudo-label. Empirically, SR-TTRL achieves substantially higher pseudo-label fidelity and sample efficiency than prior majority-voting-based TTRL methods. Extensive experiments across diverse benchmarks and model families demonstrate that SR-TTRL consistently outperforms majority-voting baselines and significantly improves generalization to novel problems. For example, SR-TTRL improves the Pass@1 accuracy of Qwen3-8B on AIME24 from 29.1 to 55.8 (a gain of +26.7), exceeding standard TTRL by an additional +9.1. The code will be released at: https://github.com/JIA-Lab-research/SR-TTRL.

Current AI systems solve hard reasoning problems by generating multiple solutions and relying on a "majority vote" to pick the winner. However, for advanced logic, the correct answer is frequently a rare minority opinion. Our new framework replaces this popularity contest with real-time critical thinking during testing. When faced with a novel puzzle, the AI instantly organizes its different attempts, condenses them into core arguments, and self-reflects to discover the hidden, correct path. By enabling on-the-fly self-evolution without extra training, this method significantly boosts problem-solving accuracy on advanced benchmarks.