TEMPO: Scaling Test-time Training for Large Reasoning Models

Large reasoning models (LRMs) remain static after training, limiting their ability to adapt to novel problems. Test-time training (TTT) addresses this by adapting model parameters on unlabeled test instances, continuously extending capabilities beyond offline training. However, existing TTT methods that rely on self-generated reward signals plateau quickly: without external grounding, the reward signal drifts as the policy evolves, causing both performance saturation and diversity collapse. We propose TEMPO, a dual-agent self-evolution framework that interleaves policy refinement by a reasoning agent on unlabeled questions with periodic recalibration of a verification agent (critic) on a labeled dataset. By formalizing this alternating procedure through the Expectation-Maximization (EM) algorithm, we reveal that prior single-agent self-rewarding methods are incomplete EM variants that omit the crucial verification step. Restoring this step tightens the evidence lower bound (ELBO) and enables sustained self-evolution. Across diverse model families (Qwen3 and OLMO3) and reasoning tasks, TEMPO improves OLMO3-7B on AIME 2024 from 33.0\% to 51.1\% and Qwen3-14B from 42.3\% to 65.8\%, while maintaining high output diversity. These results demonstrate that collaborative dual-agent self-evolution can push LRMs substantially beyond their static training boundaries. Code is available at \url{https://anonymous.4open.science/r/TEMPO-3A56}.