Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL

Large Language Models (LLMs) that continue improving at test-time budgets far beyond their training budgets can solve harder problems by leveraging additional inference compute: we refer to this property as extrapolation. Standard on-policy RL operates on fixed problem distributions and training budgets, giving rise to a distribution shift between train and test that limits the resulting model's extrapolation capabilities. To address this, we introduce RC, an iterative decoding algorithm replacing standard autoregressive decoding that enables models to extrapolate to lengths an order of magnitude longer than those seen during training. RC exploits the asymmetry between summarization and generation capabilities present in LLMs to construct a decoding process that improves consistently over iterations. Its effectiveness can be further increased through training, which amplifies the model’s ability to perform summary-conditioned reasoning while avoiding the challenges of long-horizon RL. Empirically, training a 4B instruction-following model with RC using a 16k-token training budget improves performance on HMMT 2025 from 40% to 70% when evaluated with a 512k-token test budget, substantially surpassing comparably sized LLMs.

Large language models (LLMs) can solve harder problems when given more time to "think", but today’s training methods usually teach them to reason only within short limits. When asked to think for much longer, models often become repetitive, waste computation, or stop improving altogether. We wanted to understand how to train models that can continue making progress as more test-time compute (thinking budget) becomes available. We developed a method called Reasoning Cache (RC) that changes how models reason at test time. Instead of generating one extremely long chain of thought, the model periodically summarizes its progress, stores the important ideas, and then continues reasoning from the summary rather than the full history. This allows the model to revisit, refine, and extend earlier reasoning while avoiding the problems caused by very long outputs. We also trained models specifically to reason from these summaries, enabling them to effectively use much larger reasoning budgets than those seen during training. Our approach substantially improved performance on difficult mathematics and science benchmarks. For example, a 4-billion-parameter model trained with RC improved from about 40% to 70% accuracy on a challenging math competition benchmark when given a much larger reasoning budget at test time. More broadly, our work suggests that AI systems may become better problem-solvers not only by becoming larger, but also by learning how to reason more effectively over long periods of time.