Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning

We propose Re-FORC, an adaptive reward prediction method that, given a query, enables prediction of the expected future rewards as a function of the number of future thinking tokens. Re-FORC trains a lightweight adapter on reasoning models, demonstrating improved prediction with longer reasoning and larger models. Re-FORC enables: 1) early stopping of unpromising reasoning chains, reducing compute by up to 24\% compared to fixed-budget cutoffs, while maintaining accuracy, 2) optimized model and thinking length selection that outperforms the largest model alone---reaching 1.7 percentage points higher peak accuracy while needing up to 12\% less compute to match the largest model's accuracy, 3) adaptive test-time scaling, which increases accuracy by 8.8 percentage points (on average at maximum compute) over confidence-based baselines. Re-FORC allows dynamic reasoning with length control via cost-per-token thresholds while estimating computation time upfront.

Modern AI reasoning models can often improve their answers by spending more time “thinking.” But more thinking is not always useful: for an easy question, it may waste time and money, while for a difficult or high-stakes question, extra reasoning may be worthwhile. Today, it is hard to know in advance how much computation a model should spend on each individual query. We introduce Re-FORC, a method that helps reasoning models estimate whether additional thinking is likely to improve an answer. Re-FORC learns to predict how answer quality changes as a model spends more computation, allowing the system to make informed decisions such as when to stop reasoning early, when to sample a new solution, and which model size to use. In our experiments, Re-FORC enables improvements in accuracy--compute tradeoffs compared to baselines, while enabling user control over the tradeoff.