Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision

Where do learning signals come from when there is no ground truth in post-training? We show that inference compute itself can serve as supervision. By generating parallel rollouts and converting them into reference estimates, models can learn without human labels—critically, even in non-verifiable domains like healthcare guidance where no programmatic checker exists. We call this framework Compute as Teacher (CaT) and it turns inference-time compute from parallel rollouts into supervision for RL training. The framework has two components: (1) reference estimation which aggregates rollouts into a pseudo-reference answer, and (2) reward derivation which converts that pseudo-reference into RL rewards. For (1), we explore a simple method we call synthesis, but the framework admits any aggregator. For (2), we introduce self-proposed rubrics for non-verifiable domains. These are binary, auditable criteria generated from the pseudo-reference and scored by an LLM judge. On HealthBench, models trained with CaT match or exceed inference-time aggregation quality while using 9× less test-time compute. Here, CaT also competes with learning from expert physician annotations, yielding up to +30% relative improvement over the initial policy. The framework extends naturally to verifiable rewards, matching the best existing baselines on MATH-500 in test-time RL and demonstrating 'drop-in' versatility across both types of domains.

Teaching language models specialized skills usually requires either human-written reference answers or automatic checkers but for tasks like medical advice, no single correct answer exists, experts disagree, and rule-based checking is impossible. Human annotation is expensive and slow, leaving models stuck in these domains. We propose Compute as Teacher (CaT). The idea is to have the model generate several different answers to the same question where these tend to disagree where the model is uncertain. The model then combines them into a single improved response, picking out correct parts and reconciling contradictions, like a committee resolving its own arguments. For tasks where no single right answer exists, the model also proposes its own checklist of yes/no criteria (e.g. "does the answer recommend seeing a doctor?") based on the combined response, and a judge model scores each candidate against the checklist. These scores become reward signals for reinforcement learning without the need for any human-annotated labels. On a medical advice benchmark, CaT-trained models match the quality of training with expert physician-written checklists despite using no expert input. Performance improves by up to 30% over the starting model, and the trained model uses 9× less compute when deployed than running the full procedure each time. The framework also works on math, suggesting a way to keep improving models further in domains where human supervision runs out.