Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems

Many real-world tasks are non-verifiable—there is no objective ground truth, and quality must be judged subjectively—making reward design for RL difficult. Existing approaches based on scalar rubric scores or single comparisons are often noisy, poorly calibrated, or provide sparse learning signals. We introduce Tournament Style RL (TSRL), which constructs rewards from rubric-guided pairwise judgments against a fixed set of anchor responses, using win-rate as the reward for policy optimization. This aggregation of comparisons against anchor responses yields a signal that is more robust to the judge noise by stabilizing the reference frame, reducing the variance in reward. We test across four non-verifiable tasks and two backbone LLMs, and find that TSRL improves average win-rate by $+43.8$ points over the base model and $+22.8$ points over the strongest baseline. TSRL scales with the number of anchors, remains robust under weak or partially corrupted judges, the results are supported by blinded human preference studies.

Many of the things we want AI to do well have no single correct answer. Writing a story, telling a joke, giving supportive advice, or designing a web page all come down to subjective judgments of quality. This makes such tasks hard to train an AI on, because training needs a reliable signal for what counts as a good answer, and asking a computer to put a score on something subjective tends to be noisy and unreliable. We introduce a training method that scores answers by comparison instead of on an absolute scale. Before training, we gather a small, fixed set of reference answers that span a range of quality and rank them, much like players in a tournament. During training, each answer the AI produces is matched against every reference answer, and its score is simply the share of those matchups it wins, similar to an Elo rating in chess. Because the reference answers stay the same throughout training, the scoring stays stable and is far less affected by occasional mistakes from the automatic judge. Across four subjective tasks and two different AI models, our method produces clearly better results than existing approaches, and people who compared the outputs blindly agreed.