Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains

Reinforcement learning with verifiable rewards (RLVR) on foundation models has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. We introduce Rubric Curriculum RL (RcRL), a general self-improvement method for non-verifiable tasks that requires no new data, human annotations, or stronger teacher models. RcRL exploits the generation-verification gap, where judging good outputs is easier than producing them. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring while reducing reward hacking through a non-stationary objective. Whereas baselines plateau or collapse within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL outputs are preferred 70.5% of the time compared to the base model, with consistent gains across multiple creative writing benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all HealthBench training baselines, including instance-specific rubrics.

AI models have improved dramatically at tasks with clear right answers, like maths and coding, where their work can be checked automatically. But for open-ended tasks such as giving advice or creative writing there is no single correct answer, so it is hard to tell a model how to get better, and when a model grades its own work, it learns to "cheat" the grader rather than genuinely improve. We noticed that recognizing good writing is easier than producing it: a model can often tell which of two pieces is better even when it can't write one that good itself. We turned this into a structured teaching method. Normally a model is trained by scoring each piece of writing against a fixed checklist. Instead, our judge compares pieces in pairs and gradually introduces new qualities to look for, like a teacher adding one new lesson at a time. This moving target makes cheating much harder. Models trained this way keep improving where previous methods stall after a short while. Human readers preferred our model's stories 70% of the time, and the same approach also improved health-question answering, suggesting a general recipe for AI self-improvement without human input.