Online Rubrics Elicitation from Pairwise Comparisons

Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and reasoning.

Training AI assistants to give helpful answers is hard: "helpful" has no single right answer to a question like "How should I prepare for a job interview?" A common way to train them is to evaluate their responses against a set of rubrics and rewarding them for satisfying each item using reinforcement learning. But these rubrics are written by humans in advance and never change. Just as students learn to game a fixed essay rubric, AI models invent new shortcuts during training that satisfy each item without truly answering well. We let the rubrics grow during training. At each step, our method compares two responses: one from the AI being trained, one from a stable reference version. We then ask an evaluator: "What meaningful difference is the current checklist missing?" Whatever it finds becomes a new criterion. It is like a teacher who, mid-semester, notices students exploiting a loophole and updates the grading guide to close it. Models trained this way score about 8% higher on standard benchmarks for chat quality, scientific reasoning, and math than ones trained with a fixed expert-written checklist. The criteria the method discovers target concrete failure modes like vague advice, padding to look thorough, and missing real-world context. The method plugs into any existing rubric-based training pipeline.