FairJudge : An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge improves agreement and F1 across several evaluation settings, reduces selected non-semantic biases, and achieves competitive or stronger performance than larger general-purpose LLMs on judge-oriented tasks.

Large language models are increasingly used to evaluate the outputs of other AI systems, but their judgments can be affected by irrelevant factors such as answer position, response length, formatting, or the identity of the model being evaluated. These biases can make evaluations unstable and sometimes contradictory across different evaluation settings. This paper introduces FairJudge, a trained LLM-as-a-judge designed to make more adaptive, less biased, and more consistent evaluations. By constructing specialized judging data and using staged training, FairJudge learns to focus more on the actual quality of responses rather than superficial cues. Experiments show that FairJudge provides more reliable judgments than standard instruction-tuned models and improves consistency across evaluation modes.