FairJudge : An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: \textbf{limited adaptivity} to task and domain-specific evaluation criteria, \textbf{systematic biases} driven by non-semantic cues such as position, length, format, and model provenance, and \textbf{evaluation inconsistency} that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose \textbf{FairJudge}, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models \textbf{judging behavior itself as a learnable and regularized policy}. From a data-centric perspective, we construct a high--information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT--DPO--GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released at https://anonymous.4open.science/r/FairJudge-E7CB to facilitate future research.