Reliable LLM-as-a-judge Evaluation via Transferable Calibration

LLM-as-a-judge (LaaJ) has become a standard tool for scalable evaluation of open-ended generation, but its reliability remains a central concern. Conformal inference provides statistically guaranteed intervals for LaaJ scores at a user-specified error level, yet this guarantee relies on exchangeability between calibration and target data. Density-ratio-estimation-based conformal inference methods theoretically recover target coverage under covariate shift, but in practical LaaJ settings the key bottleneck is how to represent high-dimensional text inputs for transfer. We first provide a theoretical account of why existing methods can fail in high-dimensional text spaces from a perspective of information overload. To address this, we propose Robust Weighted Conformal Inference (RoWCI), which transforms each input into a structured state Z that explains judgment error and performs weighted conformal inference in this state space rather than matching the full text distribution. We show theoretically that RoWCI recovers the target coverage guarantee in Z-space. Empirically, our practical RoWCI procedure provides coverage closer to the nominal level with more stable guarantees than conformal and non-conformal calibration baselines across evaluated model, response policy, and topic shifts.