Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck

Large language models (LLMs) have emerged as a standard paradigm for automated multilingual evaluation, yet exhibit systematic biases. In this paper, we identify ``translationese bias'', in which LLMs systematically favor machine-translated text over human-authored references, and this bias is particularly pronounced in low-resource languages. We attribute this bias to spurious correlations with (a) strong latent manifold isomorphism with English and (b) high predictive confidence. To mitigate these issues, we present DIBJudge, a robust fine-tuning framework that decouples robust features and bias representations by explicitly isolating spurious attributes into a dedicated bias branch and penalizing mutual dependence to enforce disentanglement. In particular, we present a vector-quantized compression that ensures the robust representation retains minimal and sufficient judgment-critical information. Extensive evaluations on multilingual reward modeling benchmarks and a specially designed translationese bias evaluation suite demonstrate that DIBJudge outperforms strong baselines and effectively mitigates translationese bias.

AI systems are increasingly used to judge the quality of answers in many languages. However, this paper shows that these AI judges can be unfair: they often prefer machine-translated text over text written by humans, especially for languages with less online data available. This does not necessarily happen because the translated text is better. Instead, machine translations often look more familiar to the AI system, for example because they follow English-like patterns. To address this problem, we introduce DIBJudge, a training method that helps AI judges focus more on the actual meaning and quality of an answer, rather than on clues that reveal whether the text was machine- translated. We test the method on several multilingual evaluation benchmarks and specially designed bias tests. The results show that DIBJudge substantially reduces this translation-related unfairness while still performing well as an AI evaluator. This work helps make multilingual AI evaluation more reliable and fair, particularly for underrepresented languages.