FiRE: Fine-grained Ranking Evaluation for Machine Translation

Developing reliable machine translation (MT) systems hinges on our ability to distinguish superior translations from inferior ones. However, existing evaluation paradigms, whether limited to coarse overall rankings or misaligned with human preferences, fail to deliver interpretable, fine‑grained feedback in reference‑free settings. We present a Fine-Grained Ranking Evaluation method (FiRE) that leverages off‑the‑shelf large language models to perform criterion‑driven pairwise comparison across three complementary dimensions: faithfulness, fluency, and consistency of style, instead of producing a single holistic judgment. To enable rigorous meta‑evaluation of evaluation paradigms in the absence of any suitable testbed, we construct the first human‑annotated, reference‑free benchmark for fine-grained ranking evaluation, achieving substantial inter‑annotator agreement. Through meta‑evaluation on this benchmark and existing MQM datasets, FiRE demonstrably outperforms regression‑based and error‑analysis metrics in aligning with human comparative judgments, while providing more informative insights into translation quality. Finally, our examination of LLM evaluator biases (position and self-enhancement) and their handling of tied cases offers guidance for more nuanced MT evaluation. Code and benchmark resources are available at https://github.com/wygao8/FiRE-MT.

People who build machine translation systems constantly need to know which of two translations is better. Yet the automatic tools for judging quality are surprisingly blunt: many give a single score with no explanation, others just pick a winner, and most require a human-written "correct" translation to compare against. However, those references usually aren't available. We developed FiRE, which asks a large language model, the kind of AI behind chatbots, to compare two translations directly, as an attentive human would. Instead of one verdict, it weighs three qualities people actually care about: whether the meaning is faithfully preserved (with nothing invented), whether the text reads naturally, and whether it keeps the original tone. It then combines these into an overall judgment, with no reference translation needed. FiRE matches human preferences more closely than existing tools, and, unlike a bare score, explains why one translation wins. For example, exposing a system that sounds fluent but quietly fabricates facts. We also built the first human-labeled benchmark for this task and documented the biases of using AI as a judge, helping the community use these tools more responsibly.