Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Reasoning-capable large language models (LLMs) have recently been adopted as automated judges, but their benefits and costs in LLM-as-a-Judge settings remain unclear. Through controlled comparisons between reasoning and non-reasoning judges, we show that explicit reasoning substantially improves judgment accuracy on tasks requiring structured verification (e.g., math and coding), while offering limited or even negative gains on simpler evaluations and incurring significantly higher computational cost. These findings motivate that reasoning should be used selectively rather than universally, with awareness of possible distribution shift. We propose a Robust Adaptive Cost-Efficient Routing (RACER), which dynamically selects between reasoning and non-reasoning judges under a fixed budget by formulating routing as a constrained distributionally robust optimization problem. RACER explicitly accounts for distribution shift via a KL-divergence uncertainty set, admits an efficient primal--dual algorithm, and enjoys theoretical guarantees including uniqueness of the optimal policy and linear convergence. Extensive experiments show that RACER achieves superior accuracy--cost trade-offs under distribution shift.

Large language models are increasingly used to judge the quality of other models' answers, but making them “think step by step” is not always worth the extra cost. We found that reasoning helps a lot when judging difficult tasks such as math and coding, where careful checking is needed, but it brings little benefit, and can even hurt on simpler tasks. To use reasoning more wisely, we developed RACER, a system that decides when a judge should use a fast mode and when it should use a more expensive reasoning mode. Unlike standard routing methods, RACER is designed to stay reliable even when the kinds of questions change after deployment. Across several benchmarks, RACER achieved better accuracy-cost trade-offs than existing approaches, often matching or exceeding the accuracy of always using reasoning while spending much less computation. This work shows that reasoning should be used selectively, helping make automated evaluation both more efficient and more robust in real-world settings.