Routing and Reasoned Evaluation with Large Language Models

Large language models (LLMs) are increasingly used to provide automated assessment signals for evaluating model-generated outputs. However, practical deployment faces three persistent challenges: heterogeneous reliability across models, substantial latency and token costs, and the absence of principled strategies for allocating evaluation resources. We introduce R$^2$Eval, a routing-aware automated assessment framework that formulates evaluation as a resource allocation and aggregation problem rather than relying on a single monolithic evaluator. R$^2$Eval combines difficulty-aware routing with reasoned evaluation signals to dynamically select evaluator models on a per-instance basis under explicit accuracy, latency, and cost constraints. Our study makes three contributions. First, we construct six difficulty-aware datasets spanning both reasoning-intensive (mathematics, logic, code) and non-reasoning (knowledge, roleplay, writing) tasks, with human-annotated reference assessments. Second, we provide a systematic empirical analysis of how reasoning traces produced by different evaluator models correlate with assessment outcomes, revealing substantial variance and systematic mismatches across difficulty regimes. Third, we develop and evaluate both offline and online routing strategies that adaptively allocate evaluation queries, achieving substantially improved accuracy–efficiency trade-offs compared to static baselines. Experiments across 19 language models demonstrate that R$^2$Eval significantly reduces evaluation cost and latency while maintaining close alignment with human assessments. These results highlight the importance of routing-aware automated assessment and establish R$^2$Eval as a scalable and reliable framework for large-scale model evaluation.

Large language models are increasingly used to judge or compare answers produced by other AI systems, but using a single powerful model as the evaluator can be expensive, slow, and unreliable across different types of tasks. This paper studies how to choose the right evaluator for each evaluation case instead of always relying on one fixed model. We introduce R$^2$Eval, a framework for comparing and routing automated evaluators. It evaluates both standard language models and reasoning models on tasks such as mathematics, logic, code, knowledge, roleplay, and writing. The framework also studies when a model’s intermediate reasoning helps its final judgment and when the reasoning can be misleading. Our results show that different evaluator models have different strengths: some are better for structured reasoning tasks, while others are more efficient or better suited for subjective writing and roleplay tasks. By routing each evaluation instance to an appropriate evaluator, R$^2$Eval can reduce cost and latency while maintaining close agreement with human judgments. This makes automated evaluation more practical for large-scale use.