RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models

Efficiently routing queries to the optimal large language model (LLM) is crucial for optimizing the cost-performance trade-off in multi-model systems. However, most existing routers rely on single-model selection, making them susceptible to misrouting. In this work, we formulate LLM routing as the $\alpha$-VOR problem to minimize expected set size while controlling the misrouting risk, and propose a novel method -- RACER, extending base routers to output model sets that can be subsequently aggregated for improved output. In particular, RACER constructs nested model sets via augmented scoring and utilizes finite-sample concentration bounds to calibrate a threshold that allows for both variable set sizes and abstention. We theoretically prove that RACER achieves rigorous distribution-free risk control on unseen test data in a post-hoc and model-agnostic manner. Extensive experiments verify our theoretical guarantees and demonstrate that RACER consistently enhances downstream accuracy across a wide range of benchmarks.

Modern AI assistants often have access to several language models. One model may be better at math, another at Chinese knowledge, and another at reasoning questions. Asking every model is expensive, but asking only one can fail when the system chooses the wrong one. Our work introduces RACER, a method that helps an AI system decide which small group of models to use for each question. Instead of forcing a single choice, RACER selects a few promising models while keeping the chance of missing all correct models below a user-chosen limit. It can also decide not to answer when none of the available models seems suitable. After the models respond, their answers are combined to produce the final output. RACER can be added on top of existing model-selection systems without retraining the models. In experiments on math, knowledge, reasoning, and coding tasks, RACER improved answer accuracy while often using fewer model calls than asking the full model group. This makes multi-model AI systems more reliable and efficient.