Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning

Model routing chooses which language model to use for each query. By sending easy queries to cheaper models and hard queries to stronger ones, it can significantly reduce inference cost while maintaining high accuracy. However, most existing routers treat this as a fixed choice among a small set of models, which makes them hard to adapt to new models or changing budget constraints. In this paper, we propose SCOPE (Scalable and Controllable Outcome Performance Estimator), a routing framework that goes beyond model selection by predicting their cost and performance. Trained with reinforcement learning, SCOPE makes reasoning-based predictions by retrieving how models behave on similar problems, rather than relying on fixed model names, enabling it to work with new, unseen models. Moreover, by explicitly predicting how accurate and how expensive a model will be, it turns routing into a dynamic decision problem, allowing users to easily control the trade-off between accuracy and cost. Experiments show that SCOPE is more than just a cost-saving tool. It flexibly adapts to user needs: it can boost accuracy by up to 25.7\% when performance is the priority, or cut costs by up to 95.1\% when efficiency matters most. We release the dataset and code at our project page: https://sullivan07043.github.io/SCOPE/.

Large AI models can answer difficult questions, but they are often slow and expensive to use. In many real applications, not every question needs the strongest model: simple questions can often be handled by cheaper models, while harder questions may require more powerful ones. The challenge is deciding which model to use before spending money to run it. We introduce SCOPE, a system that predicts how well different AI models are likely to answer a question and how much they are likely to cost. Instead of only memorizing model names, SCOPE looks at how each model performed on similar past questions and uses this behavior to make a better decision. This also helps SCOPE work with new models without needing to be retrained from scratch. By choosing models more carefully, SCOPE can reduce cost while maintaining strong performance, or improve accuracy when users are willing to spend more. This makes advanced AI systems more flexible, affordable, and easier to control in practice.