UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling

In real-world deployments of large language models (LLMs), balancing inference quality and computational cost has become a central challenge. Existing approaches tackle this trade-off along two largely independent dimensions: model routing, which switches among models of different scales to match request complexity, and test-time scaling (TTS), which adjusts inference-time compute within a fixed model for fine-grained control. However, this decoupled design introduces inherent limitations. Model routing yields coarse-grained, discrete performance changes due to the sparse set of model scales, while single-model TTS often encounters capacity ceilings and exhibits diminishing returns as compute increases. Moreover, treating the two mechanisms separately restricts adaptability in dynamic inference environments. To overcome these limitations, we introduce Unified Inference Scaling (UIS), which unifies model routing and TTS in a single optimization space. Building on this formulation, we propose UniScale, an online framework that models adaptive UIS as a contextual multi-armed bandit problem and learns inference policies via LinUCB. The framework incorporates efficiency-aware learning and cost modeling to ensure stable and scalable optimization over high-dimensional action spaces. Evaluation shows that UniScale effectively exploits the synergy in the UIS space to deliver a fine-grained and consistently better quality–cost trade-off across diverse, dynamic inference scenarios.

Running large language models (like the AI powering ChatGPT) in the real world requires a constant balance between response quality and computing cost. Currently, systems save costs in two independent ways: they either switch between small and large models depending on a question's difficulty, or they give a single model more time to "think." However, separating these methods creates a rigid system. Switching models only offers a few stark choices, while giving one model more time eventually hits a performance ceiling. To bridge this gap, we developed UniScale, a framework that combines both strategies into a single decision. UniScale uses a real-time learning algorithm that acts like an adaptive conductor, instantly picking the ideal combination of model size and thinking time for every incoming request. Our evaluations show that UniScale delivers a much smoother, consistently better balance between quality and cost. This allows AI services to stay highly accurate while minimizing expenses in dynamic, real-world environments.