Selective Deferred Routing: Enabling Cost-Efficient Collaboration between Local SLMs and Remote LLMs

The rapid advancement of large language models (LLMs) has led to remarkable performance across diverse domains, making them indispensable assistants in daily life and work. Currently, LLM services are primarily accessed in two ways: (i) paid access to cloud-hosted LLMs, which are powerful but introduce nontrivial cost; and (ii) deployment of small language models (SLMs) on personal devices or small clusters, which, while less powerful, are sufficient for handling relatively simple tasks. To achieve a balanced trade-off between monetary cost and task performance, we propose Selective Deferred Routing, a paradigm that enables cost-efficient collaboration between local SLMs and remote LLMs. In this framework, a user request is first processed by the local SLM, which not only generates a preliminary response but also provides rich semantic representations of the request. A lightweight decision module then leverages this information to either adopt the initial response or route the request to the most suitable remote LLM for a higher-quality response. Extensive experiments across diverse model architectures and families, including both SLMs and LLMs, as well as datasets spanning multiple task scenarios, demonstrate that our approach consistently outperforms existing multi-LLM collaboration methods under a wide range of cost–performance trade-offs.

Modern AI assistants face a dilemma: powerful cloud-based models charge per query and grow expensive over time, while smaller models running locally on your laptop or phone are free but often struggle with harder questions. Existing solutions either guess the best model from the question alone (often inaccurate) or run every model in sequence until one looks good enough (wasteful). We propose Selective Deferred Routing (SDR): the small local model always answers first. Beyond just reading its answer, we tap into the model's internal "thoughts": rich signals revealing how well it understood the question. A lightweight judge uses these signals to decide whether the local answer is good enough to return, or whether the question should be forwarded to a more capable cloud model. Crucially, we train this judge to directly optimize the overall cost-quality trade-off, rather than rating each answer in isolation. Across diverse models and tasks, SDR consistently delivers higher-quality answers at lower cost than prior approaches. This makes personal AI assistants more affordable while preserving quality, helping individuals and small organizations get the most from both the free models they run themselves and the powerful ones in the cloud.