Optimized Deferral for Imbalanced Settings

Learning algorithms can be significantly improved by routing complex or uncertain inputs to specialized experts, balancing accuracy with computational cost. This approach, known as learning to defer, is essential in domains like natural language generation, medical diagnosis, and computer vision, where an effective deferral can reduce errors at low extra resource consumption. However, the two-stage learning to defer setting, which leverages existing predictors such as a collection of LLMs or other classifiers, often faces challenges due to an expert imbalance problem. This imbalance can lead to suboptimal performance, with deferral algorithms favoring the majority expert. We present a comprehensive study of two-stage learning to defer in expert imbalance settings. We cast the deferral loss optimization as a novel cost-sensitive learning problem over the input-expert domain. We derive new margin-based loss functions and guarantees tailored to this setting, and develop novel algorithms for cost-sensitive learning. Leveraging these results, we design principled deferral algorithms, MILD (Margin-based Imbalanced Learning to Defer), specifically suited for expert imbalance settings. Extensive experiments demonstrate the effectiveness of our approach, showing clear improvements over existing baselines on both image classification and real-world Large Language Model (LLM) routing tasks.

Machine learning applications often use a "router" to direct incoming tasks to different specialized models, aiming to balance accuracy and computational cost. However, these systems frequently encounter an imbalance problem. If one, possibly cheap, model is generally the most accurate across domains, the router tends to default to it for almost every task. This leaves smaller domains, with possibly more costly models underutilized, lowering overall accuracy for a small cost saving. To address this, we developed a new routing method called MILD. Our algorithm mathematically penalizes the over-selection of the dominant model. By raising the confidence threshold required to use a model, MILD makes it easier for the router to safely hand off smaller tasks to highly specialized models. When tested on real-world large language models, our approach successfully prevented the system from collapsing to the cheap general model. By consistently matching tasks to the appropriate models, MILD balances computational resources needed to run a learning algorithm while increasing overall accuracy.