When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

Learning to Defer (L2D) enables a classifier to abstain from predictions and defer to an expert, and has recently been extended to multi-expert settings. In this work, we show that multi-expert L2D is fundamentally more challenging than the single-expert case. With multiple experts, the classifier's underfitting becomes inherent, which seriously degrades prediction performance, whereas in the single-expert setting it arises only under specific conditions. We theoretically reveal that this stems from an intrinsic expert identifiability issue: learning which expert to trust from a diverse pool, a problem absent in the single-expert case and renders existing underfitting remedies failed. To tackle this issue, we propose PiCCE (Pick the Confident and Correct Expert), a surrogate-based method that adaptively identifies a reliable expert based on empirical evidence. PiCCE effectively reduces multi-expert L2D to a single-expert–like learning problem, thereby resolving multi-expert underfitting. We further prove its statistical consistency and ability to recover class probabilities and expert accuracies. Extensive experiments across diverse settings, including real-world expert scenarios, validate our theoretical results and demonstrate improved performance.

Learning to defer (L2D) aims to enable classifiers to defer to human experts to enhance the reliability of decision systems, where incorporating more experts is expected to improve the capability of the prediction system. Surprisingly, we show that the opposite can happen: increasing the number of experts can hurt the performance of the classifier and finally hurt the overall performance of the decision system. We theoretically analyze the cause of this phenomenon and identify it as a fundamental challenge in multi-expert L2D. To address it, we propose PiCCE (Pick the Confident and Correct Expert), which trusts the empirically most reliable expert instead of assessing all experts. This reduces the multi-expert L2D problem to a single-expert-like setting. Our theory explains the statistical consistency of this method, and experiments show that PiCCE is effective.