On the Calibration of Learning to Defer to Multiple Experts

We study the calibration properties of multi-expert learning to defer (L2D). In particular, we study the framework's ability to estimate $\mathbb{P}(\rsm_{j} = \ry | \vx)$, the probability that the $j$th expert will correctly predict the label for $\vx$. We compare softmax- and one-vs-all-parameterized L2D, finding the former causes mis-calibration to propagate between the estimates of expert correctness while the latter's parameterization does not.