The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity

This work investigates theoretically the interplay between interpolation and aggregation in regression. We establish that the $\gamma$-graph dimension characterizes learnability for a broad class of natural aggregation procedures. Furthermore, we prove that an extremely simple aggregation procedure, combining three interpolating hypotheses via the median, is optimal among all these aggregation procedures, and is strictly more powerful than proper learning. Finally, we show that some hypothesis classes are learnable only by aggregating infinitely many hypotheses or by using non-interpolating aggregation rules (which may predict outside the range of their inputs), and any finite interpolating aggregation fails to achieve even trivial performance.

Many modern machine learning systems can get their training data exactly right, yet still make mistakes on new data. We study whether combining several predictors that get the training data exactly right can make future predictions more reliable than using only one such predictor. We show that using the middle prediction from just three such predictors is already optimal among many natural ways of combining predictors. This simple combination can even require less training data than any method forced to return only one predictor. Our results help explain why combining predictors can be more powerful than relying on a single predictor that gets the training data exactly right.