Cross-Validation Error Dynamics in Smaller Datasets

Cross-validation (CV) is the de facto standard for estimating a model’s generalization performance, but in smaller datasets it exhibits an underappreciated quirk: across folds, training and test errors are strongly negatively correlated, while training and holdout errors show a moderate anticorrelation and test versus holdout errors are essentially uncorrelated. Herein, we document these phenomena empirically—on both real and synthetic datasets under AdaBoost—and introduce a simple generative model that explains them. By viewing each CV split as hypergeometric sampling from a finite population and incorporating an overfitting parameter δ that shifts expected errors on train, test, and holdout sets, we derive closed-form expressions for the covariances among observed error rates. Our analysis shows that sampling-induced anticorrelation dominates in small datasets, while overfitting contributes an additional negative term, thus accounting for the observed error dynamics. We discuss the limitations of our approach and suggest directions for more refined models and extensions to regression settings.