TPV: Parameter Perturbations Through the Lens of Test Prediction Variance

We introduce test prediction variance (TPV)—the first-order sensitivity of a trained model's outputs to parameter perturbations—as a unifying framework for analyzing post-training robustness. TPV's trace form $\mathrm{Tr}(H_{\mathrm{eff}}C)$ separates the geometry of the trained model $H_{\mathrm{eff}}$ from the perturbation covariance $C$, placing SGD noise, label noise, quantization, and pruning under a single lens. The resulting expressions recover the wide-minima hypothesis for SGD and quantization noise, and yield a distinct Jacobian-spectral characterization for label noise connecting label-noise TPV with benign overfitting in nonlinear networks. Theoretically, we prove that training-set TPV converges to its test-set counterpart in the overparameterized limit, irrespective of generalization performance, providing the first result that prediction variance under local parameter perturbations can be inferred from training inputs alone. Empirically, this stability holds far more broadly, including at very low widths. Further, TPV correlates well with test loss, enabling practical applications: JBR, a label-free pruning criterion derived from TPV geometry matching state-of-the-art baselines; and training-set based model selection signal for in-distribution and transfer learning scenarios. \href{https://github.com/devansharpit/TPV/tree/main}{Code Available Here}

When a trained AI model is deployed in the real world, small disturbances inevitably nudge its internal settings — noise during training, rounded numbers from low-precision hardware, mislabeled examples it gets fine-tuned on, or parts of the network being removed to make it run faster. Each of these has traditionally been studied separately, with its own theory and tools, making it hard to see what they have in common or to predict which trained models will hold up best. We introduce a single quantity, called Test Prediction Variance (TPV), that measures how much a model's outputs change when its internal settings are slightly disturbed. We show mathematically that all four disturbance types act through the same underlying mechanism, just with different fingerprints, and we prove that TPV can be estimated using only the model's training data — no test data or true labels required. This gives practitioners a single tool to compare candidate models before deployment: predicting which will survive label errors during fine-tuning, deciding which parts of a network are safe to remove, or picking the most robust training recipe — all without needing held-out labeled data, which is often expensive or unavailable.