General Quantification of Covariate and Concept Shifts

Generalization under distribution shift remains a core challenge in modern machine learning, yet existing learning bound theory is limited to narrow, idealized settings and is non-estimable from samples. In this paper, we bridge the gap between theory and practical applications. We first show that existing definition of concept shift breaks when the source and target supports mismatch. Leveraging entropic optimal transport, we propose a key notion: $\gamma^\ast$-concept shifts, and derive a general error bound unifying covariate and $\gamma^\ast$-concept shifts, which applies to broad loss functions, label spaces, and stochastic labeling. We further develop estimators for these shifts with concentration guarantees, and the DataShifts algorithm, which can quantify distribution shifts and estimate the error bound in most applications - a rigorous and general tool for analyzing learning error under distribution shift.

Machine learning systems are often trained in one setting but used in another. For example, a model trained on data from one hospital, country, or time period may be deployed in a different one, where both the distributions of inputs (covariate, X) and the labels (concept, Y|X) can change. Understanding how such changes affect prediction error is important, but existing theory usually works only in simplified cases and often cannot estimate these distribution shifts on real data. In this work, we develop a more practical way to analyze this problem. We show that a common way of describing Y|X shift can fail when the training and test data cover different regions. To address this, we use entropic optimal transport, a mathematical tool for comparing distributions, to define a more general notion of Y|X shift. We then derive learning error bounds and provide estimators of the X (inputs) and Y|X (labels) shifts that can be computed from samples. Our DataShifts algorithm offers a general tool for quantifying these shifts and explaining model performance under distribution shift.