Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data

Estimating the generalization gap and developing optimization methods that improve generalization are crucial for deep learning models, for both theoretical understanding and practical applications. Leveraging unlabeled data for these purposes offers significant advantages in real-world scenarios. This paper introduces a novel generalization measure, $\textit{local inconsistency}$, derived from an information-geometric perspective on the parameter space of neural networks. A key feature of local inconsistency is that it can be computed without explicit labels. We establish theoretical underpinnings by connecting local inconsistency to the Fisher information matrix and the loss Hessian. Empirically, we demonstrate that local inconsistency correlates with the generalization gap. Based on these findings, we propose Inconsistency-Aware Minimization (IAM), which incorporates local inconsistency into the training objective. We demonstrate that in standard supervised learning settings, IAM enhances generalization, achieving performance comparable to that of existing methods such as Sharpness-Aware Minimization. Furthermore, IAM exhibits efficacy in semi- and self-supervised learning scenarios, where the local inconsistency is computed from unlabeled data.

(1) Problem. Deep learning models often perform well on the data they were trained on, but it is hard to tell in advance how well they will work on new, unseen data. Existing tools for predicting this "generalization gap" tend to be expensive: they require large amounts of labeled data, or they need the same model to be retrained many times from scratch — luxuries we rarely have in real-world settings. (2) Solution. We started from a simple intuition: a well-trained model should not change its predictions much when its internal settings are nudged slightly. We turned this idea into a measurable quantity called local inconsistency, which captures how sensitive a model's outputs are to small perturbations. Crucially, it can be computed from a single trained model using only unlabeled data. We then built a training method, IAM, that actively keeps this sensitivity low during learning. (3) Impact. IAM matches or beats leading methods in standard image classification, and shows especially clear gains when labels are scarce or absent — settings like semi-supervised and self-supervised learning. This makes it useful for real-world domains, such as medical imaging, where labeling data is costly.