SaTeen: Learning Structural Alignment for Continual Test-Time Adaptation

Test-Time Adaptation (TTA) aims to reconcile model generalization in the presence of distribution shifts. Current TTA methods usually leverage sample uncertainty to select reliable samples for model adjustment via entropy minimization (EM). However, sample uncertainty often relies on a plausible metric and leaves many unreliable samples into EM process, potentially leading to model collapse. Importantly, these excluded samples incur biased data features of the shifted distribution in TTA. This paper introduces SaTeen, a Structural Alignment-based Test-Time Adaptation method that performs two-fold aligning the structures of test samples with the reliable reference structures. Specifically, the two-fold alignments are 1) Intra-sample structure alignment, where SaTeen maximizes cross-entropy discrepancy between a sample (reference) and its structure-disrupted counterpart, with the assumption of stable dominant features; 2) Inter-sample structure alignment, where SaTeen minimizes the reconstruction error of test samples in the reference subspace spanned by the Incremental PCA on reliable samples, with the assumption of stale intrinsic data manifold. Our extensive experiments demonstrate that SaTeen achieves the state-of-the-art performance across various scenarios for both TTA and continual TTA.

Machine learning models often perform well on data similar to what they saw during training, but their accuracy can drop when the real-world data changes, such as images becoming blurry, noisy, or visually different. Test-time adaptation aims to help a trained model adjust itself while it is being used, without needing labeled examples from the new environment. However, existing methods can sometimes rely too much on unreliable details in the input, which may cause the model to make increasingly biased predictions or even collapse to predicting the same class for many samples. We propose SaTeen, a method that helps the model focus on more stable structural information. SaTeen aligns each test sample with a structure-disrupted version of itself and also aligns samples with a reliable structure learned from previous test data. This two-level structural alignment makes adaptation more stable across changing environments. Our experiments show that SaTeen improves robustness in both standard and continual test-time adaptation settings, making deployed machine learning systems more reliable under real-world distribution shifts.