SaTeen: Learning Structural Alignment for Continual Test-Time Adaptation

Test-Time Adaptation (TTA) aims to reconcile model generalization in the presence of distribution shifts. Current TTA methods usually leverage sample uncertainty to select reliable samples for model adjustment via entropy minimization (EM). However, sample uncertainty often relies on a plausible metric and leaves many unreliable samples into EM, potentially leading to model collapse. Importantly, these excluded samples incur incomplete data features of the shifted distribution in TTA. This paper introduces SaTeen, a Structural Alignment-based Test-Time Adaptation (SaTeen) method, by two-fold aligning the structures of test samples with the reliable reference structures. Specifically, the two-fold alignments are 1) Intra-sample structure alignment, where SeTeen maximizes cross-entropy discrepancy between a sample (reference) and its structure-disrupted counterpart, with the assumption of stable dominant features; 2) Inter-sample structure alignment, where SeTeen minimizes the reconstruction error of test samples in the reference subspace spanned by the Incremental PCA (IPCA) on reliable samples, with the assumption of stale intrinsic data manifold. Our extensive experiments demonstrate SaTeen achieves the state-of-the-art performance across various scenarios for both TTA and continual TTA.