Learning to Memorize with Attributive and Associative Memory for Online Test-Time Adaptation of Vision-Language Models

Memory-based test-time adaptation (TTA) assigns streaming test samples into class-specific memory slots based on pseudo-labels predicted by models like CLIP, and retrieves them to facilitate subsequent predictions under distribution shift. However, this process introduces two challenges: (1) Each sample is hard-assigned to a single class based on CLIP's prediction, where inaccurate CLIP prediction leads to memory contamination that biases subsequent prediction. (2) Samples are evicted under biased selection due to fixed memory capacity, which risks discarding informative samples and undermining the efficacy of the memory. To address these challenges, we propose A²Memory (Attributive-Associative Memory for Test-time Adaptation). For challenge (1), we propose Attribute-centric Memory Construction that builds prior textual representations from class-shared representative and diverse visual attributes, and applies soft assignment to generate surrogate visual representations. For challenge (2), we design Class-wise Associative Memory that dynamically compresses streaming samples into fixed-capacity memory through gradient-based optimization and data-dependent retention, then retrieves sample-adaptive class prototypes for reliable inference. Extensive experiments demonstrate consistent improvements over state-of-the-art methods across 15 benchmarks.

Modern AI vision models like CLIP can recognize objects across a wide range of visual styles and conditions. However, when these models encounter a continuous stream of new images in the real world — such as photos taken under unusual lighting, weather, or artistic styles — their performance degrades over time because they cannot adapt on the fly. A common approach is to store recent examples in a memory bank and use them to improve future predictions. But this has two pitfalls: the model sometimes mislabels an example and stores it in the wrong category, polluting the memory; and when memory is full, useful examples may be discarded unfairly. We propose A²Memory, a system that addresses both problems. Instead of rigidly assigning each image to one category, it softly distributes each image across related categories based on fine-grained visual attributes — like fur texture or ear shape — reducing the damage from mislabeling. It also uses a mathematically principled compression strategy to ensure memory stays compact without losing the most informative examples. Tested across 15 benchmarks, A²Memory consistently outperforms existing methods, making AI vision systems more reliable when deployed in the unpredictable real world.