Is Task-Specific Training Necessary for Anomaly Detection?

Current state-of-the-art multi-class unsupervised anomaly detection (MUAD) methods rely on training encoder--decoder models to reconstruct anomaly-free features. However, we argue that such task-specific training is costly under distribution shifts, and that reconstruction-based residual scoring further faces a fidelity--stability dilemma. Existing training-free alternatives, in turn, remain prone to cross-category and cross-region mismatches in MUAD. Motivated by these limitations, we propose Retrieval-based Anomaly Detection (RAD), a task-specific training-free framework that stores anomaly-free features in a memory and detects anomalies through multi-level retrieval, matching test patches against the memory. Experiments demonstrate that RAD achieves state-of-the-art performance across four established benchmarks (MVTec-AD, VisA, Real-IAD, 3D-ADAM) under both standard and few-shot settings. On MVTec-AD, RAD reaches 96.7% Pixel AUROC with just a single anomaly-free image compared to 98.5% of RAD's full-data performance. Collectively, these findings overturn the assumption that MUAD requires task-specific training, showing that state-of-the-art anomaly detection is feasible with training-free memory-based retrieval. Our code is available at https://github.com/longkukuhi/RAD.

Visual defects are often rare, diverse, and difficult to predict. This makes it impractical to collect examples of every possible failure for automated inspection. We ask whether an anomaly detector can instead learn from only examples of what “normal” looks like, without being retrained for each new setting. Recent anomaly detection methods have achieved strong performance, but many need to be trained to learn the normal patterns in a given dataset. This can be costly when the data distribution changes. We also find a limitation in methods that detect defects by reconstructing normal images: making them reproduce normal images more faithfully can make their anomaly scores less stable. This motivates a simpler question: can a system detect defects by directly comparing a new image with stored normal examples? We introduce RAD, a task-specific training-free method that stores normal images in memory and retrieves the most relevant images and regions for comparison. RAD improves these comparisons by first finding globally similar normal images and then matching corresponding local regions across multiple visual levels. Across four standard benchmarks, RAD matches or surpasses leading trained methods and remains effective with only a few normal examples. These results suggest that visual inspection systems may often adapt by updating what they remember as normal, rather than being retrained.