Shortcut-Resistant CAM Distillation for Long-Tailed Recognition

Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. We revisit this problem through the lens of shortcut learning, where models prefer the easiest predictive cues (e.g., background or textures) over object-centric semantics, especially under scarce and biased supervision. We find that this tendency is amplified for tail classes: limited examples often share similar contexts, making non-semantic signals highly correlated and thus tempting shortcuts, whereas head classes with diverse appearances and environments encourage more stable object-focused representations. Motivated by this observation, we propose Shortcut-Resistant CAM Distillation (SRCD), a plug-and-play framework that transfers object-focused explanations from head to tail classes. SRCD operates in the Class Activation Map (CAM) space, where a CAM provides a class-specific spatial evidence map for a prediction. SRCD aggregates CAMs from a small set of head-class candidates into a shortcut-resistant teacher using an energy-model weighting based on coherence and concentration, and distills it to the tail-class CAM. We provide a theoretical analysis that quantifies shortcut reliance as shortcut-region evidence mass in CAM space and shows that SRCD suppresses tail shortcuts. Extensive experiments on long-tailed benchmarks consistently improve strong baselines. The code is available at \url{https://github.com/Haifeng3/SRCD}.

Real-world image datasets are often unbalanced: some categories have many examples, while others have only a few. This makes it hard for AI models to recognize rare categories well. When only limited examples are available, the model may rely on misleading clues, such as background, color, or texture, instead of learning the actual object. For example, if a rare animal often appears with a similar background in the training data, the model may mistakenly treat that background as an important clue. In this paper, we study this problem and find that rare categories are more likely to suffer from such shortcut learning, while common categories usually provide more diverse examples that help the model focus on the object itself. Based on this idea, we propose Shortcut-Resistant CAM Distillation (SRCD), a simple method that uses more reliable visual evidence learned from common categories to guide the learning of rare categories. SRCD helps the model pay more attention to object-related regions and reduce its dependence on misleading background cues, without changing the model architecture. We also provide theoretical analysis to explain why this method can reduce shortcut learning, and experiments on several long-tailed image recognition benchmarks show that SRCD consistently improves existing methods, especially for rare categories.