Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

Adapting Detection Transformers to Incremental Object Detection (IOD) poses a systemic challenge, as set-based optimization is inherently destabilized by sequential learning. In this work, we identify Gradient Dilution as the root cause of performance degradation, wherein optimization signals required to preserve old knowledge are progressively weakened. This phenomenon manifests as a cascading erosion of preservation gradients in magnitude, direction, and support coverage, driven by three tightly coupled factors: \textit{Signal Dispersion}, where foreground gradients are overwhelmed by background noise; \textit{Assignment Drift}, where stochastic query–target matching induces inconsistent gradient trajectories; and \textit{Support Attrition}, where gradients from retained samples insufficiently cover the old-class feature space, weakening decision boundaries under interference from new classes. To counteract this, we propose FAS, a unified framework that \underline{F}ocuses, \underline{A}ligns, and \underline{S}ustains gradient flow throughout incremental learning. Specifically, we introduce prior-injected queries to focus discriminative signals by filtering background interference at the source. We further propose deterministic anchor distillation to align query–target assignments and enforce semantic consistency across stages under unstable matching. Finally, we devise manifold-support replay to sustain distributional support of old classes, counteracting representational erosion induced by continual updates. Extensive experiments show that FAS restores robust optimization dynamics and outperforms state-of-the-art methods, achieving over 5.0 AP improvement in the challenging 40+10×4 incremental setting.

Object detection models are used to find and recognize objects in images. In many real-world applications, these models need to learn new object categories over time without forgetting previously learned ones. This is especially challenging for transformer-based detectors, where learning new categories can disrupt the signals needed to preserve old knowledge. In this work, we study why this forgetting happens. We find that, during incremental training, the useful signals that help protect old categories become weaker, noisier, and less complete. As a result, the model gradually loses its ability to recognize earlier categories. To address this problem, we propose FAS, a framework that helps the model maintain these useful signals. It reduces distraction from background regions, makes learning more consistent across training stages, and preserves representative information about old categories. Experiments show that FAS helps detection transformers learn new categories while better remembering old ones, achieving stronger accuracy than previous methods.