SCNS: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection

Custom diffusion models (CDMs) have demonstrated impressive success in visual personalization tasks by enabling the generation of user-specific concepts. However, existing CDMs typically assume that personalized concepts are static and rely on costly model merging or sequential updates that are prone to catastrophic forgetting as new concepts are introduced. To address these limitations, we propose a Submodular Concept Neuron Selection method (SCNS), to solve CDMs with continual personalized concepts, which formulates continual personalization as a constrained submodular optimization problem to select a minimal yet sufficient set of concept-specific neurons under diminishing returns. SCNS combines a Facility Location-based coverage objective to suppress semantic redundancy, a Fisher-weighted risk proxy to protect previously learned concepts, and a cost-aware greedy rule to balance stability and plasticity with extreme sparsity. Extensive experiments demonstrate that SCNS achieves state-of-the-art performance in image alignment and anti-forgetting, while enabling fusion-free continual personalization by modifying only 0.41% of the total parameters for each concept on average. Our implementation is available at SCNS.