SCNS: Continual Personalization of Diffusion Models via Submodular Concept Neuron Selection

Custom diffusion models (CDMs) have demonstrated impressive success in visual personalization tasks by enabling the generation of user-specific concepts. However, existing CDMs typically assume that personalized concepts are static and rely on costly model merging or sequential updates that are prone to catastrophic forgetting as new concepts are introduced. To address these limitations, we propose a Submodular Concept Neuron Selection method (SCNS), to solve CDMs with continual personalized concepts, which formulates continual personalization as a constrained submodular optimization problem to select a minimal yet sufficient set of concept-specific neurons under diminishing returns. SCNS combines a Facility Location-based coverage objective to suppress semantic redundancy, a Fisher-weighted risk proxy to protect previously learned concepts, and a cost-aware greedy rule to balance stability and plasticity with extreme sparsity. Extensive experiments demonstrate that SCNS achieves state-of-the-art performance in image alignment and anti-forgetting, while enabling fusion-free continual personalization by modifying only 0.41% of the total parameters for each concept on average. Our implementation is available at SCNS.

Personalized AI models can generate images of user-specific concepts, like pets or unique objects. Traditional methods struggle to learn new concepts continuously without forgetting previous ones and often require storing sensitive data. We introduce SCNS, which smartly selects a small set of key neurons to update for each new concept. This approach preserves earlier knowledge, avoids costly model merges, and uses minimal parameters (~0.41% per concept). Experiments show SCNS produces high-quality, accurate images while efficiently supporting continual personalization.