More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing

Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities, yet it remains plagued by catastrophic forgetting and model collapse. Empirically, we find that recent editors resilient over long horizons share the same core strategy: Lifelong Normalization (LN), which normalizes value gradients using running statistics. Removing LN causes immediate performance collapse, and we observe a counter-intuitive positive cumulative effect where early edits can promote the success of future edits. Yet the mechanism of LN remains a "black box", leaving its precise role in lifelong stability poorly understood. In this work, we provide the first theoretical account of LN in the lifelong regime. Our analysis reveals a self-reinforcing stability loop and proves that, when combined with ridge-regularized regression, LN yields parameter updates with asymptotic orthogonality and bounded norms, directly mitigating forgetting and systemic collapse. Based on these insights, we derive StableEdit, which strengthens this stability loop via an explicit warm-up stage and full whitening, improving long-horizon stability at minimal overhead. Extensive experiments validate our theory and demonstrate competitive performance. Our code is available at https://github.com/MINE-USTC/StableEdit.

Large language models often contain facts that become outdated, but retraining them from scratch is expensive. Model editing aims to update selected facts directly, yet repeatedly changing a model can accidentally make it forget other knowledge or degrade its general abilities.This paper studies a key component, called Lifelong Normalization, that appears in several recent editing methods and is crucial for keeping such long-term editing stable. We provide a theoretical explanation of why this component works: it uses information accumulated from past edits to better calibrate future updates, making the changes more controlled and less likely to interfere with each other. Based on this insight, we develop StableEdit, which adds a short preparation stage and applies more balanced updates. Across extensive experiments, StableEdit maintains strong editing performance and stability over tens of thousands to millions of facts. This can help keep deployed language models up to date, though such editing tools should be used with safeguards to prevent misuse.