The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning

Machine unlearning aims to eliminate the influence of specific data from trained models to safeguard privacy. However, this presents a significant challenge in the context of continual learning (CL), where models update sequentially on dynamic datasets. A major limitation is that current certified unlearning algorithms fail to account for the complex, cumulative model evolution inherent to CL framework. In this work, we establish the first theoretical foundation bridging CL and machine unlearning. We formulate the CL's unlearning objective as the minimization of post-unlearning excess risk, which decomposes into CL excess risk and unlearning loss, characterizing the fundamental trade-off between preserving historical knowledge and targeted forgetting. Under mild assumptions, we first establish an upper bound for the CL excess risk in non-convex models. We then adapt two certified unlearning approaches, gradient-based and Hessian-based, to the CL framework. Our analysis reveals that while the gradient-based approach is less effective than the Hessian-based method in minimizing unlearning loss, it offers the distinct advantage of nearly zero storage overhead for enabling unlearning. This insight motivates a hybrid strategy that reduces storage costs while maintaining post-unlearning performance. Experimental results further validate our theoretical findings.

Machine learning models are increasingly updated over time as new data arrive, rather than being trained once on a fixed dataset. This creates a privacy challenge: when a user asks for their data to be deleted, the system should reliably remove the influence of that data without retraining the model from scratch. Informal deletion methods may appear to work in practice, but without theoretical guarantees, it is difficult to know whether the deleted data still affect the model. Our work studies certified unlearning in continual learning, where a model learns a sequence of tasks and information from earlier data can be carried into later model states. We develop a theoretical framework for measuring how well a continual-learning system performs after certified unlearning. This framework separates two goals: preserving useful knowledge from previous tasks and provably removing the influence of requested data. Building on this framework, we adapt and analyze two certified unlearning methods for continual learning. One is highly storage-efficient but less accurate, while the other gives stronger unlearning performance at higher storage cost. We then combine their strengths to reduce storage while maintaining theoretical guarantees and good post-unlearning performance.