Theory of Continual Learning Against Data Poisoning Attacks

Continual learning (CL), where a model is trained on a sequence of data tasks, is increasingly being adopted across key fields such as large language models and image recognition, yet it remains highly vulnerable to data poisoning that triggers learning divergence or severe excess risk. Despite these threats, a principled theoretical foundation in CL for understanding attack and defense remains lacking. In this paper, we develop a theoretical framework to analyze strategic attacks and defenses in regularization-based CL, a cornerstone of recent CL theory. By framing the adversary-defender interaction as an online zero-sum game, we first establish a fundamental performance limit: no defense succeeds when an adversary poisons a linear proportion of tasks by injecting unbounded noise or pattern shifts in regularization-based CL. We then analyze two possibly defensible scenarios: infrequent attacks and bounded noise per attack. For the former regime, we propose a task-to-task verification mechanism to detect data poisoning and reduce cumulative bias for learning convergence. For the latter regime, we derive a robust defense that minimizes the model’s sensitivity to poisoned features, provably accelerating the convergence rate. Extensive experiments on realistic tasks further validate our theoretical results.

Many AI systems are not trained just once. They keep learning from new batches of data over time. This is useful, but it also gives attackers repeated chances to slip poisoned data into the learning process and slowly push the system away from reliable behavior. Our work theoretically studies when such attacks are impossible to defend against, and when they can still be handled. We model the conflict between the attacker and the learner as a game, where the attacker chooses harmful data changes and the learner chooses how strongly to protect previous knowledge. We show that some attacks are fundamentally too powerful, especially when many tasks are poisoned with very large or distribution-shifting changes. For more limited attacks, we design two defenses: one that compares consecutive learning updates to detect suspicious tasks, and another that reduces the model’s sensitivity to the most vulnerable directions in the data. Experiments on image classification tasks show that these defenses improve robustness against poisoning. These results help clarify how continual learning systems can be made safer when they are updated repeatedly after deployment.