LMCleaner: Efficient and Certified Online Unlearning via Influence Propagation Truncation

Existing machine unlearning methods primarily focus on removing data influence after training completes, which is effective for many scenarios, but a complementary capability is needed when removal requests arise during ongoing training. We propose LMCleaner, an efficient and certified *online* unlearning framework that can process unlearning requests at any training step without waiting for training completion. Our key insight is that influence propagation can be decomposed into a trust region where linear approximation is accurate, and a residual that concentrates in a low-dimensional subspace and can be efficiently masked by calibrated noise. Building on this insight, we design an influence propagation truncation mechanism that treats mini-batch influence as atomic units, computes influence within a truncation window for efficient removal, and injects subspace-aware noise for certified privacy. Our theoretical analysis proves that the truncation residual decays exponentially with window size and that the unlearned model is $(\varepsilon, \delta)$-indistinguishable from retraining. Experiments demonstrate that LMCleaner achieves over $100\times$ computational savings compared to baselines while maintaining model utility and defending against membership inference attacks.

Modern AI systems like ChatGPT learn from enormous amounts of text scraped from the internet. Some of that text turns out to be harmful, copyrighted, or contains private information. Laws like Europe's "right to be forgotten" require companies to remove such data from their models. But how do you make a trained AI truly forget something? Retraining from scratch costs millions of dollars. Existing shortcuts only act after training is finished, by which point the unwanted influence has already spread through the model. We developed LMCleaner, a method that removes the influence of unwanted data while the model is still being trained, the moment that data is detected. Our key insight is that this influence does not need to be tracked through the entire training history. Most of it concentrates within a short recent window of training steps. The small remainder can be neutralized with carefully calibrated random noise. We mathematically prove that the cleaned model behaves as if it had never seen the unwanted data. In experiments, LMCleaner runs over 100 times faster than competing methods while preserving the model's overall abilities. This makes verifiable data deletion practical for today's largest AI systems, helping organizations comply with privacy laws and respond quickly when problematic training data is found.