ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Large language models can unintentionally memorize harmful, private, or outdated information from the massive amounts of text they are trained on. Removing this information is difficult because existing methods often require expensive retraining or damage the model’s overall abilities. In this paper, we introduce ZeroUnlearn, a new approach that removes unwanted knowledge in a more precise and efficient way. Instead of broadly changing the model, our method redirects sensitive information to a neutral state so the model no longer produces the original harmful response. At the same time, it carefully preserves the model’s general language understanding and reasoning skills. We also develop an efficient mathematical solution that allows the method to work with only a small number of examples. Experiments on several widely used language models show that ZeroUnlearn removes targeted knowledge more effectively than previous approaches while maintaining the model’s overall performance and usefulness.