Representation Unlearning: Forgetting through Information Compression

Machine unlearning seeks to remove the influence of specific training data from a model, a need driven by privacy regulations and robustness concerns. Existing approaches typically modify model parameters, but such updates can be unstable, computationally costly, and limited by local approximations. We introduce Representation Unlearning, a framework that performs unlearning directly in the model’s representation space. Instead of modifying model parameters, we learn a transformation over representations that imposes an information bottleneck: maximizing mutual information with retained data while suppressing information about data to be forgotten. We derive variational surrogates that make this objective tractable and show how they can be instantiated in two practical regimes: when both retain and forget data are available, and in a zero-shot setting where only forget data can be accessed. Experiments across several benchmarks demonstrate that Representation Unlearning achieves more reliable forgetting, better utility retention, and greater computational efficiency than parameter-centric baselines.

As AI models are trained on increasingly massive datasets, there is a growing need for "machine unlearning"—the ability to make a model forget specific information, like private user data or harmful content, upon request. Retraining a model from the ground up guarantees the data is forgotten, but it is incredibly slow and computationally expensive. Most current shortcuts try to tweak the model's billions of complex internal parameters to erase the memory, but these updates can be unstable, costly, and often damage the model's overall performance. In this paper, we introduce a much simpler and faster approach called "Representation Unlearning". Instead of modifying the model's complex internal wiring, we place a lightweight transformation—or filter—at the very end of its processing pipeline. This filter acts as an information checkpoint: it is trained to preserve and let through all the useful knowledge the model needs to keep, while completely blocking the specific information it was asked to forget. Our method proves to be highly reliable, preserves the model's usefulness, and requires drastically less computing power than existing methods. Crucially, we demonstrate that this filter can be successfully built even in "zero-shot" scenarios—situations where you only have the data you want to forget and no longer have access to the original retained data, making it a highly practical solution for real-world privacy compliance.