Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance

We study machine unlearning in large generative models by framing the task as density ratio estimation to a target distribution rather than supervised fine-tuning. While classifier guidance is a standard approach for approximating this ratio and can succeed in general, we show it can fail to faithfully unlearn with finite samples when the forget set represents a sharp, concentrated data distribution. To address this, we introduce Temper-Then-Tilt Unlearning (T3-Unlearning), which freezes the base model and applies a two-step inference procedure: (i) tempering the base distribution to flatten high-confidence spikes, and (ii) tilting the tempered distribution using a lightweight classifier trained to distinguish retain from forget samples. Our theoretical analysis provides finite-sample guarantees linking the surrogate classifier's risk to unlearning quality, proving that tempering is necessary to successfully unlearn for concentrated distributions. Empirical evaluations on the TOFU benchmark demonstrate that T3-Unlearning improves forget quality and generative utility over existing baselines, while training only a fraction of the parameters with a minimal runtime.

Modern artificial intelligence models can memorize private, copyrighted, or outdated information, but removing specific information after training is difficult without retraining the whole model from scratch. This paper introduces Temper-Then-Tilt Unlearning, a method that keeps the original model fixed while using a small classifier to steer generated samples away from data that should be forgotten and toward data that should be retained. We show theoretically and experimentally that this approach reduces unwanted data leakage while preserving useful model capabilities and requiring far less computation than retraining or fine-tuning the full model.