Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

Mitigating sensitive and harmful outputs is fundamental to ensuring safe deployment of LLMs. Existing approaches typically follow two paradigms: Knowledge Deletion (KD), which erases undesirable information during training, and Distinguishable Refusal (DR), which steers models away from using sensitive knowledge during inference. Despite rapid progress, KD-based unlearning struggles with biased deletion due to suppressing specific token sequences as a substitute for complete knowledge removal, whereas DR-based unlearning risks the re-emergence of harmful knowledge because the underlying knowledge remains intact. To address these issues, we propose Distinguishable Deletion ($\mathrm{D^2}$), a paradigm that restricts the response distribution in the latent representation rather than specific tokens to erase undesirable knowledge, while distinguishing it from retained knowledge, enabling a refusal mechanism to handle unlearned inputs safely and coherently. To implement $\mathrm{D^2}$, we introduce an energy index that quantifies the presence of knowledge and the separation between unlearned and retained content. Mathematical and empirical analyses show that energy is both accurate and efficient, enabling Energy-based Unlearning Alignment (EUA) to enforce energy-boundary unlearning during training and apply an energy-based refusal mechanism at inference. Extensive experiments demonstrate that EUA significantly outperforms previous methods, indicating the superiority of $\mathrm{D^2}$. Our code is available at https://github.com/Puning97/EUA-for-LLM-Unlearning.

Existing unlearning methods typically focus on one or a few specific answers and therefore steer the model away from generating those particular outputs. However, such approaches overlook the fact that related knowledge is distributed in the latent space of LLMs, leading to biased unlearning: the target knowledge may not be fully removed, while non-target knowledge may be unintentionally degraded. We argue that unlearning should instead drive the relevant content toward an “unknown” state. That is, at each generation position, the model’s output distribution over the vocabulary should approach a uniform distribution, so that generation resembles random sampling from the dictionary rather than favoring any specific answer. To achieve this, we introduce an energy-based formulation and use free energy to quantify the uniformity of the model’s token distribution at each position, which serves as our optimization objective. Experiments show that our method achieves more thorough, reasonable, and effective unlearning than existing approaches.