Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models

Large language models can memorize information that must be removed—ranging from copyright-sensitive content (e.g., book chapters) to personally identifiable information (e.g., income)—to ensure responsible and compliant behavior. Unlearning has emerged as an efficient alternative to full retraining, aiming to remove specific knowledge. However, users may still expect model to leverage the removed information when it is re-introduced in the prompt. Existing evaluations of unlearning methods focus on (1) the extent of forgetting of the target knowledge (forget set) and (2) performance preservation on the retain set (i.e., utility), but overlook this critical usability dimension. Through a systematic evaluation of six state-of-the-art unlearning methods, we show that they consistently degrade such contextual utility—the model's ability to use forgotten knowledge when it is provided in context. To address this, we augment unlearning objectives with a plug-in term that explicitly preserves contextual utility. Extensive experiments demonstrate that our approach restores contextual utility to near original levels while still maintaining effective forgetting and retain-set utility.

Large language models can memorize information during training that later needs to be removed—copyrighted book passages, outdated facts, or personal details someone wants taken down. Because retraining a model from scratch is prohibitively expensive, researchers developed "unlearning" techniques that selectively erase specific knowledge from an already-trained model. We found a hidden side effect: after unlearning, the model doesn't just forget the targeted information—it also loses the ability to use that information when a user explicitly provides it in the prompt. It's like asking someone to forget a phone number, then finding they can't read the number back even when you hand it to them on paper. This matters because people routinely paste documents into their messages and expect the model to work with them. We propose a small addition to existing unlearning methods that lets a model forget knowledge from memory while still being able to use the same content when it appears in the input. In our experiments, this restored the model's ability to handle such content to near-perfect levels, without weakening how well it forgets or harming overall performance—making unlearned AI more practical for real-world use.