Memory Adapters Enable Fast, Flexible Knowledge Unlearning in LLMs

We introduce memory adapters, a fine-tunable adapter for LLMs that allows isolating sequence-level gradient updates to small modular sets of parameters. The modularity of memory adapters enables instantaneous unlearning of any combination of documents, with empirically strong unlearning performance on the TOFU benchmark. Iterative unlearning is costless, and documents can even be flexibly included or excluded for individual queries in a batch. These unique properties are especially valuable for domains where data-removal requests may be unpredictable, granular, or user- or input-dependent.