Demystifying Language Model Forgetting with Low-rank Example Associations

Unintended forgetting of learned knowledge imposes an inherent risk on fine-tuning LLMs with new task data. In this paper, we adapt a classical framework for risk management to address forgetting by predicting their occurrence and perform targeted mitigation. Specifically, we show empirically that the associations between learned tasks and forgotten upstream examples are often well-approximated with low-rank matrices in diverse setups. Leveraging such empirical associations collected in the past, we predict forgetting of upstream examples when fine-tuning on unseen tasks with matrix completion. At the fine-tuning stage, we upweight examples predicted to be more forgotten for replay and demonstrate statistically significantly reduced forgetting over held-out, never-replayed upstream data. In summary, we show that predicting forgetting with low-rank example associations enables efficient examination and mitigation of forgetting.