Poster
in
Workshop: Challenges in Deployable Generative AI
Predicting Task Forgetting in Large Language Models
Anat Kleiman · Jonathan Frankle · Sham Kakade · Mansheej Paul
Keywords: [ Finetuning ] [ Empirical ] [ catastrophic forgetting ] [ Language Models ] [ forgetting ] [ Deep Learning ] [ large language models ]
In this paper, we offer a comprehensive evaluation of forgetting in large language models (LLMs) during sequential learning of finetuning tasks in a pretrained model. We empirically track the degradation of performance across diverse tasks and find that the validation perplexity can be predicted using a linear function, regardless of the specific task, model architecture, or task order. This knowledge sheds light on the dynamics of knowledge acquisition and retention, offering practical implications for managing and mitigating task forgetting in LLM-based systems.