Skip to yearly menu bar Skip to main content


Poster
in
Workshop: Challenges in Deployable Generative AI

Predicting Task Forgetting in Large Language Models

Anat Kleiman · Jonathan Frankle · Sham Kakade · Mansheej Paul

Keywords: [ large language models ] [ Deep Learning ] [ forgetting ] [ Language Models ] [ catastrophic forgetting ] [ Empirical ] [ Finetuning ]


Abstract:

In this paper, we offer a comprehensive evaluation of forgetting in large language models (LLMs) during sequential learning of finetuning tasks in a pretrained model. We empirically track the degradation of performance across diverse tasks and find that the validation perplexity can be predicted using a linear function, regardless of the specific task, model architecture, or task order. This knowledge sheds light on the dynamics of knowledge acquisition and retention, offering practical implications for managing and mitigating task forgetting in LLM-based systems.

Chat is not available.