Timezone: »

 
Predicting Task Forgetting in Large Language Models
Anat Kleiman · Jonathan Frankle · Sham Kakade · Mansheej Paul

@
in Challenges in Deployable Generative AI »
Event URL: https://openreview.net/forum?id=0BMg0OgNTP »

In this paper, we offer a comprehensive evaluation of forgetting in large language models (LLMs) during sequential learning of finetuning tasks in a pretrained model. We empirically track the degradation of performance across diverse tasks and find that the validation perplexity can be predicted using a linear function, regardless of the specific task, model architecture, or task order. This knowledge sheds light on the dynamics of knowledge acquisition and retention, offering practical implications for managing and mitigating task forgetting in LLM-based systems.

Keywords: Deep Learning · catastrophic forgetting · large language models · forgetting · Language Models · Empirical · Finetuning

Author Information

Anat Kleiman (Harvard University)
Jonathan Frankle (MosaicML / Databricks)
Sham Kakade (Harvard University and Amazon Scholar)
Mansheej Paul (Stanford University)

More from the Same Authors