Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

Adapting language models (LMs) to new tasks via post-training carries the risk of degrading existing capabilities -- a phenomenon classically known as catastrophic forgetting. In this paper, toward identifying guidelines for mitigating this phenomenon, we systematically compare the forgetting patterns of two widely adopted post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a consistent trend across LM families (Llama, Qwen) and tasks (instruction following, general knowledge, and arithmetic reasoning): RL leads to less forgetting than SFT while achieving comparable or higher target task performance. To investigate the cause for this difference, we consider a simplified setting in which the LM is modeled as a mixture of two distributions, one corresponding to prior knowledge and the other to the target task. We identify that the mode-seeking nature of RL, which stems from its use of on-policy data, enables keeping prior knowledge intact when learning the target task. We then verify this insight by demonstrating that the use on-policy data underlies the robustness of RL to forgetting in practical settings, as opposed to other algorithmic choices such as the KL regularization or advantage estimation. Lastly, as a practical implication, our results highlight the potential of mitigating forgetting using approximately on-policy data, which can be substantially more efficient to obtain than fully on-policy data.

Language models are often adapted after their initial training to improve performance on specific tasks, such as instruction following or mathematical reasoning. However, this adaptation can degrade capabilities the model had already acquired, a phenomenon known as catastrophic forgetting. In this paper, we compare two widely used adaptation methods: supervised fine-tuning, which trains on example responses, and reinforcement learning, which trains using feedback on the model’s own responses. Across several model families and tasks, we find that reinforcement learning consistently causes less forgetting than supervised fine-tuning while achieving comparable or better performance on the new task. To explain this difference, we analyze a simplified model of how adaptation changes the model’s behavior. The analysis suggests that reinforcement learning is more robust because it learns from responses produced by the model itself, allowing it to improve on the new task without unnecessarily changing existing capabilities. We validate this explanation experimentally, showing that this use of model-generated training data is the main driver of reduced forgetting. Our results suggest that using data close to what the model itself would produce can be an efficient way to mitigate forgetting when adapting language models.