Paper #54: Rethinking Post-training Diversity Collapse: Is Diversity-preserving Post-training Enough?

Recent studies have shown that post-training procedures, such as preference-tuning and RLHF, could lead to reduced diversity among the outputs from post-trained large language models (LLMs), also known as diversity collapse. Although plenty of efforts have been made to mitigate such a collapse, the effectiveness of these efforts is measured by surface-level diversity metrics, such as lexical and embedding-based diversity. In this work, we introduce two non-superficial creative writing patterns, emotional arc and opening type, and show diversity collapse in those two patterns even when the LLM is trained with diversity-preserving post-training techniques. Preliminary diagnosis shows that such a collapse stems from implicit preference signals being hacked and thus amplified throughout the training process. We believe our work provides a foundation for a more in-depth analysis of diversity collapse in post-training due to reward hacking.