Understanding LLM generalization through fine-tuning

Recent work shows that it is still unclear when and how large language models (LLMs) generalize when trained on new data. We investigate how two distinct finetuning interventions - training on stated beliefs in multi-turn conversations and implanting factual information via Synthetic Document Finetuning (SDF) - impact related queries and downstream tasks. In the first setting, we finetune on liberal and conservative conversational data, evaluating across direct political questions, broader worldview queries, and a gender-bias downstream task. We find that ideological shifts diminish with semantic distance but influence model behavior on downstream tasks. In the second setting, we finetune models on synthetic documents about factory farming, the gig economy, and medical overdiagnosis. This shifts a model's sentiment on directly related queries but not its behavior on downstream tasks and on tangentially related queries. Together, these results suggest that finetuning on stated preferences generalizes to downstream actions more readily than finetuning on beliefs implanted as facts through pretraining-style documents.