Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today's language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model's dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.

When we train AI language models to be helpful, we show them pairs of responses and tell them which one people preferred. The problem is that preferred answers often share superficial traits: they tend to be longer or more agreeable, even when these traits have nothing to do with answer quality. Popular training methods latch onto these surface traits as shortcuts, which helps explain why chatbots sometimes give unnecessarily long answers or tell users what they want to hear. We built a mathematical theory explaining why these shortcuts get learned. The theory shows that more training data does not fix this problem unless we change the structure of the supervision. As a remedy, we introduce tie training, where two responses are marked equally good even though one is longer or more agreeable, teaching the model to ignore these shortcuts without degrading its core performance. As AI assistants become part of everyday decision-making, they must be genuinely useful rather than merely polished or agreeable. Our work shows that some unwanted chatbot behaviors are not accidents of scale or bad prompting, but can arise from how preference data is structured. This gives researchers a clearer path toward training language models that are more reliable.