Mind the Gap: Structure-Aware Consistency in Preference Learning

Aligning Large Language Models (LLMs) with human intent, whether through explicit reward modeling or direct methods such as DPO, fundamentally relies on minimizing a surrogate loss as a proxy for the true pairwise ranking objective. We prove that this reliance is flawed for the standard surrogate losses used: for the equicontinuous hypothesis sets characteristic of neural networks, *no* standard surrogate provides a meaningful consistency guarantee. Minimizing the surrogate loss to zero can leave the true ranking error arbitrarily high. To resolve this, we formulate LLM alignment within a margin-shifted ranking framework and derive $H$-consistency bounds showing that enforcing a confidence margin $\gamma$ is not merely beneficial but *necessary* for consistency. We further introduce Structure-Aware $H$-consistency and a corresponding objective (SA-DPO) that adapts the margin to the semantic distance between responses, preventing instability on near-synonymous pairs. Finally, we analyze the trade-off between the margin required for consistency and the model's finite capacity to satisfy it, revealing a strict hierarchy of surrogate losses: heavy-tailed surrogates (e.g., the Polynomial Hinge family) offer strictly superior consistency guarantees for capacity-bounded models compared to the logistic loss used in DPO. Experiments on UltraFeedback and Argilla DPO-Mix-7k confirm that SA-DPO consistently outperforms DPO and SimPO, with a 58.5\% head-to-head win-rate in downstream generation quality.

When AI language models are trained to match human preferences, surrogate loss functions are used in place of the original zero-one ranking objective, which is intractable to optimize directly. The most popular method, Direct Preference Optimization (DPO), and related approaches all rely on these surrogates as stand-ins for the true goal: correctly ranking responses. We prove that these surrogates are fundamentally unreliable for neural networks: a model can appear to perform perfectly on the surrogate while still ranking responses incorrectly. The root cause is that the model can "cheat" by making score differences vanishingly small, rather than actually learning which response is better. We fix this by requiring the model to maintain a minimum confidence gap between responses it ranks differently, and prove this gap is mathematically necessary. We further show that this gap should vary with how different the responses are: near-identical responses need only a small gap, while distinct ones require a large one. This leads to Structure-Aware DPO (SA-DPO), which adapts its margin to response similarity. We also establish a strict hierarchy among loss functions for capacity-bounded models. In experiments, SA-DPO consistently outperforms existing methods, achieving a 58.5% win-rate in head-to-head output quality comparisons.