Capability Traps in DPO

Direct Preference Optimization (DPO) is often tuned by selecting checkpoints with high preference margins. We show that this selection rule can fail. Across dense $\beta$ sweeps in three 7B open-weight families under controlled DPO recipes, the DPO preference margin can strongly anticorrelate with capability probes, with the strongest observed case in Llama logic probes (Pearson $r=-0.91$, $n=13$). In Mistral, matched-duration controls show that transient exposure to higher $\beta$ produces persistent arithmetic ($d_z = 3.73$, $p = 0.002$) and format ($d_z = 3.38$, $p = 0.003$) degradation relative to a duration-matched constant-$\beta$ run, while logic and sycophancy show no significant effect. An expanded 14-probe logic suite further shows that aggregate capability scores can hide stable opposition between probe clusters: direct-inference probes are consistently negative while fallacy-detection probes are consistently positive, with 13 of 14 probes sign-stable across 3 seeds and 3 $\beta$ values. Sensitivity profiles differ across architectures: Mistral shows elevated seed variance near $\beta \approx 10^{-2}$, Llama shows capability-specific rigidity, and Qwen trades off smoothly. These findings motivate probe-resolved $\beta$ sweeps rather than margin-based checkpoint selection.

AI developers regularly fine-tune language models to follow human preferences better. To decide when fine-tuning is done, they typically check a single internal score that measures how strongly the model prefers "good" answers over "bad" ones. But does that score actually track whether the model is becoming more capable? We tested three popular open-source language models across many training settings. Alongside the standard score, we measured how the models did on tasks like arithmetic, simple logic, formatting, and resistance to false statements. For one model, as the preference score went up, logic ability went down sharply. We also found that the order of training matters: briefly exposing a model to aggressive training and then easing off leaves lasting damage to math and formatting abilities, even when the model ends at the same intensity as one trained gently throughout. Choosing AI models by a single score can systematically pick worse ones. Practitioners should test capabilities directly across multiple categories instead of relying on one number, validate sensitive regions across multiple runs, and remember that training history can leave hidden damage.