AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

Direct Preference Optimization (DPO) is an offline decision-making algorithm: a policy is trained entirely from a fixed preference dataset, with no online interaction. Recent theoretical analysis uncovers an asymmetric gradient pathology in DPO that misallocates this offline learning signal: the loss suppresses dispreferred responses substantially faster than it promotes preferred ones, so the model predominantly learns to avoid bad answers rather than to generate good ones. We propose AdaDPO, a Self-Adaptive variant of the Direct Preference Optimization algorithm that introduces per-preference-pair, stop-gradient-based coefficients derived directly from the policy model's generation probabilities. AdaDPO is constructed to balance gradient magnitudes between preferred and dispreferred probabilities; the practical implementation balances per-token gradients and applies a numerical clipping bound for stability, while retaining DPO's original hyperparameter structure. In preliminary experiments on Llama-3-8B-Instruct trained on UltraFeedback, AdaDPO consistently outperforms DPO on AlpacaEval 2: it achieves higher length-controlled win rates (LC) in 81% of hyperparameter combinations and enlarges the LC-over-WR margin in 88% of combinations, indicating effective mitigation of length bias. Because it operates purely at the loss level, AdaDPO is a drop-in correction for preference-based alignment pipelines.