Autoregressive Direct Preference Optimization

Direct preference optimization (DPO) has emerged as a promising approach for aligning large language models (LLMs) with human preferences. However, the widespread reliance on the response-level Bradley-Terry (BT) model may limit its full potential, as the reference and learnable models are assumed to be autoregressive only after deriving the objective function. Motivated by this limitation, we revisit the theoretical foundations of DPO and propose a novel formulation that explicitly introduces the autoregressive assumption prior to applying the BT model. By reformulating and extending DPO, we derive a novel variant, termed Autoregressive DPO (ADPO), that explicitly integrates autoregressive modeling into the preference optimization framework. Without violating the theoretical foundations, the derived loss takes an elegant form: it shifts the summation operation in the DPO objective outside the log-sigmoid function. Furthermore, through theoretical analysis of ADPO, we show that there exist two length measures to be considered when designing DPO-based algorithms: the token length $\mu$ and the feedback length $\mu'$. To the best of our knowledge, we are the first to explicitly distinguish these two measures and analyze their implications for preference optimization in LLMs.

Large language models (LLMs) are increasingly trained to match human preferences, for example, to give helpful, harmless, and honest answers. A key technique for this is Direct Preference Optimization (DPO), which teaches the model to prefer "good" responses over "bad" ones by comparing pairs of examples. However, DPO has a subtle internal inconsistency: it treats each response as a single whole when deciding which is better, even though LLMs actually generate text one word (token) at a time, left to right. We fixed this inconsistency by introducing Autoregressive DPO (ADPO), which applies the preference comparison at each step of generation rather than only at the end. This leads to a cleaner mathematical formulation and reveals a previously unrecognized distinction between two notions of length: the length of a response as measured by the model versus as measured by human feedback. Experiments across mathematical reasoning and conversational tasks show that ADPO consistently outperforms standard DPO, offering a theoretically sounder and practically stronger approach for aligning LLMs with human preferences.