The Differences Between Direct Alignment Algorithms are a Blur

Direct Alignment Algorithms (DAAs) simplify LLM alignment by directly optimizing policies, bypassing reward modeling and RL. While DAAs differ in their use of SFT (one-stage vs. two-stage) and the scalar score they optimize (likelihood vs. odds ratios), the key performance drivers remain underexplored. We present a systematic comparison and analyze a previously overlooked axis - the ranking objective (pairwise vs. pointwise). To isolate this factor, we propose a unified training framework across DAAs by (i) converting one-stage methods (ORPO, ASFT) into a two-stage pipeline with an explicit SFT phase and (ii) introducing a $\beta$ parameter that places all methods in the same hyperparameter space and improves the quality of odds-ratio DAAs (ORPO, ASFT). Under this setup, the ranking objective emerges as the primary determinant of alignment quality, whereas the particular scalar score (policy–reference ratio vs. odds ratio) is secondary. We corroborate this on instruction-following tasks and further confirm it on math-reasoning benchmarks across model scales. Evidence suggests that this stems from how these objectives interact with prompt-specific biases, supported both by strictly controlled experiments and by observations on real data. Our findings underscore the need for nuanced evaluations in DAA research to avoid oversimplified claims of superiority.

When developers improve AI assistants, they often show the model pairs of answers and mark which one people prefer. Many algorithms claim to use these comparisons better than others, but they are usually tested with different training recipes, so it is unclear which design choice actually matters. We compared popular direct alignment algorithms in a single, controlled framework. For methods that normally mix two steps, teaching the model to follow instructions and then using preferences, we separated those steps, and we gave all methods the same tuning knob. We then tested them on instruction-following and math-reasoning tasks across several language models. The main difference was not the exact score an algorithm uses, but how it learns from each pair of answers. Methods that directly compare the better answer with the worse one tended to work better than methods that score each answer separately, especially for tasks of medium difficulty. We also found that only a small amount of instruction training, about 5–10% of the data, was enough to get most of the benefit. This suggests that progress in AI alignment should be judged with careful, controlled comparisons rather than broad claims that one method is simply best.