Normalized Rewards for Preference Optimization

Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences. However, DAAs have been observed to over-optimize their implicit reward model and decrease the likelihood of preferred responses. This results in a decrease in the total likelihood assigned to responses seen in the preference dataset, potentially resulting in undesirable behavior. To counteract this undesired side-effect of DAAs, we examine the effect of using objectives that add a regularization term to maintain the total length-normalized probabilities of the chosen and rejected responses. To better understand over-optimization, we investigate how response likelihood changes are distributed over the tokens with and without regularization. We find that a significant portion of the likelihood changes are due to a small set of outlier tokens, which explains how DAAs improve generation quality despite decreasing the likelihoods of chosen responses. We apply the proposed regularization to reference-based (DPO) and reference-free (SimPO) methods and find (1) improved trade-offs between generation quality and general benchmark capability and (2) improvements in reward modeling across datasets. For example, on Llama-3.1-8B-Instruct, we see both a >20\% relative increase in AlpacaEval2 scores and >9\% relative performance gains on general benchmarks. Additionally, we find that the added regularization term effectively mitigates the amount of displacement within preferred responses overall, and for the outlier tokens specifically, by utilizing low-likelihood tokens.

Language models learn to interact and communicate with humans in ways that are aligned with human values through feedback on the types of responses people prefer. One issue is that the models can learn from the feedback in a way that does not generalize. We hypothesized that this is due to the model focusing only on making sure it is more likely to respond with response A compared to response B if A is preferred over B without considering factors such as how often A or B should appear as a response in the first place. We proposed a method that helps the model account for these factors and found both through theory and experiments that this helps the model generalize the preference feedback. Our findings have implications for ensuring models can behave in a safe and beneficial way more consistently and can help build an understanding of model behavior.