Alignment-Aware Decoding

Alignment of large language models remains a central challenge in natural language processing. Preference optimization has emerged as a popular and effective method for improving alignment, typically through training-time or prompt-based interventions. In this paper, we introduce alignment-aware decoding (AAD), a method to enhance model alignment directly at inference. Theoretically, AAD can be interpreted as implicit reward optimization, yet it requires no specialized training beyond the standard DPO setup. Empirically, AAD consistently outperforms strong baselines across diverse alignment benchmarks and model scales. Moreover, in data-constrained settings, AAD can produce high-quality synthetic data to improve alignment under standard decoding, providing a practical solution when labeled data is limited.

Modern AI assistants like ChatGPT need to learn what people find helpful and avoid what people find harmful. The standard approach trains them on pairs of preferred and rejected responses, but this preference data is expensive and slow to collect from humans. Even after such training, AI models often fall back on habits learned earlier and don't fully use what they learned from preferences. We introduce alignment-aware decoding (AAD), a simple method that runs two copies of the AI side by side while it writes a response. At every word, we look at where the preference-trained copy and its earlier version disagree most strongly, and we pick the words the preference-trained copy favors. AAD listens to the "echo" of what the model learned about preferences, instead of letting old habits dominate. Across many benchmarks, AAD produces responses that humans and automated judges find better aligned than existing methods, without any extra training. AAD can also generate higher-quality training examples when human-labeled data is scarce, making alignment cheaper and easier.