The Hidden Link between RLHF and Contrastive Learning

Alignment of large language models (LLMs) with human values has recently garnered significant attention, with prominent examples including the canonical yet costly Reinforcement Learning from Human Feedback (RLHF) and the simple Direct Preference Optimization (DPO). In this work, we demonstrate that both RLHF and DPO can be interpreted from the perspective of mutual information (MI) maximization, uncovering a profound connection to contrastive learning. Within this framework, both RLHF and DPO can be interpreted as methods that performing contrastive learning based on the positive and negative samples derived from base model, leveraging the Donsker–Varadhan (DV) lower bound on MI (equivalently, the MINE estimator). Such paradigm further illuminates why RLHF may not intrinsically incentivize reasoning capacities in LLMs beyond what is already present in the base model. Building on the perspective, we replace the DV/MINE bound with the Jensen–Shannon (JS) MI estimator and propose the Mutual Information Optimization (MIO). Comprehensive theoretical analysis and extensive empirical evaluations demonstrate that MIO mitigates the late-stage decline in chosen-likelihood observed in DPO, achieving competitive or superior performance across various challenging reasoning and mathematical benchmarks

Large language models are often tuned so that their answers better match human preferences. Popular methods such as Reinforcement Learning from Human Feedback and Direct Preference Optimization can be effective, but they can also be costly, unstable, and may fail to improve reasoning when the original model rarely produces the right answer. This paper explains these issues from a new perspective: these alignment methods can be understood as a form of contrastive learning, where the model learns by comparing preferred and rejected responses. The paper shows that the usual training objective can give weak or misleading learning signals when some answers have very low probability. To address this, the authors propose Mutual Information Optimization (MIO), a more stable preference-optimization method. Experiments on mathematical, reasoning, and broader alignment benchmarks show that MIO can improve training stability and often outperform existing methods, suggesting a practical way to better align language models while preserving their reasoning ability.