Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

Standard RLHF relies on transitive scalar rewards, failing to capture the cyclic nature of human preferences. While some approaches like the General Preference Model (GPM) address this, we identify a theoretical limitation: their implicit formulation entangles hierarchy with cyclicity, failing to guarantee dominant solutions. To address this, we propose the Hybrid Reward-Cyclic (HRC) model, which utilizes game-theoretic decomposition to explicitly disentangle preferences into orthogonal transitive (scalar) and cyclic (vector) components. Complementing this, we introduce Dynamic Self-Play Preference Optimization (DSPPO), which treats alignment as a time-varying game to progressively guide the policy toward the Nash equilibrium. Synthetic data experiments further validate HRC's structural superiority in mixed transitive--cyclic settings, where HRC converges faster and achieves higher accuracy than GPM. Experiments on RewardBench 2 demonstrate that HRC consistently improves over both BT and GPM baselines (e.g., +1.23\% on Gemma-2B-it). In particular, its superior performance in the Ties domain empirically validates the model's robustness in handling complex, non-strict preferences. Extensive downstream evaluations on AlpacaEval 2.0, Arena-Hard-v0.1, and MT-Bench confirm the efficacy of our framework. Notably, when using Gemma-2B-it as the base preference model, HRC+DSPPO achieves a peak length-controlled win-rate of 44.75\% on AlpacaEval 2.0 and 46.8\% on Arena-Hard-v0.1, significantly outperforming SPPO baselines trained with BT or GPM. Our code is publicly available at https://github.com/lab-klc/Hybrid-Reward-Cyclic.

To make AI systems safe and useful, we need them to understand what humans prefer. Today's dominant approach gives the AI a single "score" for each possible answer, much like grading on a straight line from worst to best. But real human preferences are often more complicated — think of rock-paper-scissors: rock beats scissors, scissors beats paper, yet paper beats rock. A simple score cannot capture this kind of circular, contradictory preference. We developed a new way to model human preferences called the Hybrid Reward-Cyclic (HRC) model. Instead of forcing all preferences onto a straight line, HRC separates them into two components: one that captures consistent, ranked preferences and another that captures the circular preferences. We also designed a training method called Dynamic Self-Play Preference Optimization (DSPPO), which teaches the AI in stages — first mastering the clear, universal preferences before gradually learning the more subtle, subjective ones. This staged approach leads to more stable and reliable learning. Our experiments show that models trained with HRC and DSPPO consistently outperform older methods across a range of challenging benchmarks. In particular, HRC excels at handling ambiguous or conflicting preferences where traditional methods struggle. This work takes a step toward AI systems that better reflect the richness and complexity of human judgment.