Mitigating Reward Hacking in LLM-based Recommendation: A Preference Optimization Approach

Post-training adaptation has become the central paradigm for leveraging large language models (LLMs) in recommendation. While recent preference optimization methods, such as Direct Preference Optimization (DPO), enhance pairwise preference discrimination, they remain vulnerable to \emph{reward hacking}: models exploit imperfections in reward signals, leading to inflated training metrics without genuine recommendation gains. We analyze this issue from a gradient perspective and formalize the concept of the \emph{$\varepsilon$-insensitive region}, where pairwise updates exert little influence on the ordering between positives and unsampled negatives. Under the Bradley–Terry model, we further show that these regions can occupy a substantial fraction of the preference space, inevitably leading to misaligned rankings. To address this issue, we propose Simulated Preference Optimization for Reward-hacking mitigation using Pseudo-negatives (SIRIUS). Our framework introduces pseudo-negative samples to enrich contrastive signals and reduce the prevalence of $\varepsilon$-insensitive regions. Extensive experiments on three public benchmarks show that \our{} consistently improves ranking quality and effectively mitigates reward hacking, providing both theoretical and practical insights for advancing LLM-based recommendation. Our code is available at \url{https://anonymous.4open.science/r/C557-id}

Large language models are increasingly used in recommendation systems to suggest music, movies, products, and other content. However, we found that these systems can sometimes improve their training objective without actually becoming better at making recommendations. In some cases, the model appears to be learning successfully, while the quality of its recommendations stops improving or even gets worse. To understand this problem, we studied how recommendation models learn from preference data and identified a common training behavior that can lead them toward the wrong objective. Based on this insight, we developed a new training method called SIRIUS. The key idea is to provide additional learning signals that help the model better distinguish relevant items from irrelevant ones, rather than focusing too narrowly on a small set of comparisons. Experiments on several recommendation benchmarks show that SIRIUS consistently improves recommendation quality and reduces the tendency of models to optimize the wrong objective. Our findings provide a better understanding of how large language model recommenders learn and offer a practical way to make future recommendation systems more reliable and effective.