$f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone technique for post-training large language models. While most existing approaches rely on the reverse KL-regularization, recent empirical studies have begun exploring alternative divergences (e.g., forward KL, chi-squared) as regularizers in RLHF. However, a unified theoretical understanding of general $f$-divergence regularization remains under-explored. To fill this gap, this work develops a comprehensive theoretical framework for online RLHF with an $f$-divergence regularized objective. Rather than treating each divergence in isolation, we adopt a holistic perspective across the entire class and propose two algorithms based on distinct sampling principles. The first extends the classical optimism principle with a carefully designed exploration bonus, while the second introduces a new method that exploits the sensitivity of the optimal policy to reward perturbations under $f$-divergence regularization. Theoretical analysis shows that $O(\log T)$ regret and $O(1/T)$ sub-optimality gap are achievable, establishing provable efficiency of both algorithms and, to the best of our knowledge, the first performance bounds for online RLHF under general $f$-divergence regularization.

Large language models, such as chatbots, can be improved by learning from people’s preferences between different answers. While improving the model, we also want to prevent it from moving too far away from its original behavior in unexpected or harmful ways. Most existing theory studies the common way to control this change, while recent practical methods have started using other ways that may be more stable or robust. This paper develops a general framework for understanding many such control methods, instead of analyzing each one separately. It proposes two strategies for deciding what feedback the model should collect while it is learning: one focuses on places where the model is uncertain, and the other uses how sensitive the learning process is to small changes as a guide. The paper proves that both strategies can learn efficiently. Overall, this work helps explain how different ways of controlling human-feedback training work, and provides researchers with principled tools for designing more reliable AI systems.