Distributed Direct Preference Optimization

Preference-based reinforcement learning (RL) is a key paradigm for aligning policies with human judgments, yet its theoretical behavior in distributed settings where preference data are fragmented across heterogeneous users remains poorly understood. Direct Preference Optimization (DPO) avoids explicit reward modeling but lacks convergence guarantees under federated and decentralized training, where communication constraints and non-IID preferences fundamentally alter optimization dynamics. We provide the first convergence and time-complexity analysis of DPO in distributed environments. Modeling personalized offline RL with user-specific preference distributions, we characterize the induced global optimization landscape. For federated DPO, we derive convergence rates that quantify the impact of client drift, communication frequency, and preference heterogeneity; for decentralized DPO, we establish convergence over general communication graphs and show how spectral connectivity governs optimization speed and consensus. Empirically, we corroborate our theoretical insights on standard alignment benchmarks, demonstrating that our proposed methods not only enjoy strong theoretical guarantees but also deliver robust and scalable performance in practice. The code base is available here.

Today’s AI assistants are often improved by learning from human feedback, e.g., by comparing which answers people prefer. Most current methods assume all of this feedback can be collected and trained in one central place. However, in reality, feedback data are usually spread across many users, companies, hospitals, or devices that cannot easily share their data because of privacy, security, or ownership concerns. This paper studies how AI systems can still learn effectively in these distributed settings without requiring all data to be centralized. We develop two approaches: one where many devices coordinate through a central server, and another where devices communicate directly with one another without any central controller. We also provide mathematical guarantees showing that these approaches remain stable and effective even when users have very different preferences or communication is limited. Experiments on real AI alignment datasets further confirm that the methods work well in practice.