From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning

Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.

Agents such as robots usually need a reward function that tells them what behavior is good. However, in many real-world tasks, designing such rewards is difficult, and collecting large amounts of human preference feedback is expensive. This makes preference-based reinforcement learning challenging when only a small number of preference labels are available. To address this, we propose FB-PbRL. Rather than first learning a reward model from preferences, our method first learns reward-free representations from offline data. These representations capture useful information about how the environment works, providing the agent with a compact map of possible behaviors. We then use preference feedback to align this representation space with the behaviors humans prefer, allowing the agent to turn limited comparisons between behaviors into an effective policy. Across locomotion, navigation, and robotic manipulation tasks, FB-PbRL consistently learns stronger policies than existing methods. It also remains effective when the number of preference labels is greatly reduced, showing that reward-free representations can make human feedback more useful. This work bridges reward-free representation learning with human preferences, offering a new way to rethink offline preference-based reinforcement learning.