PAWS: Preference Learning with Advantage-Weighted Segments

Preference-based reinforcement learning (PbRL) learns policies from human trajectory-level comparisons, avoiding explicit reward design and expert demonstrations. Existing methods typically train utility functions on trajectory or segment-level preferences while relying on per-step utility estimates during policy optimization. This training and inference mismatch induces a distribution shift that severely degrades temporal credit assignment and limits policy learning. We analyze this issue and propose PAWS, a segment-based preference learning method that performs policy updates directly using segment-level advantage functions. By aligning utility training with policy optimization, PAWS preserves trajectory-level preference information and avoids unreliable per-step learning signals. Experiments on simulated robotic manipulation and locomotion tasks demonstrate that PAWS consistently outperforms existing PbRL approaches, highlighting the importance of distribution-consistent preference learning.

Robots can learn complex skills like opening a door or inserting a peg, but mathematically describing what counts as "good" behavior is hard. One promising approach lets a person simply watch two attempts and say which one they prefer, sparing them from hand-designing detailed reward criteria. A person judges a whole sequence of movements at once, but existing methods usually learn from individual steps, even though no one ever said which individual steps were good, only that the whole sequence was better. This guesswork is unreliable and often steers learning in the wrong direction. We developed PAWS, a method that keeps the robot's learning consistent from start to finish. Instead of forcing the robot to break a preferred sequence into separate moments, PAWS learns from and acts on entire sequences, exactly as the person judged them. It also offers a more intuitive dial for controlling how boldly the robot updates its behavior. Across many simulated robot tasks, with feedback from both automated judges and real people, PAWS learned better skills than existing methods.