Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL (PbRL) offers a promising alternative by learning reward functions from human feedback, but its scalability is hindered by high labeling costs. Inspired by advances in Video Foundation Models (ViFMs), we present Video-based Optimal Transport Preference (VOTP), a semi-supervised framework that learns effective reward functions from only a handful of labels. By leveraging optimal transport to align visual trajectories within the rich representation space of ViFMs, VOTP effectively generates high-fidelity pseudo-labels for large amounts of unlabeled data, substantially reducing human supervision. Extensive experiments across locomotion and manipulation benchmarks demonstrate the superiority of VOTP, which outperforms state-of-the-art offline PbRL methods under limited feedback budgets. We also showcase the robustness of VOTP in the presence of visual distractors and validate its utility on real robotic tasks, where it learns meaningful rewards with minimal human input.

Training robots usually requires designing a reward function by hand. A reward function is like a scoring rule: it tells the robot which behaviors are good and which are bad. However, creating these rules can take a lot of time, expertise, and trial-and-error, especially for complex real-world tasks. To avoid manually writing reward functions, researchers have explored teaching robots directly from human feedback. For example, a person might watch two robot videos and choose which behavior looks better. While this approach works well, collecting enough human feedback is still expensive and difficult to scale. To address this problem, we developed a system that acts like a smart automatic evaluator. With only a small number of human-labeled examples (for instance, around 10 comparisons), our method can automatically score large collections of unlabeled robot videos. It does this by analyzing how visually similar the robot’s behavior is to successful expert demonstrations using modern video-understanding AI. This dramatically reduces the amount of human effort needed to train robots. By learning human preferences from only a handful of examples, our approach makes robot training faster, cheaper, and more practical for real-world applications.