Reward Learning through Ranking Mean Squared Error

Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems. A popular alternative is reward learning, where reward functions are inferred from human feedback rather than manually specified. Recent work has proposed learning reward functions from human ratings rather than traditional binary preferences, enabling richer and potentially less cognitively demanding supervision. Building on this paradigm, we introduce a new rating-based RL method, Ranked Return Regression for RL (R4). At its core, R4 uses a novel ranking mean squared error loss that learns from a dataset of trajectory–rating pairs, treating the human-provided discrete ratings (e.g., "bad," "neutral," "good") as ordinal targets. Unlike prior rating-based approaches, R4 offers formal guarantees: its solution set is provably minimal and complete under mild assumptions. Empirically, using both human-provided and simulated ratings, we demonstrate that R4 consistently matches or outperforms existing rating and preference-based RL methods on robotic benchmarks from OpenAI Gym and the DeepMind Control Suite.

Designing a good reward signal to train AI agents is harder than it sounds, and getting it wrong can lead to unintended or unsafe behaviors. Instead of hand-crafting these signals, we can ask humans to provide feedback — but most existing methods require people to repeatedly compare two robot behaviors and pick the better one, which is slow and can't capture whether either behavior was actually any good in an absolute sense. We developed a new method, R4, that instead asks humans to rate individual robot behaviors on a simple scale — like "bad," "okay," or "good" — and uses those ratings to automatically learn what the robot should be rewarded for. The key idea is that ratings naturally imply a ranking, and we train the AI using a clean mathematical objective that matches those rankings, with provable guarantees that no valid solution is missed or wrongly included. We tested R4 on a range of robotic movement tasks, using both computer-simulated feedback and real human participants, and found that it consistently taught robots to walk, run, and balance better than competing methods — even when different people rated the same behaviors very differently. Participants also found the rating task easy and low-stress. Our results are a step toward AI systems that can learn reliably from the kind of simple, natural feedback that any person can provide.