TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance

Designing dense rewards is crucial for reinforcement learning (RL), yet in robotics it often demands extensive manual effort and lacks scalability. One promising solution is to view task progress as a dense reward signal, as it quantifies the degree to which actions advance the system toward task completion over time. We present TimeRewarder, a simple yet effective reward learning method that derives progress estimation signals from passive videos, including robot demonstrations and human videos, by modeling temporal distances between frame pairs. We then demonstrate how TimeRewarder can supply step-wise proxy rewards to guide reinforcement learning. In our comprehensive experiments on ten challenging Meta-World tasks, we show that TimeRewarder dramatically improves RL for sparse-reward tasks, achieving nearly perfect success in 9/10 tasks with only 200,000 interactions per task with the environment. This approach outperforms previous methods and even the manually designed environment dense reward on both the final success rate and sample efficiency. Moreover, we show that TimeRewarder pretraining can exploit real-world human videos, highlighting its potential as a scalable approach to rich reward signals from diverse video sources.

Teaching a robot is difficult when the only feedback it gets is whether it succeeded at the end of a task. A more useful signal would tell the robot along the way whether its actions are moving it closer to success. Unfortunately, designing this kind of step-by-step feedback by hand takes a lot of expert effort and must usually be repeated for every new task. We introduce TimeRewarder, a simple and effective way to create this feedback from videos. The main novelty of TimeRewarder is that it turns temporal distance in ordinary videos into a practical learning signal for robots. By learning this sense of progress, TimeRewarder can guide a robot during practice by judging whether the robot is getting closer to finishing the task. In our experiments on challenging robot manipulation tasks, this simple idea helps robots learn much faster and succeed more often than previous approaches. In many cases, it even performs better than carefully hand-designed feedback. TimeRewarder can also learn from videos of humans, not just robots, suggesting that everyday videos could become a practical source of guidance for teaching robots new skills.