Learning Transferable Interaction Primitives from Game Videos for Humanoid Locomotion

Learning humanoid control from video provides a scalable alternative to the scarcity of high-fidelity robot data. Existing methods, however, often rely on curated datasets and treat video as passive kinematic priors. They fail to capture dynamic humanoid interactions with the environment, which are essential for robust control in complex physical environments. To address this, we propose **TRansferable Interaction Primitives (TRIP), a framework designed to extract and ground interactions from unlabeled game videos for locomotion control. TRIP explicitly models dependencies between motion dynamics and environmental context via a discrete library of interaction-based action primitives. To bridge the reality gap, we introduce a shared context latent space that aligns implicit video-domain features with functional target-domain observations, enabling the seamless transfer of video-mined strategies to reinforcement learning policies. Our experiments on complex terrain navigation demonstrate that TRIP achieves significant improvements in task performance, sample efficiency, and robustness.

Humanoid robots need to navigate complex real-world terrain—stairs, slopes, uneven ground—but collecting the data to teach them is expensive and time-consuming. Existing methods learn from video by copying human poses frame by frame, without understanding why a person moves differently on different terrain. We asked: can a robot learn not just how humans move, but how humans respond to their surroundings? Our system watches videos of people traversing diverse landscapes and automatically extracts a library of reusable movement strategies—each capturing how the body adapts to a specific terrain condition. A simulated humanoid robot then learns to read its environment and select the right strategy on the fly. The result is a robot that moves more robustly and naturally across challenging and unseen environments. This opens a scalable path to teaching robots complex physical skills through videos.