Olaf-World: Orienting Latent Actions for Video World Modeling

Scaling action-controllable world models is limited by the scarcity of action labels. While latent action learning promises to extract control interfaces from unlabeled video, learned latents often fail to transfer across contexts: they entangle scene-specific cues and lack a shared coordinate system. This occurs because standard objectives operate only *within* each clip, providing no mechanism to align action semantics across contexts. Our key insight is that although actions are unobserved, their *semantic effects* are observable and can serve as a shared reference. We introduce **Seq$\Delta$-REPA**, a sequence-level control-effect alignment objective that anchors integrated latent action to temporal feature differences from a frozen, self-supervised video encoder. Building on this, we present **Olaf-World**, a pipeline that pretrains action-conditioned video world models from large-scale passive video. Extensive experiments demonstrate that our method learns a more structured latent action space, leading to stronger zero-shot action transfer and more data-efficient adaptation to new control interfaces than state-of-the-art baselines.

How can we build video world models that can be controlled without requiring large amounts of action-labeled data? In this paper, we address this by learning actions directly from ordinary videos. Our key idea is that even when the action itself is not recorded, its visual effect is still visible: the camera moves, objects shift, and the scene changes. We use these effects to make learned actions more consistent across different scenes and viewpoints. Based on this idea, we build Olaf-World, a controllable video world model that enables stronger zero-shot action transfer and faster adaptation to new control settings with minimal labeled data.