RealisMotion: Decomposed Human Motion Control and Video Generation in the World Space

Generating human videos with realistic and controllable motions is a challenging task. While existing methods can generate visually compelling videos, they lack separate control over four key video elements: foreground subject, background video, human trajectory, and action patterns. In this paper, we propose a decomposed human motion control and video generation framework that explicitly decouples motion from appearance, subject from background, and action from trajectory, enabling flexible mix-and-match composition of these elements. Concretely, we first build a ground-aware 3D world coordinate system and perform motion editing directly in the 3D space. Trajectory control is implemented by unprojecting edited 2D trajectories into 3D with focal-length calibration and coordinate transformation, followed by speed alignment and orientation adjustment; actions are supplied by a motion bank or generated via text-to-motion methods. Then, based on modern text-to-video diffusion transformer models, we inject the subject as tokens for full attention, concatenate the background along the channel dimension, and add motion (trajectory and action) control signals by addition. Such a design opens up the possibility for us to generate realistic videos of anyone doing anything anywhere. Extensive experiments on benchmark datasets and real-world cases demonstrate that our method achieves state-of-the-art performance on both element-wise controllability and overall video quality.

Creating realistic videos of people is now possible, but giving users precise control over the result remains difficult. Existing methods often cannot separately control who appears in the video, what background they are placed in, where they move, and what action they perform. In this paper, we present a method that separates these four parts so they can be edited independently. A user can choose a person from one image, place them into a different scene, define the path they should follow, and specify the action they should perform. This makes video creation much more flexible. Our method first builds a simple 3D understanding of the scene and motion, which helps keep the person’s movement natural and consistent with the environment. It then uses a video generation model to produce the final video. Experiments show that our method gives better control over video content while also improving visual quality. This can be useful for applications such as animation, filmmaking, and digital content creation.