MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. To solve this, this work explores a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model “imagines” the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Extensive experiments on the three datasets and platforms demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices. Project page is available at \url{https://mercerai.github.io/MVISTA-4D/}.

Robots need to predict how the world will change before deciding how to act. This paper introduces MVISTA-4D, a method that lets a robot imagine future scenes from multiple viewpoints using only one initial color-and-depth observation and a language instruction. By combining these imagined views, the robot obtains a more complete understanding of how objects move in 3D, especially when some parts are hidden. MVISTA-4D also turns imagined futures into actions. Instead of predicting actions one by one, it learns a compact plan for the whole motion and refines it at test time to better match the imagined scene. Experiments in simulation and on real robots show that this leads to more consistent future predictions and higher manipulation success rates than prior methods.