Structured 4D Latent Predictive Model for Robot Planning

Video predictive models are emerging as a powerful paradigm in robotics, offering a promising path toward task generalization, long-horizon planning, and flexible decision-making. However, prevailing approaches often operate on 2D video sequences, inherently lacking the 3D geometric understanding necessary for precise spatial reasoning and physical consistency. We introduce a Structured 4D Latent Predictive Model, which predicts the evolution of a scene’s 3D structure in a structured latent space conditioned on observations and textual instructions. Our representation encodes the scene holistically and can be decoded into diverse 3D formats, enabling a more complete and 3D consistent scene understanding. This structured 4D latent predictive model serves as a planner, generating future scenes that are translated into executable actions by a goal-conditioned inverse dynamics module. Experiments demonstrate that our model generates futures with strong visual quality, substantially better 3D consistency and multi-view coherence compared to state-of-the-art video-based planners. Consequently, our full planning pipeline achieves superior performance on complex manipulation tasks, exhibits robust generalization to novel visual conditions, and proves effective on real-world robotic platforms. Our website is available at https://structured-4d-model.github.io/.

Robots that operate in the physical world need a kind of imagination: before they move, they should be able to predict how the world around them will change. Recent methods often do this by predicting the future as videos, but videos are tied to particular camera views and can miss the 3D structure that matters for real-world action. We introduce a model that merges images from different camera views to recover 3D scene information, then predicts how the scene will evolve over time. The robot can follow these predicted 3D futures to decide what actions to take. This is especially important for tasks such as stacking objects or inserting one object into another, where a small error in 3D position or alignment can cause the action to fail. By predicting the future in 3D rather than only through 2D videos, our method gives robots a more reliable basis for planning. Our long-term goal is to build a general 3D-aware model that helps robots predict future physical changes and use those predictions to act more effectively in the real world.