Self-supervised Hierarchical Visual Reasoning with World Model

3D open-world environments with adversarial opponents remain a core challenge for reinforcement learning due to their vast state spaces. Effective reasoning representations are essential in such settings. While existing self-supervised visual foresight reasoning approaches often suffer from multi-step error accumulation, many recent studies resort to injecting domain-specific knowledge for more stable guidance. Our key insight is that the photorealistic fidelity of visual reasoning representations is secondary; what truly matters is providing informative, task-relevant signals. To this end, we propose ResDreamer, a hierarchical world model in which each higher-level layer is trained to reconstruct the residuals of the layer below. This design enables progressive abstraction of increasingly sophisticated world dynamics and fosters the emergence of richer latent representations. Drawing inspiration from the “Bitter Lesson,” ResDreamer trains its reasoning representations in a purely self-supervised manner. The higher-level residual representations are used to modulate lower-level predictions, allowing the world model to scale effectively with only linearly increasing cross-layer communication costs. Experiments show that ResDreamer achieves state-of-the-art sample efficiency and parameter efficiency. This scalable hierarchical visual foresight reasoning architecture paves the way for more capable online RL agents in open-ended, dynamic environments.

Training AI in complex 3D open worlds is a major challenge, especially when facing active, adversarial opponents. To succeed, an AI must anticipate the future, but current methods that try to visually "imagine" what happens next quickly accumulate errors. We built ResDreamer, a system that learns how the world works through a layered hierarchy. Instead of trying to predict a perfect picture, it focuses only on the most important, task-related changes. Crucially, information flows in both directions: lower levels send their prediction mistakes up, while higher levels send guidance back down to correct the lower-level predictions. This continuous two-way communication allows the AI to independently build a deep, stable understanding of its environment. This efficient design paves the way for highly capable AI agents that can adapt and thrive in unpredictable, fiercely competitive environments.