Beyond Pixel Histories: World Models with Persistent 3D State

Interactive world models continually generate video by responding to a user's actions, enabling open-ended generation capabilities. However, existing models typically lack a 3D representation of the environment, meaning 3D consistency must be implicitly learned from data, and spatial memory is restricted to limited temporal context windows. This results in an unrealistic user experience and presents significant obstacles to downstream tasks such as training agents. To address this, we present PERSIST, a new paradigm of world model which simulates the evolution of a latent 3D scene: environment, camera, and renderer. This allows us to synthesise new frames with persistent spatial memory and consistent geometry. Both quantitative metrics and a qualitative user study show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods, enabling coherent, evolving 3D worlds. We further demonstrate novel capabilities, including synthesising diverse 3D environments from a single image, as well as enabling fine-grained, geometry-aware control over generated experiences by supporting environment editing and specification directly in 3D space.

Imagine exploring a video game or virtual world generated entirely by AI in real time. Current AI models try to do this by only remembering the last few seconds of video shown on your screen. Because of this short-term memory, they quickly forget the layout of the space, leading to glitchy, shifting environments the moment you turn around or revisit an old area. To fix this, we created PERSIST, an AI system that tracks the world in 3D. Instead of just relying on a history of flat 2D images, our system builds and continuously updates a persistent 3D map of the environment around the user. When you move or take an action, the AI uses this stable 3D memory as a reference to accurately draw what you should see next. This approach dramatically improves the realism and long-term stability of generated worlds, keeping them physically coherent over extended periods. Beyond creating more immersive virtual experiences, this allows users to easily edit 3D environments on the fly and provides highly reliable simulators for safely training future technologies.