DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

Multimodal LLMs struggle to systematically model the temporal evolution of visual scenes in videos or multi-image sequences. Such inputs require models to predict or simulate multiple levels of dynamic constituents, such as actions taken in the visual sequence, and the associated changes to the visual environment that result. To address this challenge, we propose a dynamic schema-guided world model, DynaVieW, optimized for visual dynamic prediction and simulation. DynaVieW achieves an in-depth understanding of visual dynamics by learning interleaved state-transition sequences, where states cover broad visual scenes from video keyframes, and transitions capture comprehensive dynamic constituents within a hierarchical schema. DynaVieW jointly models transition prediction and state simulation under a mixture-of-experts architecture, with a cross-expert selective attention and a schema token re-weighted loss, to ensure effective and robust learning. DynaVieW's superior visual dynamic understanding boosts its downstream performances on both visual narrative creation and world simulation, showing improved consistency and controllability of visual generation and better instruction-following ability.

Today’s AI models often struggle to grasp how human activities unfold in the real world. They cannot reliably predict how events or actions in a real-world scene will progress over time, and how small details in the scene will change as those events or actions happen. This limits their usefulness for tasks that require understanding dynamics (such as cause and effect) in the visual world, or generating consistent visual sequences (such as videos or storyboards). To address this problem, we introduce DynaVieW, a new AI model designed to better understand and predict visual dynamics in human world activities. DynaVieW learns broad visual world states, showing what a scene of activity looks like at a sequence of key moments, and meanwhile transitions between the states, explaining how the scene shifts between the key moments. It organizes the transitions using a multi-level data structure, which captures both high-level progress of the activity and low-level changes of visual details. Our model also combines two specialized sub-models, who pay attention to each other and collaborate to predict transitions and simulate future states. DynaVieW develops a deeper understanding of why the visual world looks like the way it does and how it will evolve over time. This results in our model’s better performances on the tasks of creating coherent visual stories and following instructions to simulate real-world interactions. By improving how AI models reason about dynamics in the human activities, our work pushes forward AI systems to better understand and interact with the complex visual world around us.