DynaMem: Consistent Long Video Generation via Hierarchical Memory and Motion Priors

Recent text-to-video diffusion models can synthesize visually compelling clips from natural language prompts. However, practical applications increasingly demand long-form videos with evolving narratives and persistent identity. A common solution is autoregressive generation, where the video is produced clip by clip over long horizons, yet coherence often degrades as errors compound. In this work, we study long-video generation under an autoregressive setting, where videos are synthesized clip by clip over long horizons. Despite strong short-clip quality, existing approaches often suffer from semantic drift, motion decay, and appearance instability as the sequence grows. We present DynaMem, a unified framework that improves long-horizon coherence via three components: Semantic-Adaptive Hierarchical Memory for long-range semantic preservation, Motion-Prioritized Optimization for motion-coherent learning, and Reference-Anchored Perceptual Alignment for stabilizing appearance. Extensive experiments show that DynaMem produces more consistent semantics, stronger temporal dynamics, and more stable appearance on long videos compared to competitive baselines.

As the AI-generated video gets longer, small errors build up, causing characters to change, movements to freeze, and colors to shift. Current AI video generators work well for short clips but struggle to maintain consistency over longer sequences because they can only "see" a few recent frames, losing track of what happened earlier. We developed DynaMem, a system that addresses these problems through three coordinated strategies. First, it uses a smart memory system that stores and retrieves key moments from the video's history, much like how a film editor keeps reference shots to maintain continuity. Second, it trains the AI to prioritize learning realistic motion rather than just copying still frames, preventing the common problem of videos becoming increasingly static. Third, it anchors the color and lighting of new frames to early reference frames, stopping the gradual visual drift that makes long videos look inconsistent. In experiments, DynaMem produced longer videos with more stable characters, smoother motion, and more consistent visual quality compared to existing approaches. This work brings video generation closer to producing the kind of extended, coherent video content needed for filmmaking, education, and entertainment.