VideoMAETok: Boosting Video Diffusion Models via Masked Autoencoders as Tokenizers

Latent diffusion models have become the dominant paradigm for video generation, making the video tokenizer a critical role. While most existing tokenizers are trained primarily for reconstruction, diffusion models are optimized to denoise heavily corrupted latents, which creates a mismatch between tokenizer training objectives and downstream generative learning. As a result, reconstruction metrics (e.g., rFVD) can be a poor proxy for generation quality (gFVD), and overly prioritizing reconstruction may even hinder diffusion training. We propose VideoMAETok, a simple family of ViT-based video tokenizers trained explicitly as corruption-inversion models for latent video diffusion. VideoMAETok builds on masked autoencoders: we (i) apply high-ratio token masking and encode only visible spatiotemporal tokens for efficiency, and (ii) corrupt latent tokens with interpolative Gaussian noise to better match the denoising nature of diffusion generators. Training under such corruption encourages latents that remain informative and well-conditioned for downstream denoising. Extensive experiments show that VideoMAETok consistently improves generation quality when paired with off-the-shelf diffusion models (SiT and LightningDiT), achieving state-of-the-art gFVD on Kinetics-600 and UCF-101 while remaining compute-efficient. Code is available at https://github.com/yztongzhan/VideoMAETok.

Many AI systems for generating videos first compress each video into latent representation, and then learn to create new videos from this compressed information. A common assumption is that the best compression method is the one that can most accurately reproduce the original video. We show that this assumption can be misleading: a system that copies videos very well may still provide poor foundations for generating new ones. In this work, we introduce VideoMAETok, a method that learns video representations by reconstructing videos from incomplete and deliberately disturbed information. This encourages the model to capture meaningful motion and visual structure, rather than relying on details. Our results suggest that building better video-generation systems requires more than improving video reconstruction alone. This work may support more efficient and higher-quality video creation.