VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling, e.g., conditional video generation. Beyond compression, tokenizers define what information is preserved and how it is organized. A de facto standard approach is to represent a video with a spatiotemporal 3D grid of tokens, each corresponding to a local patch in the original signal. This requires a downstream model, e.g., a text-to-video model, to learn to predict all low-level details ``pixel-by-pixel'' irrespective of the video's inherent complexity, resulting in high computational cost during training. We present VideoFlexTok, a tokenizer that represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens capture abstract information like semantics and motion and later tokens provide fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the tokens count to particular downstream needs and encode videos longer than the 3D grid approach under the same budget. We evaluate VideoFlexTok on class-to-video and text-to-video generative tasks and show that it leads to more efficient training compared to 3D grid tokens, e.g., achieving comparable generation quality (gFVD and ViCLIP Score) with a 10x smaller model (0.4B vs 3.6B). Finally, we demonstrate how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.