VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing local information from the original signal. This requires the downstream model, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video's inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens (emergently) capture information such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count to downstream needs and encoding videos longer than the baselines within the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it yields more efficient training than 3D grid tokens, achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we show how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Videos are expensive for AI systems to process because their raw form contains many pixels across both spatial and temporal dimensions. To make this manageable, most modern AI systems first convert videos into “tokens”: compact building blocks that a model can work with. However, common video tokenizers arrange these tokens into a fixed-size 3D grid, so every video receives the same kind of representation regardless of its inherent complexity. This work introduces VideoFlexTok. Instead of a fixed grid, it produces a flexible-length, coarse-to-fine representation in which early tokens capture important semantic content and motion, while later tokens progressively add more fine-grained detail. This allows a model to use fewer tokens when a coarse representation is enough, and more tokens when fine detail matters. In our experiments, VideoFlexTok represents videos with up to 256 times fewer tokens than a standard 3D grid tokenizer and achieves similar performance on common generative tasks with 5 to 10 times less computational resources. By lowering the computational cost of video modeling, VideoFlexTok can make video research more accessible to a broader and more diverse group of researchers and could enable long-horizon video understanding, generation, and reasoning.