Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers

Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality–compute trade-off is largely constrained by the tokenizer’s fixed compression ratio. Variable-length tokenizers (VLTs) promise adaptive compression by varying token counts, allowing diffusion models to flexibly balance quality and compute. However, conventional VLTs modulate length by truncating ordered token sequences, which makes token semantics depend on token position and breaks representational alignment across lengths. This leads to a cross-length shift in the latent distribution that hinders a single variable-length diffusion model from operating effectively. To address this, we propose a novel variable-length tokenizer that modulates length by merging tokens. We show that encouraging similar tokens to merge enables direct cross-length representation alignment when the diffusion transformer operates according to the merging pattern. Since conventional merging methods are data-dependent, making the merging pattern inaccessible during generation, we introduce learnable global merging, which is data-independent, to ensure compatibility with diffusion transformers. On ImageNet 256$\times$256 generation, our merging-based variable-length tokenizer integrated with a diffusion transformer achieves a superior gFID–compute trade-off compared to prior VLT methods.

Many image generation systems rely on a tokenizer, which converts images into compact learned tokens that the generation model can process efficiently. The tokenizer’s compression level largely determines the efficiency of the generation model, but increasing compression usually comes at the cost of quality — for example, making the output look blurry. Variable-length tokenizers aim to provide more flexibility by letting a single system use fewer tokens when efficiency matters and more tokens when quality matters. However, the common approach of simply discarding tokens can make the short and long versions inconsistent for the generation model. To address this, the proposed method learns to merge related tokens into a smaller representation while keeping their relationships more consistent across different lengths. Experiments show that this approach helps preserve image quality while giving diffusion-based image generators more flexible control over computation. The result makes adjustable-cost image generation more practical, letting one model serve applications with very different speed and quality needs.