Hierarchical Image Tokenization for Multi-Scale Image Super Resolution

We introduce a multi-scale Image Super Resolution (ISR) method building on recent advances in Visual Auto-Regressive (VAR) modeling. VAR models break image tokenization into additive, gradually increasing scales, using Residual Quantization (RQ), an approach that aligns perfectly with our target ISR task. Previous works taking advantage of this synergy suffer from two main shortcomings. First, due to the limitations in RQ, they only generate images at a predefined fixed scale, failing to map intermediate outputs to the corresponding image scales. They also rely on large backbones or a large corpus of annotated data to achieve better performance. To address both shortcomings, we introduce two novel components to the VAR training for ISR, aiming at increasing its flexibility and reducing its complexity. In particular, we introduce a) a Hierarchical Image Tokenization (HIT) approach that progressively represents images at different scales while enforcing token overlap across scales, and b) a Direct Preference Optimization (DPO) regularization term that, relying solely on the (LR,HR) pair, encourages the transformer to produce the latter over the former. Our proposed HIT acts as a strong inductive bias for the VAR training, resulting in a small model (300M params vs 1B params of VARSR), that achieves state-of-the-art results without external training data, and that delivers multi-scale outputs with a single forward pass.

Current AI models for image super resolution are often rigid and can only magnify images to a single, fixed scale. To achieve high-quality results, these models also usually rely on massive amounts of computing power and complex datasets. We introduce a new method coined Hierarchical Image Tokenization (HIT), combined with a regularization term. Instead of producing directly a fixed final size, HIT trains the model to progressively build the image through multiple intermediate scales. The regularization term acts as a guide, teaching the model to separate between image enhancing and simple bilinear upsampling. Our proposed method is a strong inductive bias that helps the training of small models to produce state of the art results. This approach is a powerful guide for the learning process. As a result, we achieve state of the art results in image enhancement using a smaller, more efficient model, without needing external data annotated with "human preference". It flexibly delivers high-quality images at multiple sizes in just one single pass.