LiteVSR: Lightweight Adaptation of Frozen Diffusion Transformers for Video Super-Resolution

Adapting large-scale pre-trained video generators for Video Super-Resolution (VSR) in novel domains remains computationally prohibitive. Methods that reformulate generation as direct Low-Quality to High-Quality mappings deviate from the original generative formulation, demanding extensive fine-tuning. ControlNet-style adapters lose their efficiency under modern Diffusion Transformers since the absence of encoder-decoder hierarchy forces duplication of the entire backbone. We observe that flow matching offers a principled alternative for cross-domain VSR adaptation. By predicting a constant velocity field across all timesteps, the adaptation task reduces to learning a fixed injection pattern rather than time-varying transformations. Building on this insight, we propose LiteVSR, a minimalist framework that performs VSR using a completely frozen Diffusion Transformer with a lightweight State-Aware Adapter. The adapter employs a dual-stream architecture that extracts static structural cues from the LQ input and dynamic cues from intermediate denoising states, aligning them through time-dependent cross-attention to enable adaptive transition from structural alignment to texture refinement as denoising proceeds. LiteVSR achieves competitive restoration quality with only 11.25% trainable parameters and 12 GPU-hours of training on a single A100, while maintaining fast sampling (down to a single step) compatibility.

Restoring blurry or low-resolution videos has become possible with modern AI, but the best systems are enormous: adapting one to a new type of content typically requires retraining on millions of videos using dozens of expensive GPUs. This puts high-quality video restoration out of reach for most research groups. Instead of retraining a large pre-trained video AI, we keep it completely frozen and attach a small "guide" that points it toward the structural cues in the blurry input, letting it fill in realistic details on its own. Because the guide only has to learn this narrow bridging task, it can be trained on a few hundred clips using a single GPU in roughly twelve hours. The result matches the quality of systems trained with hundreds of times more compute, making custom video restoration practical for everyone.