Beyond Static Allocation: Dynamic Sensitivity-Aware Fine-Tuning for Vision Transformers

Existing Parameter-Efficient Fine-Tuning (PEFT) methods are fundamentally constrained by a static allocation paradigm, which overlooks the model's evolving optimization priorities during training. To address this, we introduce Dynamic Adaptive Fine-tuning (DAF), a novel framework that periodically evaluates and reconfigures the trainable structure based on a context-aware decoupled sensitivity analysis. DAF employs a Rebuild-and-Refocus strategy to preserve learned knowledge by freezing outdated modules while decisively reallocating the parameter budget to newly identified critical regions. Extensive experiments on challenging vision benchmarks demonstrate that DAF significantly outperforms mainstream static PEFT methods and achieves superior performance and efficiency, particularly under extreme parameter budgets. Our work fundamentally challenges the static nature of the field, offering a more intelligent and efficient paradigm for adapting large pretrained models. The code is available at https://github.com/E-green11/DAF.

Adapting large vision models to new tasks usually requires either fine-tuning many parameters or choosing a fixed small set of trainable modules in advance. However, a model’s learning needs can change during training: the parts that are important at the beginning may not remain the most useful later. This paper introduces Dynamic Adaptive Fine-tuning (DAF), a method that periodically checks which parts of a vision model are currently most useful for adaptation and reallocates the limited trainable budget to those parts. DAF keeps previously learned knowledge by freezing older useful modules, while focusing new training effort on newly emerging bottlenecks. This dynamic strategy allows the model to adapt more flexibly than static parameter-efficient fine-tuning methods. Experiments on a wide range of visual recognition tasks show that DAF improves accuracy while using only a small fraction of trainable parameters. The method can help make large vision models easier and cheaper to adapt to new applications without adding extra cost during inference.