Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing

Adapting pre-trained vision models using parameter-efficient fine-tuning (PEFT) remains challenging, as it aims to achieve performance comparable to full fine-tuning using a minimal number of trainable parameters. When applied to complex dense prediction tasks, existing methods exhibit limitations, including input-agnostic modeling and redundant cross-layer representations. To this end, we propose ParaX, a new adapter-style method featuring a simple mixture-of-experts (MoE) architecture. Specifically, we introduce shared expert centers, where each expert is a trainable parameter matrix. During a feedforward pass, each ParaX module in the network dynamically generates weight matrices tailored for the current module via a simple dynamic parameter routing mechanism, which selectively aggregates parameter matrices in the corresponding expert center. Dynamic weight matrices in ParaX modules facilitate low-rank adaptation in an input-dependent manner, thus generating more customized and powerful feature representations. Moreover, since ParaX modules across multiple network layers share the same expert center, they improve feature diversity by promoting implicit cross-layer feature interaction. Extensive experimental results demonstrate the superiority of ParaX across diverse visual recognition tasks. Code is publicly released at: https://github.com/LMMMEng/ParaX.

Pre-trained vision models have significantly driven advances in visual recognition, but adapting them to new tasks efficiently remains a major challenge. Existing methods that only update a small fraction of model parameters struggle with complex tasks like object detection and semantic segmentation. They use fixed transformations that work the same for all images, and different layers of the network learn redundant information, limiting performance. We introduce ParaX, a new approach that treats small sets of trainable parameters as "experts" stored in a large, shared center. For each input image, ParaX dynamically selects and combines the most relevant experts to create customized parameters optimized for that specific image. This input-specific adaptation, combined with sharing experts across network layers, helps the model learn more useful features and avoid redundancy. ParaX achieves strong performance across diverse visual tasks, even outperforming updating the full model in some cases while using less than 4% of the trainable parameters.