Attentive Multi-Layer Fusion for Vision Transformers

With the rise of large-scale foundation models, efficiently adapting them to downstream tasks remains a central challenge. Linear probing, which freezes the backbone and trains a lightweight head, is computationally efficient but often restricted to last-layer representations. We show that task-relevant information is distributed across the network hierarchy rather than encoded solely in the last layers. To leverage this distribution of information, we apply an attentive probing mechanism that dynamically fuses representations from all layers of a Vision Transformer. This attentive layer fusion (ALF) learns to identify the most relevant layers for a target task and combines low-level structural cues with high-level semantic abstractions. Across 20 diverse datasets and multiple pretrained foundation models, ALF achieves consistent, substantial gains over standard linear probes. Attention heatmaps further reveal that tasks different from the pre-training domain benefit most from intermediate representations. Overall, our findings underscore the value of intermediate layers and demonstrate a principled, task-aware approach for unlocking their potential for probing-based adaptation.

Modern AI vision models process images through a series of computational stages, called "layers". Early layers detect simple patterns like edges and textures, while deeper layers build up abstract concepts like "fluffy animal with pointy ears". These models are expensive to build, so ideally, we reuse the same model across many different tasks without modifying it. While it has been known that intermediate layers also carry useful information, the standard approach reads off only the final layer's output. We propose a lightweight mechanism that learns, for each new task, which layers matter most and how to combine them, while the underlying model itself remains untouched. This automatic, task-aware fusion consistently outperforms approaches that rely on the final layer alone, especially on tasks that differ substantially from the kinds of images the model was originally trained on. Our findings provide a principled way to make use of information from all stages inside a model by automatically identifying which layers are most relevant for a given task. This enables more efficient adaptation of large general-purpose models to specialized applications, particularly when computational resources are limited.