Vision Transformer Finetuning Benefits from Non-Smooth Components

The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their *plasticity*. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments -- over $1,000$ finetuning runs on large-scale vision transformers -- showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on transformers' functional properties. The code is available at \url{https://github.com/ambroiseodt/vit-plasticity}.

1) Smoothness is often seen as a desirable property in machine learning. However, its role in transferring knowledge to a new task remains unclear. 2) In our work, we introduce the notion of plasticity that captures how transformer modules respond to changes in the data. It is connected to smoothness as follows: a high plasticity means amplification of the changes in the input and non-smooth components, while low plasticity means robustness to input changes and smooth components. We use this measure to identify the most sensitive layers and show, via large-scale experiments on vision transformers, that they adapt better to new data. 3) This provides a novel perspective on the role of smoothness in machine learning, with valuable insights towards adapting models more efficiently.