Attention Weight Decomposition for Vision Model Compression

Weight decomposition is a practical compression approach, yet prior methods overlook that multi-head attention consists of paired linear projections ($Q$–$K$ and $V$–$O$). We propose \textit{Uni}lateral decomposition for attention \textit{Com}pression (\textit{UniCom}), which decomposes only one side of each pair to better manage approximation error while leveraging attention’s linear operations. Since $Q$-$K$-$V$-$O$ weights exhibit different low-rank sensitivities across heads, UniCom adaptively chooses which side to decompose based on rank sensitivity. Experiments show strong generalization across vision and vision–language tasks. Under 60\% MHA reduction, UniCom improves Top-1 by +11.2\% / +5.7\% / +6.4\% over combined decomposition on DeiT-S / DeiT-S(distill) / DeiT-B, respectively.