From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers

Feature-map knowledge distillation (KD) transfers internal representations well between comparably sized Vision Transformers (ViTs), but it often fails in compression. We revisit this failure and uncover a paradox. Sample-wise SVD shows that each image is highly compressible, which seems to suggest that a narrow student with a linear projector should match the teacher "in principle". However, a dataset-level view contradicts this intuition: PCA shows that the teacher is a union of low-rank subspaces with significant subspace rotation across inputs. We further introduce token-level Spectral Energy Patterns (SEP) and find an architecture-invariant encoding law: tokens spread energy broadly across channel modes even when they live in low-rank subspace, creating a bandwidth mismatch. We refer to this combined phenomenon as an encoding mismatch. We propose two minimal remedies, Lift or WideLast: (i) Lift retains a lightweight lifting projector at inference to provide wider channel, or (ii) WideLast widens only the student’s last block, enabling an input-dependent expansion. On ImageNet-1K, these fixes revive feature KD for ViT compression, improving DeiT-Tiny distilled from CaiT-S24 from 74.86% to 77.53%/78.23% top-1 accuracy, and they also strengthen students trained without distillation. Our analyses clarify when and why feature-map KD fails and then how to fix it. Code and raw data are provided in https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch.

Neural networks recognize images by turning pixels into internal descriptions of shapes, objects, and visual patterns. When training a smaller network, a common idea is to ask it to copy the internal descriptions produced by a larger, stronger network. Our paper studies why this copying often works poorly for modern vision networks. We find that the larger network’s description of any single image can look simple, but the way it organizes those descriptions changes from image to image. At the same time, each small part of an image uses information spread across many internal channels, rather than a small fixed subset. This creates a mismatch: the smaller network does not have enough room, or enough flexibility, to encode visual features in the same way as the larger network. We propose two simple fixes that give the smaller network more space only at the final stage where these descriptions are matched. These changes make feature copying useful again and improve accuracy while adding much less cost than making the whole network large. The results offer a clearer view of how neural networks encode visual information and how to design compact models that learn from larger ones more effectively.