Rotation Symmetry in Vision Quantization: The Objective Function is the Bottleneck

Rotation-based post-training quantization (PTQ) is widely used to suppress activation outliers in large language models. In vision classification, however, rotations learned with the standard task loss perform worse than random rotations on ViT- family models. We evaluate rotation learning on the Stiefel manifold across five architectures (DeiT-S, Swin-T, ViT-S, ResNet-50, MobileNetV2). Replacing the task loss with Final KD or Block Output MSE eliminates this reversal in all five models. The gain is small on ViT-family models at +0.3∼0.7pp. On MobileNetV2, where rotation is restricted to conv1×1, task-loss accuracy of 18.79% rises to a 4-seed KD accuracy of 67.11±0.29%, a gap of about +48pp that exposes a strong architecture heterogeneity. The result suggests that the gain from rotation symmetry is determined more by the choice of optimization objective than by the symmetry itself.