Compass-RoPE: Isotropic Rotary Position Embeddings for Vision Transformers

Recent works introduce Rotary Position Embeddings (RoPE) into vision transformers (ViTs) to enhance their extrapolation capability, i.e., maintaining performance when inference is conducted on higher resolution images. RoPE encodes positions via rotating phases whose change is controlled by frequency components. Strandard 2D RoPE does not generalize well to input resolution changes as it only applies axial frequencies separately along each individual axis. To solve this issue, Mix-RoPE combines xy‑axis frequencies, such that it can model position relations in diagonal direction. However, in practice, we observe that the learned 2D frequencies become anisotropic in their direction distributions due to the axial spectral bias in image features, limiting the extrapolation ability of ViTs. Motivated by this observation, we propose Compass‑RoPE. We replace the xy cartesian coordinates with a polar parameterization that explicitly decouples frequency scale and angle. By initializing the angle vectors uniformly over [0,2π), it ensures the isotropic direction coverage. Besides, we further introduce discrete Fourier transform (DFT) mixing for the angle vectors, allowing each transformed individual angle vector element to nest multipule angles and thus to enrich angular expressiveness. Extensive experiments on multi-resolution classification and dense prediction tasks show that our Compass-RoPE achieves more stable extrapolation performance under large-scale resolution changes.

Modern image-understanding AI systems are often trained on images of one size but are expected to work well on larger or differently sized images. This is challenging because these systems need to know where each image patch is located, and this position information may not transfer smoothly when the image resolution changes. We found that the common patching process used in vision transformers can make the model prefer horizontal and vertical directions, while paying less attention to other directions. This imbalance can hurt performance when the model sees images at new resolutions. We propose Compass-RoPE, a new way to encode image positions. Instead of treating positions only through horizontal and vertical coordinates, Compass-RoPE represents spatial information using both direction and scale, like a compass. It also mixes multiple directions inside each attention head, helping the model capture richer spatial relationships. Experiments show that Compass-RoPE improves robustness across image classification, high-resolution evaluation, object detection, semantic segmentation, and image generation tasks. Our results suggest that more directionally balanced position encoding can help vision models generalize better to images of different sizes.