Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane

Rotary Position Embedding (RoPE) is the de facto positional encoding in large language models due to its ability to encode relative positions and support length extrapolation. When adapted to vision transformers, the standard axial formulation decomposes two-dimensional spatial positions into horizontal and vertical components, implicitly restricting positional encoding to axis-aligned directions. We identify this directional constraint as a fundamental limitation of the standard axial 2D RoPE, which hinders the modeling of oblique spatial relationships that naturally exist in natural images. To overcome this limitation, we propose Spiral RoPE, a simple yet effective extension that enables multi-directional positional encoding by partitioning embedding channels into multiple groups associated with uniformly distributed directions. Each group is rotated according to the projection of the patch position onto its corresponding direction, allowing spatial relationships to be encoded beyond the horizontal and vertical axes. Across a wide range of vision tasks including classification, segmentation, and generation, Spiral RoPE consistently improves performance. Qualitative analyzes of attention maps further show that Spiral RoPE exhibits more concentrated activations on semantically relevant objects and better respects local object boundaries, highlighting the importance of multi-directional positional encoding in vision transformers.

When an AI system "looks at" a photograph, it actually chops the image into a grid of small tiles and learns how those tiles relate to one another. For this to work, the model needs to know where each tile sits in the picture. The standard method for providing this location information was inherited from language models, and it treats only two directions as special: left-right and up-down. But real photographs are full of edges, curves, and objects oriented at every angle, so this two-axis bias quietly limits what vision models can perceive. We propose Spiral RoPE, a small redesign that spreads position information evenly across many directions instead of just two. When visualized, the resulting pattern traces a spiral, giving the model a more balanced sense of image geometry. The change adds no extra parameters and no extra computation. Across image classification, segmentation, and generation, Spiral RoPE consistently improves results, and the model focuses its attention more sharply on the objects that matter. Our findings suggest that even such a subtle modification about how a model is told "where" can meaningfully shape what it learns to see.