Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models

Rotary Position Embedding (RoPE) is widely adopted in large language models, but when applied to vision-language models (VLMs) it couples text and image position indices and can introduce spurious cross-modal relative-position bias. We propose *Per-Token Distance* (PTD) to quantify cross-modal positional disentanglement, and we prove that $\mathrm{PTD}=0$ is a sufficient condition to eliminate the geometric attention bias induced by RoPE. Guided by this criterion, we introduce Circle-RoPE, which remaps 2D image-token coordinates onto an annulus orthogonal to the text position axis, yielding a cone-like geometry where each text token is equidistant to all image tokens while preserving intra-image spatial structure. We further propose Alternating Geometry Encoding (AGE) to synergize complementary geometric priors by alternating the decoupled geometry of Circle-RoPE and the grid-based prior of standard RoPE across layers. This design ensures both rigorous cross-modal disentanglement and the preservation of fine-grained intra-image spatial structure, and experiments on diverse VLM backbones and multimodal benchmarks show consistent gains in spatial grounding and visual reasoning. The code is available at https://github.com/lose4578/CircleRoPE.

Many AI systems can answer questions about images, but they must connect words in a question to the right parts of a picture. A hidden difficulty is how the system represents position. Common methods convert image patches and words into a single ordered list, which can make some image regions seem artificially closer to a word simply because of how the list is arranged, not because they are truly relevant. This can hurt tasks that require spatial understanding, such as finding something “high on” an object. We introduce Circle-RoPE, a new way to represent positions for vision-language models. Instead of forcing image patches and words into the same line, it places image patches in a circular spatial layout and separates text from images geometrically. This gives every word a more balanced positional relationship to the whole image while preserving the image’s own layout. As a result, the model can rely more on visual meaning and less on accidental ordering, improving visual reasoning and spatial grounding across benchmarks.