Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation

Referring Expression Segmentation (RES) aims to generate pixel-wise segmentation masks from complex and implicit textual queries. While recent advances in Multimodal Large Language Models (MLLMs) have substantially boosted RES performance, their prohibitive computational overhead remains a critical bottleneck, which, however, is rarely explored. To fill this gap, we first evaluate typical token compression methods on this task and observe a surprising performance degradation. In this paper, we aim to understand this phenomenon for a solution. By extensive experiments, we find that token compression for RES requires preserving the original position embeddings and local neighboring spatial structures, indicating that visual token position information is far more critical than in other tasks. Building on this insight, we ask: Can we design the token compression method purely based on the position information? Therefore, we propose PAYN, a plug-and-play, training-free token compression method that relies solely on position information. PAYN retains tokens that are adequately distributed in every local neighboring region while strictly preserving original positional indices, thereby maintaining spatial relational consistency. Experiments on multiple RES benchmarks demonstrate that our method outperforms existing token compression methods, verifying that position is indeed all you need for token compression in the MLLM-based RES task. Codes are avaliable at https://github.com/YuhanLiu231/PAYN.

When people ask AI systems to identify an object in an image using natural language — such as "the dog sitting on the bench" — the system must understand both the text and the detailed visual layout of the image to produce an accurate pixel-level mask. Recent multimodal large language models (MLLMs) have greatly improved this ability, but they are often too computationally expensive for practical use. A common way to speed up these models is to compress visual information by removing less important image tokens. However, we found that existing compression methods cause an unexpected and severe drop in performance for this task. Through extensive experiments, we discovered that this task strongly depends on preserving the original spatial structure of the image, especially the relative positions between nearby regions. Based on this insight, we developed PAYN, a simple training-free method that compresses visual tokens using only positional information. PAYN keeps tokens evenly distributed across the image while preserving their original spatial relationships. Experiments on multiple benchmarks show that PAYN can generate more accurate segmentation results while using fewer visual tokens than previous methods.