PixCLIP: Towards Fine-grained Vision-Language Understanding via Any-granularity Pixel-Text Alignment

While CLIP has achieved strong performance across vision–language tasks, fine-grained image–text alignment remains challenging. Recent efforts improve textual granularity by leveraging long, detailed descriptions and replacing CLIP’s text encoder with LLM, but often overlook the visual-side bottleneck: achieving finer alignment requires region- and pixel-level visual grounding. To address it, we propose PixCLIP, a framework that jointly enhances both sides by accommodating visual prompt regions and long-form text within a unified training objective. Firstly, to support training at this granularity, we develop an automated annotation pipeline that produces long-form descriptions with pixel-level localization, and use it to construct LongGRIT, a large-scale dataset with 1.5M samples. Furthermore, we introduce a three-branch pixel–text alignment framework that aligns image regions with corresponding textual descriptions across multiple granularities. Experiments show that PixCLIP achieves state-of-the-art performance on pixel- and region-level alignment tasks while preserving strong results on standard global image–text retrieval benchmarks, even with arbitrarily shaped region prompts and long texts. Our code is available at https://github.com/StuHude/PixCLIP.

Modern AI systems can often match images with text, but they still struggle when the match needs to be very precise, such as finding the exact object, part, or irregular area in an image that corresponds to a detailed sentence. Many previous methods mainly improve the text side by using longer and richer descriptions, but this alone is not enough: the visual side also needs to understand fine details inside an image. In this paper, we introduce PixCLIP, a new framework that helps image–text models connect detailed language with specific regions of an image, even when those regions have arbitrary shapes. To train such a system, we build an automatic data-generation pipeline and create LongGRIT, a large dataset of nearly 1.5 million examples with detailed descriptions and precise visual locations. Experiments show that PixCLIP performs strongly on tasks that require matching text to specific image regions, while still maintaining good performance on standard image–text retrieval tasks. This makes the model more useful for applications that require fine-grained visual understanding, such as image search, visual question answering, and region-level image analysis.