Pix2Key: Controllable Open-Vocabulary Retrieval with Semantic Decomposition and Self-Supervised Visual Dictionary Learning

Composed image retrieval uses a reference image plus a natural-language edit to retrieve images that apply the requested change while preserving other relevant visual content. Classic fusion pipelines typically rely on supervised triplets and can lose fine-grained cues, while recent zero-shot approaches often caption the reference image and merge the caption with the edit, which may miss implicit user intent and return repetitive results. We present Pix2Key, which represents both queries and candidates as open-vocabulary visual dictionaries, enabling intent-aware constraint matching and diversity-aware reranking in a unified embedding space. A self-supervised pretraining component, V-Dict-AE, further improves the dictionary representation using only images, strengthening fine-grained attribute understanding without CIR-specific supervision. On the extended DeepFashion-MM benchmark, Pix2Key improves Recall@10 up to 3.2 points, and adding V-Dict-AE yields an additional 2.3-point gain while improving intent consistency and maintaining high list diversity.

This paper studies how to search for images using both an example image and a requested change. For example, a user may want a product that looks like a reference photo but has a different color, material, or style. This is challenging because the system must understand what should stay the same and what should change. We propose Pix2Key, a method that represents images and user requests as editable visual attributes, such as color, pattern, material, and style. These attributes allow the system to clearly separate what the user wants to add, remove, or preserve. This makes the search process more controllable and easier to understand than methods that compress everything into one hidden representation. We also design a new evaluation setting to measure whether search results follow detailed user intent and provide diverse options. Our experiments show that Pix2Key gives better and more useful results for flexible image search.