SOLAR: Self-supervised Joint Learning for Symmetric Multimodal Retrieval

In this work, we address the critical yet underexplored challenge of symmetric multimodal-to-multimodal (MM2MM) retrieval, where queries and contexts are interchangeable. Existing universal multimodal retrieval works struggle with this task, as they are constrained by the labeled asymmetric datasets used. We produce SOLAR (Self-supervised jOint LeArning for symmetric multimodal Retrieval), a novel two-stage self-supervised framework that leverages readily available unlabeled web-scale image-text pairs. Based on the observation that both semantic alignment and discrepancies exist between two modalities, in the first stage, we learn the intersection mask of image-text pair, allowing us to align intersection while preserving semantic of difference. In the second stage, the learned mask is further utilized to construct positive and hardnegative samples via masking different parts of image/text, which enable us to conduct self-supervised multimodal embedding learning. Complementing this framework, we present a new benchmark featuring high-quality human-verified positive and hard-negative pairs to evaluate symmetric MM2MM retrieval under realistic conditions, as well as the corresponding pipeline. Extensive experiments against ten SOTA methods show SOLAR surpasses the strongest supervised VLM by 7.08 points on this benchmark, with over 50x fewer model parameters and a 5x smaller embedding dimension. Code, model and benchmark are available at https://github.com/codefuse-ai/SOLAR.

Modern search engines often struggle with a specific type of task called "symmetric multimodal retrieval." This happens when users want to search for information using a combination of images and text, where the query and the result are essentially interchangeable—like searching for a t-shirt’s back using an image of its front. Existing AI models are rarely built for this; they are usually trained on "asymmetric" data, where the query and the result play distinct roles, making them poor at understanding how different modalities relate to the same object. Furthermore, teaching AI to understand these relationships usually requires massive amounts of expensive, human-labeled data. To solve this, we developed SOLAR, a new AI framework that learns entirely on its own using unlabeled web-scale image-text pairs. We designed a method that teaches the AI to "disentangle" information—separating the concepts that are shared between an image and its text from the details unique to each. By masking these specific parts, SOLAR automatically generates its own high-quality training examples. In tests, SOLAR outperformed leading, much larger AI models on our new benchmark, achieving higher accuracy with over 50 times fewer parameters. Our work shows that we can build highly effective, efficient search tools without relying on costly human labor.