Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

Dual-encoder vision-language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like “umbrella and no person” retrieve images containing both, even when concept detection is reliable. We trace this to an interface-level Bag-of-Concepts effect, where similarity scores approximate mean pooling of concept evidence regardless of operators. Although operator-dependent signals exist in text embeddings, they are too weak or misaligned to affect rankings. Fine-tuning does not reliably resolve this failure because the dominant bottleneck is how similarity aggregates evidence rather than what encoders represent. We propose factored inference, which separates evidence extraction from constraint execution, and introduce LCSE (Logic-Constrained Score Editing), a training-free method that executes constraints externally using concept scores from frozen encoders. We also introduce FACTOR-Bench, where LCSE achieves 85.5% accuracy versus 73.2% for the best fine-tuned baseline, 90.7% when applied to SigLIP 2, and improves NegBench COCO MCQ accuracy from 27.2% to 65.2% while preserving retrieval performance.

Modern AI systems can search images using text, but they often stumble on simple logic. Ask for “a room with no cat,” and the system may still return rooms with cats. Ask for “an umbrella and a person,” and it may return images with only one of them. The problem is that many image-search models compare an image and a sentence using a single matching score. That score can tell that “cat,” “umbrella,” or “person” is relevant, but it does not reliably follow words like “and,” “or,” and “not.” We show that this is a bottleneck in the matching step itself, so simply retraining the model only partly helps. Our method, Logic-Constrained Score Editing (LCSE), keeps the original model unchanged. It checks each requested object separately, then applies the logic outside the model. On logical image-search queries, this raises accuracy from about 58.3% to 85.5% with CLIP, a widely used image-search model, and to 91% with a stronger model, without hurting ordinary image search. We also release FACTOR-Bench, a new benchmark for measuring progress on this problem.