Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration

Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-Attention Divergence, in which output logits are heavily biased while internal attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40\% compared to baselines. Code is available at https://github.com/brightXian/LAD.

When AI is asked to pick the most relevant image from a group, it sometimes takes a shortcut: instead of carefully looking at each image, it simply favors whichever one appears first. This means the answer can change just by shuffling the order of images, which is clearly wrong. We looked inside the model to understand why this happens. Interestingly, the model actually "looks at" the right images internally, but something goes wrong when it turns that understanding into a final answer. The bias sneaks in at that last step. Using this insight, we built a simple fix that catches and corrects this error on the fly, without retraining the model or adding much extra computation. In experiments, our method makes AI image retrieval far more consistent and accurate, improving performance by over 40% compared to previous approaches.