Beyond Single-View Indexing: Structure-Aware Multi-View Retrieval for Knowledge-Based VQA

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieval from large-scale knowledge bases, yet this stage is often treated simplistically. Existing methods typically adopt single-view indexing or naive multi-view fusion, leading to systematic coverage gaps. In this work, we demonstrate that different views exhibit strong complementarity in retrieval. Motivated by this observation, we propose SCAR, a Structure-aware Cross-View Retrieval framework that exploits cross-view structural complementarity at inference time without additional training. SCAR enhances retrieval via structure-aware similarity propagation within each view and explicit cross-view redundancy regulation. Experiments on multiple KB-VQA benchmarks demonstrate that SCAR substantially improves retrieval recall, approaches retrieval coverage upper bounds, and consistently boosts end-to-end KB-VQA performance with negligible inference overhead.

The Problem: LLM Visual QA systems answering questions often need to search external databases for facts. Current methods usually search using just one perspective, like an image or a text summary. This narrow approach creates systematic blind spots, causing the LLM to miss vital information. These search failures strictly limit how accurately the LLM can generate the final answer. The Solution: We developed SCAR, a new method that simultaneously looks across multiple indexes without needing extra training. SCAR groups related concepts within each view to boost relevant information while suppressing isolated errors. It then coordinates across different views to combine unique clues while actively filtering out repetitive search results. The Impact: Our experiments show SCAR dramatically improves search accuracy, coming close to perfect coverage. By successfully finding the right facts, SCAR consistently helps Visual LLM generate much more accurate answers. Best of all, it achieves this with almost negligible extra processing delay in the VQA pipeline.