EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering

Knowledge-based Visual Question Answering (KB-VQA) remains a challenging task, particularly when queries require precise identification and grounding of fine-grained entities within large-scale knowledge base. Existing methods often treat visual and textual signals in isolation and rely heavily on image-centric retrieval, which makes them sensitive to visual ambiguities. To address these limitations, we propose EntRAG, an entity-centric retrieval-augmented generation framework. Our approach first introduces EntBind to align query representations with multimodal entity embeddings by explicitly binding entity tokens to latent visual features, retrieving a set of relevant candidate entities. A reranking mechanism is applied to these candidate entities to select the most informative context by combining entity-level alignment with overall contextual relevance. The selected evidence is incorporated into context-aware generation module to produce final answer. By explicitly operating at the entity level, EntRAG achieves more consistent and reliable results. Extensive experiments demonstrate that EntRAG consistently outperforms prior methods, achieving scores of 46.1 on E-VQA and 44.5 on InfoSeek.

We study how to improve knowledge-based visual question answering (KB-VQA), where systems must answer questions about images using both visual content and external knowledge. This is challenging when answers depend on recognizing specific, fine-grained entities that can be visually ambiguous. We propose EntRAG, an entity-centric framework that directly retrieves and reasons over relevant entities instead of relying mainly on image-level similarity. Given a question, EntRAG aligns it with multimodal entity representations using a module called EntBind, which links query signals to visual features associated with entities. A reranking step then selects the most relevant entities by combining query alignment and contextual usefulness. These selected entities are used as evidence for a generation module that produces the final answer. By explicitly operating at the entity level, EntRAG improves grounding and reasoning consistency. It achieves strong results on benchmarks such as E-VQA and InfoSeek, outperforming prior approaches