Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph

Effectively retrieving, reasoning, and understanding multimodal information remains a critical challenge for agentic systems. Traditional Retrieval-augmented Generation (RAG) methods rely on linear interaction histories, which struggle to handle long-context tasks, especially those involving information-sparse yet token-heavy visual data in iterative reasoning scenarios. To bridge this gap, we introduce VimRAG, a framework tailored for multimodal Retrieval-augmented Reasoning across text, images, and videos. Inspired by our systematic study, we model the reasoning process as a dynamic directed acyclic graph that structures the agent states and retrieved multimodal evidence. Building upon this structured memory, we introduce a Graph-Modulated Visual Memory Encoding mechanism, with which the significance of memory nodes is evaluated via their topological position, allowing the model to dynamically allocate high-resolution tokens to pivotal evidence while compressing or discarding trivial clues. To implement this paradigm, we propose a Graph-Guided Policy Optimization strategy. This strategy disentangles step-wise validity from trajectory-level rewards by pruning memory nodes associated with redundant actions, thereby facilitating fine-grained credit assignment. Extensive experiments demonstrate that VimRAG consistently achieves state-of-the-art performance on diverse multimodal RAG benchmarks.

When modern AI assistants look for answers to complex questions, they often need to scan massive amounts of external data, including texts, images, and videos. However, visual information is heavily dense, meaning it quickly exhausts the AI's limited short-term memory or forces the system into repetitive loops where it mistakenly asks the same questions over and over. To solve this problem, we developed VimRAG, a framework that organizes the AI's reasoning process into a structured "memory graph" similar to a human mind map. This map connects clues logically so the AI can track its history, while a smart filtering system automatically focuses on pivotal visual details and filters out irrelevant background noise to save memory space. We also introduced a precise training strategy that correctly rewards the AI for every valid individual search step it takes. This research successfully unlocks the full capability of advanced vision-language models, allowing them to accurately comprehend massive video and image databases. Ultimately, it enables the deployment of highly efficient, trustworthy, and human-like AI assistants within practical computational budgets.