Bridging the Grounding Gap in VideoQA via Typed Memory for Language-based Belief-State Reasoning

VideoQA models can be accurate yet often fail to align answers with the correct video segments (the \emph{grounding gap}). We introduce \textbf{LINGUA} (\textbf{L}anguage-based \textbf{IN}ference for \textbf{G}rounded Video \textbf{U}nderstanding \textbf{A}gent), a memory-based agent that performs grounded VideoQA by reasoning in an explicit \emph{linguistic belief state}. LINGUA uses five mechanisms: (1) event-driven perception (retains 8--12\% of frames while preserving 94\% of question-relevant events); (2) typed memory for episodic narratives, semantic affordances, and procedural scripts; (3) Belief-Action-Verification loops with postcondition and temporal checks; (4) meta reflection with contrastive refinement; and (5) Bayesian reliability tracking for continual learning without gradient updates. Built with Gemma3-4B (Ollama, 4-bit), LINGUA outperforms strong baselines on five VideoQA benchmarks, reaching 82.4\% on NExT-QA and 42.3\% Acc@GQA on NExT-GQA (answer + IoU$\geq$0.5 temporal localization), while running 2.6$\times$ faster than dense-frame methods. In continual learning over 100 videos, accuracy rises from 45.2\% (first 10) to 61.8\% (last 10) without catastrophic forgetting, indicating online adaptation via memory refinement.

--- Imagine asking a friend to watch a video and tell you what someone is doing. A good friend doesn't just give an answer — they point to the exact moment that justifies it. Today's AI systems that answer questions about videos often get the right answer for the wrong reasons: even when correct, they frequently cannot show you the part of the video that supports their answer. Researchers call this the "grounding gap" — recent work found that, for a state-of-the-art system, only about 16% of correct answers are actually backed by the right moment in the video. Our work, LINGUA, takes a different approach. Instead of reasoning in hidden mathematical patterns, the system reasons in plain language, the way a person might think out loud while watching a video. It organizes what it sees into three kinds of memory, loosely inspired by human memory: specific events that happened ("she opened the fridge"), general knowledge about objects and what they are used for ("a knife is for cutting"), and reusable recipes for everyday activities ("to prepare a meal, you gather ingredients, cook them, then serve them"). When asked a question, LINGUA forms a guess, checks whether the video evidence actually supports it — did the expected outcome really happen, roughly when expected? — and revises if not. When repeated attempts fail, it pauses to diagnose what went wrong and updates its understanding, without needing to be retrained. Despite running on a relatively small AI model, LINGUA matches or beats much larger systems on five standard video-question-answering tests while running more than twice as fast. Crucially, when it gives a correct answer, it can far more often point to the moment in the video that justifies it. The system also keeps getting better as it watches more videos, without forgetting what it learned earlier. This matters because video AI is increasingly used in places where being right is not enough — we also need the system to show its work. Examples include tools for visually impaired users, educational software, medical training, and content moderation. By making the AI's reasoning visible in plain language and tied to specific moments in the video, LINGUA aims to make such systems more trustworthy and easier for people to check, correct, and rely on. ---