From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations

We propose a new paradigm for long video understanding by treating a long video as a Neural Knowledge Representation (NKR). NKR represents video contents neither as a stream of tokens nor pre-organized databases, but as an individual small portion of network weights attached to the VLM backbone. The NKR weights are optimized to encapsulate the video's semantic content via a novel Agentic Knowledge Distillation (AKD) process, where an agent automatically synthesizes dense descriptions and question-answer pairs to distill the video's knowledge into the NKR. While AKD serves as a comprehensive, one-time encoding phase, the resulting NKR transforms the video into a portable, reusable asset. At inference, the lightweight NKR is mounted onto a frozen Vision-Language Model (VLM), enabling direct, query-based understanding without reloading or re-encoding the original video. This approach decouples video length from inference cost, offering high amortized efficiency for multi-turn video understanding. Experiments on the LVBench benchmark show our method achieves performance comparable to state-of-the-art approaches while reducing end-to-end latency by over two orders of magnitude, opening new possibilities for interactive long-video understanding.

Imagine asking an AI assistant a quick question about a two-hour movie you just watched. Today, this is surprisingly slow and expensive: the AI has to re-watch the entire video from scratch every time you ask something, and longer videos quickly overwhelm its memory. Existing solutions either skim the video and miss important moments, or run a long, multi-step search that can take several minutes per question. We propose a different idea: instead of repeatedly feeding the raw video to the AI, we let the AI study the video once and store everything it learned as a small, plug-in "memory chip" of its own. To create this chip, we built an automatic tutor that watches the video, writes detailed notes, and quizzes itself with thousands of questions and answers, then condenses all of that knowledge into a tiny set of numbers attached to the AI. Once the chip is ready, the AI can answer questions about the video almost instantly, using essentially no extra memory, and our results show it understands hour-long videos as well as much heavier systems. This opens the door to fast, interactive conversations with AI about long videos — from lectures and sports matches to surveillance footage and personal memories.