ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory

Streaming video understanding (SVU) must answer queries that arrive asynchronously while visual tokens stream continuously under strict GPU-memory and query-time latency budgets. A key challenge is delayed query: decisive cues may appear briefly, yet many subsequent updates occur before the query arrives, increasing the risk that those cues are evicted or diluted under bounded memory. We propose ProtoKV, a constant-footprint SVU memory that represents far history as a fixed-capacity summary state rather than retaining token instances. ProtoKV keeps an exact near-window KV cache and aggregates older content into a semantic–spatial prototype bank with residual statistics. At query time, each prototype is exposed through a bounded pseudo-token interface that is drop-in compatible with standard attention. Under matched budgets and comparable query-time cost, ProtoKV improves accuracy by up to 12.5 points over token-retention baselines on SVU benchmarks in the long-delay regime, with gains that grow as query delay increases.

Modern AI systems are increasingly expected to understand video streams as they happen, such as in assistants, robots, or real-time monitoring tools. A key difficulty is that a user may ask a question long after the important visual clue has already appeared, while the system has only limited memory to store what it has seen. If the system simply keeps the most recent video information, it may forget the crucial moment before the question arrives. We introduce ProtoKV, a memory method for streaming video models that keeps recent information exactly while compressing older video history into a small set of meaningful summaries. Instead of trying to remember every past video token, ProtoKV groups older information into prototype memories that can still be used by the model when answering later questions. This makes video AI more reliable under memory limits, especially when the answer depends on something that happened far in the past. Our experiments show that ProtoKV improves delayed-question answering while keeping memory use and response time practical for streaming applications.