Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users' real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user's movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.

Future AI assistants may help people by understanding what they see, where they are, and how their surroundings change over time. However, today’s AI models still struggle to remember object locations across long videos, especially when the user moves around or when objects are moved. To study this challenge, we introduce UCS-Bench, a large benchmark built from more than 170 hours of first-person video and over 7,000 time-stamped questions. These questions test whether an AI system can remember where things are, reason about changes in a space, and answer questions based on the user’s current location. We also introduce DirectMe, a method that helps AI systems build and update a memory of the surrounding environment as new video arrives. This memory allows the system to keep track of objects over time and answer questions that require looking back across long periods. Our experiments show that DirectMe improves the ability of leading AI models to reason about spaces from first-person videos, outperforming many existing video and spatial reasoning systems. We hope UCS-Bench and DirectMe will support future research toward more helpful AI assistants that can understand and remember the spaces around their users.