RVAS: Referring Video Active Exploration and Segmentation

Existing referring video object segmentation (RVOS) is largely built on passive perception and assumes the target is already visible in the observed video, which limits real-world use when queries refer to objects beyond the current view. To address this gap, we introduce Referring Video Active Exploration and Segmentation (RVAS), a new task that focuses on reasoning about exploration policy and then locating and segmenting the object according to an input referring expression. To support RVAS, we build a large-scale dataset with manually annotated exploration actions and reference reasoning traces, enabling supervised training and evaluation. We benchmark representative RVOS and related video understanding baselines and find that they struggle to perform active target search and incur substantial overhead when coupled with online decision making. Motivated by these challenges, we propose LESA, a baseline framework that introduces a state controller and hierarchical memory for efficient streaming processing and sparse MLLM reasoning. LESA substantially reduces inference cost while maintaining competitive planning quality, and consistently improves segmentation accuracy on the RVAS dataset. Code is available at https://github.com/FudanCVL/RVAS.

Consider a household robot asked to “check the time” when the clock is in another room. Most current video AI systems are not designed for this setting: they describe objects in a recorded video, but cannot decide where to look when the target is outside the current view. This limits their usefulness in practical scenarios such as robotics and smart glasses. We introduce Referring Video Active Exploration and Segmentation (RVAS), a task in which an AI agent must first plan how to explore an environment, then locate and segment the referred object at the pixel level once it is observed. To support this task, we collect a large dataset of 911 first-person videos from homes, offices, and outdoor scenes, with 18,600 annotated actions and detailed human reasoning traces. We further propose LESA, a system inspired by human perception: it performs lightweight continuous monitoring most of the time and invokes deeper reasoning only when needed. This design makes LESA roughly 10 times faster than applying a large AI model to every frame, while also improving object localization over prior methods.