ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response

While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human's next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21\%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58\%. Code is available at https://github.com/ZhuXMMM/ProAct.git

Many robots and AI assistants today only respond after a person gives a command. In many real situations, however, useful assistance requires acting earlier, such as preparing the next tool, replacing a missing item, or noticing a safety issue. This paper introduces ProAct-75, a benchmark for studying this kind of proactive assistance from video. The benchmark contains 75 tasks with detailed labels for what is happening, whether help is needed, what may happen next, and what action an assistant should take. It also includes task structures that describe which steps must happen in order and which steps can be done at the same time. We also present ProAct-Helper, a baseline system that uses these structures to make more useful and less disruptive assistance decisions.