From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning

Modern vision-language models achieve strong performance in static perception, but remain limited in the complex spatiotemporal reasoning required for embodied, egocentric tasks. A major source of failure is their reliance on temporal priors learned from passive video data, which often leads to spatiotemporal hallucinations and poor generalization in dynamic environments. To address this, we present EgoTSR, a curriculum-based framework for learning task-oriented spatiotemporal reasoning. EgoTSR is built on the premise that embodied reasoning should evolve from explicit spatial understanding to internalized task-state assessment and finally to long-horizon planning. To support this paradigm, we construct EgoTSR-Data, a large-scale dataset comprising 46 million samples organized into three stages: Chain-of-Thought supervision for explicit reasoning, weakly supervised tagging for internalizing reasoning, and long-horizon sequences composed of orthogonal atomic subtasks. We further introduce a Reasoning-Enhanced Task Decomposition mechanism that explicitly models causal dependencies between actions and objects, enabling controllable planning from high-level task descriptions. To rigorously validate our approach, we establish a Dual-Level Evaluation Framework that covers both atomic spatial perception and logical planning. Extensive experiments demonstrate that EgoTSR effectively eliminates chronological biases, achieving 92.4% accuracy on long-horizon logical reasoning tasks while maintaining 88.2% fine-grained perceptual precision, significantly outperforming existing open-source and closed-source state-of-the-art models.

Robots and AI assistants need to understand not only what they see, but also whether a task is actually moving toward completion. For example, when preparing food or tidying a desk, the same objects may appear in many different states, and simply knowing the order of video frames is not enough. Existing vision-language models often rely too much on the usual order of events in videos, which can make them wrongly assume that later-looking scenes are always closer to the goal. This paper introduces EgoTSR, a training framework that helps models reason about first-person task progress more reliably. Instead of teaching the model everything at once, EgoTSR trains it in stages: first to explain visual changes step by step, then to make direct task-state judgments, and finally to handle longer tasks made of multiple smaller actions. The paper also builds a large dataset with 46 million examples and designs tests that separately measure short-term visual understanding and long-term task planning. Experiments show that EgoTSR can better tell whether a task has progressed, reduce mistakes caused by misleading time order, and perform strongly across human demonstrations, simulations, and real-robot settings.