Spatio-Temporal LLM: Reasoning about Environments and Actions

Despite significant recent progress of Multimodal Large Language Models (MLLMs), current MLLMs are challenged by "spatio-temporal" prompts, i.e., prompts that refer to 1) the entirety of an environment encoded in a point cloud that the MLLM should consider; and simultaneously also refer to 2) actions that happened in part of the environment and are encoded in a short ego-centric video clip. However, such a holistic spatio-temporal understanding is important for agents operating in the real world. To address this challenge, we first develop a framework to collect a large-scale dataset. Using the collected "Reasoning about Environments and Actions" (REA) dataset, we show that recent MLLMs indeed struggle to correctly answer "spatio-temporal" prompts. Building on this dataset, we study two spatio-temporal LLM (STLLM) baselines: 1) STLLM-3D, which directly fuses point cloud, video, and text representations as inputs to the LLM; and 2) STLLM-Aligner, which aligns spatial context with video and text before LLM decoding. Both baselines aim to enhance spatial understanding of environments and temporal grounding of egocentric observations. On REA, the STLLM baselines outperform existing models, demonstrating the effectiveness of our designs.

AI systems are increasingly expected to assist people and robots in real-world environments, but understanding what is happening around them is still difficult. In many situations, an agent must connect a local first-person video of an ongoing action with a broader understanding of the surrounding space, such as where objects are located in the room. Existing multimodal models often struggle with this kind of joint spatial and temporal reasoning. In this work, we introduce Reasoning about Environments and Actions (REA), a dataset designed to test whether models can answer questions that require both egocentric video understanding and global 3D scene understanding. We also study two Spatio-Temporal LLM baselines that combine video, text, and 3D point-cloud information to better reason about actions in their environment. Our results show that current models still have difficulty with these tasks, while our proposed baselines improve performance by using explicit 3D context. This work provides a step toward AI systems that can better understand actions, objects, and navigation in real-world spaces.