General Covariant Action Modeling: Constructing Generalized Manifolds via Spatio-Temporal Decoupling

Achieving robust generalization from limited data is a central challenge in embodied intelligence. Prevailing methods fail by regressing absolute coordinates, which violates the principle of general covariance. Theoretically, this conflates the intrinsic task geometry with rigid execution patterns, binding policies to specific motion styles and fixed speeds. To resolve this, we propose the Generalized Action Manifold (GAM) framework that enforces general covariance through structural disentanglement. Specifically, GAM constructs the manifold by enforcing invariance across two orthogonal dimensions: (1) Temporal Invariance, utilizing an Arc-Length Parameterizer to orthogonalize the spatial path geometry from temporal dynamics, ensuring robustness to velocity variations; (2) Geometric Invariance, where a Schema-Affine-Factorization mechanism maps trajectories to canonical “world lines” in the Lie-algebraic tangent space. This distinguishes invariant topological schemas from affine modulations, ensuring spatial generalizability. By integrating GAM within a structured Vision-Language-Action (VLA) architecture, we expand sparse training data into a continuous, valid action manifold. Empirical results demonstrate that GAM enables superior transfer and robustness capabilities, significantly outperforming geometry-agnostic baselines.

Robots learn manipulation skills, like picking up a cup or stacking blocks, by watching human demonstrations and imitating the recorded motions. But humans rarely perform the same task twice in exactly the same way: speeds vary, starting positions differ, objects sit at different angles. When today's robots try to learn from these natural variations, they tend to "average" them, sometimes reaching toward the empty space between two valid grasps or smoothing over the precise moment of contact. The result is a brittle policy that works in the lab but fails in the real world. This paper traces the problem to a single root cause: current methods learn from raw coordinates and timestamps, which entangle the intent of a motion ("grasp from above") with the incidental details of how it was performed. We propose a framework called the Generalized Action Manifold (GAM) that pulls these two layers apart. One module strips away differences in execution speed, so the robot focuses on the shape of the motion, not its pace. Another strips away differences in position and orientation, so the robot focuses on the geometry of the motion, not the specific frame it happened in. What remains is a clean library of reusable motion patterns that the robot can flexibly adapt to new objects and scenes. When integrated into a modern vision-language-action model, GAM reaches state-of-the-art results on standard manipulation benchmarks, and remains stable when the robot is asked to act faster, slower, or under unfamiliar spatial layouts. We view this as a step toward robots that learn the structure of physical tasks rather than just their surface appearance, an essential ingredient for reliable behavior in the unpredictable real world.