Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation

Cross-task generalization is a core challenge in open-world robotic manipulation, and the key lies in extracting transferable manipulation knowledge from seen tasks. Recent in-context learning approaches leverage seen task demonstrations to generate actions for unseen tasks without parameter updates. However, existing methods provide only low-level continuous action sequences as context, failing to capture composable skill knowledge and causing models to degenerate into superficial trajectory imitation. We propose Decompose and Recompose, a skill reasoning framework using atomic skill-action pairs as intermediate representations. Our approach decomposes seen demonstrations into interpretable skill--action alignments, enabling the model to recompose these skills for unseen tasks through compositional reasoning. Specifically, we construct a task-adaptive dynamic demonstration library via visual-semantic retrieval combined with skill sequences from a planning agent, complemented by a coverage-aware static library to fill missing skill patterns. Together, these yield skill-comprehensive demonstrations that explicitly elicit compositional reasoning for skill composition and execution ordering. Experiments on the AGNOSTOS benchmark and real-world environments validate our method's zero-shot cross-task generalization capability.

A central challenge for general-purpose robots is performing tasks they have never seen before. A promising approach lets a model use a few examples of seen tasks to decide how to act on a new task, with no retraining. The problem with existing methods is that they only show the model the raw action sequences from those examples, so the model merely copies the example trajectories in a superficial way. Our method instead represents each example as a pair consisting of a skill and an action, such as "pick up" or "pour." The model first breaks seen examples into these skill and action pieces, and then, through reasoning, recombines the skills to accomplish a new task. To ensure that the needed skills from seen tasks are fully available, we select examples that are visually and semantically relevant, guided by a planning component, and additionally provide a backup set of examples chosen for coverage. On a benchmark for unseen robot tasks and on real robots, our method improves the robot's ability to perform new tasks without any retraining.