SkillNet: Hierarchical Skill Modeling for Compositional Generalization in Vision-Language Action Models

Transfer across diverse task compositions and unseen behaviors remains a significant challenge for vision-language action (VLA) models. Skills are repeatable and atomic components for various tasks, and similarities shared with different skills provide evidence for transferability across behaviors. However, existing skill-centric methods have two problems. First, skills are often loosely organized, lacking a hierarchy that can capture similarities and differences across skills. Second, they lack a mechanism which has the capacity to express transferable skill attributes in a structured parametric space. To this end, we propose SkillNet, which models skill attributes in a hierarchical way and regulates compositional model structure with transferable skill attributes. SkillNet exploits motion code and VerbNet Framework to explicitly model similarities of skills on mechanical properties and semantic roles, and organizes skills in a hierarchical way. Based on this hierarchy, SkillNet leverages the scalability of the mixture-of-experts (MoE) mechanism and develops skill embeddings as soft constraints to enable compositional generalization via similar expert activations on similar skills. On zero-shot and few-shot transfer experiments in simulators and real-world environments, SkillNet achieves an improvement of performance by 16.0% and 23.9%. Meanwhile, SkillNet achieves state-of-the-art performance on in-domain settings.

Robots trained from vision, language, and action data can perform many manipulation tasks, but they often struggle to transfer what they have learned to new behaviors. For example, a robot may learn to brush or wipe an object, yet still need many demonstrations to master a related behavior such as erasing or sweeping. This paper introduces SkillNet, a method that helps robots understand tasks through reusable skills rather than isolated task labels. SkillNet organizes skills in a hierarchy that captures both physical motion properties, such as contact, force, and tool use, and semantic meanings, such as what object is being acted on and why. This hierarchy allows the model to recognize when different behaviors share useful structure, even if their exact goals or motions are not identical. SkillNet then uses these skill representations to guide a modular neural network, so related behaviors can share parts of the model while still keeping their own execution patterns. This makes it easier for robots to transfer knowledge from known skills to unseen but related behaviors, and also to recombine familiar skills in new task sequences. Experiments in simulation and on real robots show that SkillNet improves few-shot transfer to new behaviors, zero-shot generalization to new task compositions, and standard task performance. Overall, the work suggests that explicitly modeling skill relationships can make robot learning more flexible, interpretable, and data-efficient.