SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition

Skeleton-based action recognition aims to understand human behaviors from body joint sequences and is especially challenging in the one-shot setting, where only a single labeled exemplar is available for each novel action. A key challenge is learning representations that capture the hierarchical and compositional structure of human motion while aligning effectively with high-level action semantics under extreme data scarcity. Existing approaches, largely based on Euclidean embeddings and low-level motion cues, struggle to model the tree-like organization of skeleton data, limiting cross-modal alignment and generalization to unseen action categories. We propose SkelHCC, a unified skeleton hyperbolic CLIP-driven cache adaptation framework for one-shot skeleton-based action recognition. SkelHCC introduces an Explicitly Hierarchical Hyperbolic CLIP (EH-HCLIP) module that embeds skeleton sequences and action language into a shared hyperbolic space. By leveraging the negative curvature and exponential volume growth of hyperbolic geometry, EH-HCLIP naturally encodes the joint–part–body hierarchy of human anatomy and yields structurally consistent cross-modal representations. To support efficient one-shot adaptation, SkelHCC further integrates a training-free LLM-guided Multi-granularity Voting Cache (LMV-Cache) for context-aware inference. Experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD II demonstrate that SkelHCC consistently outperforms state-of-the-art methods. Code will be available at https://github.com/lya19971103/SkelHCC

Recognizing human actions from skeleton data means teaching computers to understand what a person is doing by looking only at the movement of body joints, such as hands, arms, legs, and torso. This is useful for applications such as healthcare monitoring, sports analysis, human-computer interaction, and intelligent surveillance. However, the task becomes much harder when the computer is given only one example of each new action, because it must learn to recognize unfamiliar movements from extremely limited information. Our work addresses this challenge by making better use of the natural structure of the human body and the meaning of action names. Human motion is not just a collection of separate joint movements: joints form body parts, and body parts work together to form a complete action. For example, “putting on headphones” mainly involves the hands, arms, and head, while “kicking” depends more on the legs. We design a method called SkelHCC that helps the model understand actions at these different body levels and connect them with language descriptions of actions. SkelHCC also uses large language models to provide useful prior knowledge about which body parts are likely to be important for each action, without requiring extra training during testing. Experiments on three widely used skeleton action recognition benchmarks show that our method recognizes new actions more accurately than previous approaches.