Universal Skeleton Understanding via Differentiable Rendering and MLLMs

Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet remain confined to their native modalities and cannot directly process structured, non-visual data such as human skeletons. Existing methods either compress skeleton dynamics into lossy feature vectors for text alignment, or quantize motion into discrete tokens that generalize poorly across heterogeneous skeleton formats. We present SkeletonLLM, which achieves universal skeleton understanding by translating arbitrary skeleton sequences into the MLLM's native visual modality. At its core is DrAction, a differentiable, format-agnostic renderer that converts skeletal kinematics into compact image sequences. Because the pipeline is end-to-end differentiable, MLLM gradients can directly guide the rendering to produce task-informative visual tokens. To further enhance reasoning capabilities, we introduce a cooperative training strategy: Causal Reasoning Distillation transfers structured, step-by-step reasoning from a teacher model, while Discriminative Finetuning sharpens decision boundaries between confusable actions. SkeletonLLM demonstrates strong generalization on diverse tasks including recognition, captioning, reasoning, and cross-format transfer---suggesting a viable path for applying MLLMs to non-native modalities. Code is available in the Supplementary Material.

Human movements are often recorded as skeleton data: a set of points showing where body joints are over time. This kind of data is useful for recognizing actions, describing motion, and understanding human behavior, but it is difficult for modern AI systems that are mainly designed to understand images and language. This paper introduces SkeletonLLM, a method that turns skeleton motion into visual representations that these AI systems can naturally process. Instead of forcing the model to learn a new technical format for every type of skeleton data, our method learns how to draw the motion in a way that highlights the important body parts and movement patterns. This makes it possible to use one system across different skeleton formats and different tasks, such as action recognition, motion description, and question answering about movement. Experiments show that this approach improves generalization across datasets and makes skeleton-based human activity understanding more flexible and easier to connect with powerful vision-language AI models.