Augmenting Video Models with Pose Tokens for Human-Centric Understanding

Video-language models often encode each frame as hundreds of dense patch tokens, which is token-expensive and sometimes redundant. We show that explicit skeletal pose is a compact signal which is not only a useful complement to the patch stream on human-centric video tasks, but can serve as a near-complete substitute for it, recovering much of the model's motion-task accuracy at under 10\% of the input-token cost. Concretely, we extract per-frame MediaPipe body, face, and hand landmarks, project them through a small two-layer MLP into a handful of language-model tokens, and append these pose tokens to Qwen3.5-9B's hidden states just before the LM head. The projector adds only 275M trainable parameters (~3\% of the Qwen3.5-9B model's parameters) and leaves the rest of the pipeline untouched. On a mixed Kinetics-400 + HMDB51 + QVID benchmark, pose tokens raise strict accuracy from 71.6\% to 76.3\% over a no-pose fine-tuning baseline, with the largest gains on motion-discriminative action recognition. Furthermore, we show that this method transfers well to two clinical autism-behavior datasets where subtle motor cues dominate. In a no-video ablation, the model answers using only pose tokens (no video tokens) and reaches 44.5\% strict accuracy on the mixed benchmark, nearly double the 25.5\% text-only floor, while consuming under 10\% of the input tokens of a video inference of the same fps. Our experiments indicate pose is an efficient modality for human-centric video understanding.