Zero-Shot Text-to-Motion Evaluation using Video Language Models

Text-to-motion (T2M) generation has become a fundamental task, yet existing evaluation metrics often fail to capture whether a generated motion semantically matches its text description. We propose VeMo, a zero-shot evaluation framework that renders generated human motions into videos and uses pretrained video-language models (VLMs) to assess text-motion alignment. Instead of training an evaluator on scarce motion-specific labels, VeMo transfers the semantic reasoning ability of VLMs to T2M evaluation through normalized likelihood-based scoring. To reduce the effect of 3D-to-2D projection ambiguity, we introduce an entropy-driven uncertainty analysis for identifying reliable rendered views. To address the lack of rigorous standards in the field, we further contribute a test-only and human-annotated meta-evaluation benchmark, covering motions generated by multiple representative T2M models. Extensive experiments show that VeMo correlates better with human judgments than existing reference-based and reference-free metrics. Additional analyses on view selection, rendering protocols, textual prompt robustness, and computational trade-offs characterize both the promise and limitations of VLM-based T2M evaluation.

Text-to-motion systems can turn sentences such as “a person waves hello” into animated 3D human movements, but it is still difficult to automatically judge whether the generated motion truly matches the text. Existing evaluation scores often compare a generated motion with one reference motion, even though many different motions may be correct, or rely on scarce human-labeled motion data. We propose VeMo, a zero-shot evaluator that judges motion more like a person watching a video: it renders the generated 3D motion into a short video and uses a pretrained video-language AI model to decide whether the action matches the description. Since a single camera view can hide important body movements, VeMo estimates when the model is uncertain and selects more reliable views for evaluation. We also build a human-annotated benchmark to fairly compare evaluation methods. Experiments show that VeMo agrees better with human judgments than existing automatic metrics, suggesting a practical way to evaluate text-to-motion models without training a new evaluator from scratch.