Towards Trustworthy Video Anomaly Understanding: A Class-Guided Chain-of-Evaluation Metric and An Anomaly-focused Meta-Benchmark

The trustworthiness of evaluation is critical to reliable model comparison and deployment in Video Anomaly Understanding (VAU). However, existing metrics are sensitive to expression styles and normal content, and this field lacks a diagnostic benchmark to validate metric validity and robustness. To bridge this gap, we propose: (1) a Class-Guided Chain-of-Evaluation (CG-CoE) metric, which structures assessment by extracting anomalous events and matching them under a class-specific semantic tolerance boundary, thereby decoupling anomaly semantics from descriptive style; and (2) an anomaly-focused meta-evaluation benchmark with two subsets: Anomalous Event-level Annotations (AEA) for measuring the validity of reflecting VAU models’ anomaly understanding ability and Controlled Variant Pairs (CVP) with fixed anomalies for quantifying robustness to stylistic perturbations. Extensive experiments demonstrate that CG-CoE achieves SOTA validity and robustness.

Video anomaly understanding systems are used to identify unusual or risky events in videos, such as accidents, fights, or other abnormal situations. To compare different systems fairly, we need reliable ways to judge whether their answers truly understand what happened. However, existing evaluation methods can be easily affected by how the answer is written, or by normal background details that are not actually related to the anomaly. In this paper, we propose a new evaluation method that focuses directly on the abnormal events in a video. Instead of judging the whole response mainly by wording, our method first identifies the key anomalous events, then checks whether these events match the correct anomaly category. This makes the evaluation less sensitive to writing style and irrelevant normal content. We also build a new benchmark to test whether evaluation methods are both accurate and stable. One part compares metric scores with human judgments of anomalous events, while the other checks whether scores remain consistent when the same anomaly is described in different styles. Experiments show that our method better reflects human evaluation and is more robust to changes in expression. This provides a more trustworthy way to evaluate video anomaly understanding models.