Token-Level Verification under Controlled Evaluation: Protocol Sensitivity Shapes Apparent Performance

Token-level signals such as entropy, log-probabilities, and confidence scores offer an appealing way to verify LLM reasoning correctness without additional model calls. Yet reported AUROC values vary widely, from near chance to above 0.80, and it remains unclear how much of this variation reflects the signal itself rather than the evaluation protocol. We show that protocol choices can change apparent verification performance more than the choice of token-level method. Under controlled within-problem leave-one-out evaluation on hard math problems, entropy-based verifiers that appear to achieve 0.80+ AUROC under less controlled settings fall to 0.60--0.75. Moreover, their excess AUROC over a permutation null is only 0.00--0.17. We further show that removing direction-agnostic scoring, an oracle-like evaluation convention that selects the better direction after observing labels, causes final-token entropy to drop from 0.72--0.75 AUROC to 0.47--0.48 under fixed-direction scoring. These results suggest that shallow token-level statistics are useful diagnostics but weak standalone verifiers. We release our controlled evaluation protocol to help future work distinguish genuine verification signal from protocol-dependent effects.