Does the Optimal Hallucination Detector for Agentic Tool Calls Depend on Model Scale?
Abstract
When an LLM agent hallucinates a tool call, the call executes. No hedge, no confidence dip, no second pass. The hallucinated function name is syntactically valid, the arguments parse, the token probabilities look normal, and the next stage of the agent loop consumes the output without question. By the time a human reviews the trace, a malformed transaction has already cleared. Standard uncertainty signals are blind to this failure mode in exactly the way deployment requires them not to be: token log-probabilities and N=5 sampling consensus reach AUC 0.51 to 0.62 across the seven models we test, near chance, at five times the inference cost. Two recent lines of work, spectral analysis of the attention routing graph and hidden-state probes on token-role positions, both clear this bar from a single forward pass. The open question has been which to deploy. We answer it across seven models from 1B to 14B on identical data, splits, and labels, and find the optimal detector depends on scale non-monotonically. Our \textbf{Layerwise Multi-Metric} (LMM) probe on attention alone reaches AUC 0.957 at 3B, exceeding the hidden-state probe (0.947); at 8B, hallucination signal diffuses across 32 layers and trajectory aggregation (SpRich) takes the lead (0.888); at 14B, layer specialisation re-concentrates the signal and LMM recovers (0.917 vs. 0.882). Merging the two signal families never helps in our experiments. The deployment rule is to match the guardrail to depth and attention type, not to combine.