Residual Stream Contrast: A Training-Free Counterfactual Listening Test for Whisper Hallucinations

The failure we target is simple: Whisper can emit a fluent transcript that remains internally plausible even when the audio is removed. Confidence alone does not test whether the transcript \emph{needed} the audio. We introduce \emph{Residual Stream Contrast} (RSC), a training-free white-box score that teacher-forces the same decoded transcript under real and null audio and measures whether the decoder residual trajectory changes. The claim is deliberately narrow: on the strict 325-row all-score causal subset, RSC is the best single detector, reaching .948 AUROC, .922 AUPRC, and .822 TPR at 5\% FPR. On 1,663 matched activation rows, RSC improves over AvgLogP by +.051 AUROC and +.186 TPR@5 with paired bootstrap intervals excluding zero. Composite diagnostics such as CLE and sparse out-of-fold witnesses are stronger ceilings, but they are not the headline; they support the same conclusion that residual evidence is load-bearing. A grounding-head atlas localizes the signal to a sparse set of cross-attention heads, led by L8H8 in Whisper-small.