MondegreensEval: A Phonetic Benchmark for Measuring Language-model Bias in Automatic Speech Recognition

Decoder-based automatic speech recognition (ASR) systems embed a strong language-model prior that can override degraded acoustic evidence. We introduce MondegreensEval, the first controlled benchmark designed to isolate this acoustic-prior override, distinct from the broader hallucination phenomena recently catalogued in speech foundation models. The benchmark consists of 190 mondegreen pairs (phonetically near-identical phrases with distinct meanings) synthesized via neural TTS and evaluated across five Whisper checkpoints and a wav2vec2 baseline. We define two complementary metrics: the Mondegreen Confusion Rate (MCR), a hard substitution measure, and the log-probability bias score, a soft preference measure via teacher-forced decoding. MCR rises monotonically with noise but not with model scale, peaking at 20.5% for whisper-medium at SNR = -5 dB. The soft signal reveals that 72–80% of pairs carry positive LM bias, phonetic similarity strongly predicts confusion (r = −0.36, p < 10e-6), and a condition on prev tokens ablation fails to weaken the bias, indicating the prior is encoded in attention weights rather than autoregressive conditioning. MondegreensEval reveals failure modes invisible to WER and complements existing hallucination metrics.