Mechanistic Insights into Audio-Language Models for Impaired Speech

Frozen audio-language models can transcribe dysarthric speech worse when prompted with clinical context than with audio alone. We study this failure mode in Gemma-4-E2B on the Speech Accessibility Project (SAP) rated subset and trace it to the residual stream. To test whether the degradation is encoded in the model's internal state rather than in the audio representation itself, Matched Residual Patching (MRP) replaces the final-input-token residual at a middle decoder layer under the clinical prompt with the corresponding audio-only residual and closes 57-60% of the Word Error Rate (WER) gap bidirectionally. The repairable component lies mostly outside a sparse prompt-identity axis: ~27 residual coordinates identify the prompt at 99.8% accuracy, but patching them recovers only 1-10% of the gap, while their 1,516-coordinate complement carries 40-59%. We then introduce Self-Contrastive Residual Alignment (SCRA), a one-pass inference-time edit that learns a low-rank linear corrector for the prompt-induced residual shift. At rank 128, SCRA matches the MRP ceiling, closes 60% of the aggregate gap, and slightly but significantly improves over the audio-only baseline on difficult samples.