Evaluating Open-Weight Audio Models for Privacy Verification in Clinical Speech Redaction

Clinical research in psychiatry increasingly relies on spoken language data to identify acoustic and linguistic biomarkers. Protecting participant privacy when individuals withdraw consent poses significant technical challenges. Voiceprint is a HIPAA-protected identifier, and removing a target speaker from multi-speaker clinical recordings requires automated verification at scale. We evaluate open-weight audio models for verifying completeness of speaker redaction in clinical recordings, framing the task as detection of residual therapist speech in audio from which the clinician track has been removed. Our comparison includes multimodal audio models (Nemotron-3-Nano-Omni, Phi-4-Multimodal), specialized speaker verification architectures (ECAPA-TDNN, WavLM-ECAPA-TDNN), and a text re-analysis baseline combining Whisper transcription with an instruction-tuned language model (Gemma-4). Evaluated across 14 annotated clinical sessions, the text baseline achieves the highest recall (0.978) but the lowest precision (0.316, F1=0.477), flagging large numbers of patient turns as potential leakage. Speaker verification models achieve the best overall F1, with WavLM-ECAPA-TDNN reaching F1=0.564 (precision=0.412, recall=0.891), though they show elevated false-positive rates in recordings with high intra-speaker voice variability. Among multimodal models, Phi-4-Multimodal with transcript augmentation performs best (F1=0.540, precision=0.384, recall=0.913), while Nemotron-3-Nano-Omni underperforms across both audio-only and audio+transcript conditions, with transcript augmentation degrading rather than improving performance. Our findings provide empirical benchmarks for privacy-preserving speech data stewardship in clinical research and release an open-source pipeline for prospective deployment.