From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection

With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations.

Recent tools can create highly realistic videos of people speaking or singing, which makes it increasingly important to detect whether a video is real or fake. Many existing detection methods are designed for talking videos. They often check whether the sound and facial movements match well. However, singing is different from ordinary speech. Singers may hold notes longer, move their mouths more expressively, and follow rhythm and melody rather than regular speaking patterns. As a result, methods that work well for talking videos may fail on singing videos. In this work, we study this new challenge by building a dataset of fake singing head videos. This dataset helps researchers test whether current detection methods can still work when the video changes from talking to singing. We also propose a new detection approach that combines information from the face, the audio, and text descriptions of visual authenticity cues. Instead of relying only on whether the mouth and voice are synchronized, our method also learns more general signs of whether a face looks natural and consistent. Experiments show that our method works better than existing methods on both talking and singing videos and remains reliable under different kinds of video changes.