Alethia: a Foundational Encoder for Voice Deepfakes

Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.

Mainstream audio and speech foundation models rely on masked token prediction for pretraining. However, we found that quantized/discrete tokens and using only the predictive objective lead to the loss of acoustic details and sub-optimal performance on voice deepfake tasks. We therefore propose a new pretraining method, consisting of masked latent reconstruction and spectrogram generation. The outcome Alethia demonstrates superior performance to Wav2vec, WavLM, Hubert across 5 deepfake tasks across 56 datasets.