Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditionally synthesizing audio from neurally generated invertible TF features still struggle to produce audio at satisfying quality. In this article, focusing on the short-time Fourier transform, we discuss the challenges that arise in audio synthesis based on generated invertible TF features and how to overcome them. We demonstrate the potential of deliberate generative TF modeling by training a generative adversarial network (GAN) on short-time Fourier features. We show that by applying our guidelines, our TF-based network was able to outperform a state-of-the-art GAN generating waveforms directly, despite the similar architecture in the two networks.
Andres Marafioti (Austrian Academy of Sciences)
Andrés Marafioti studied audio engineering at the National University of Tres de Febrero (UNTREF), Buenos Aires, Argentina. He graduated with the Master's thesis entitled "Automatic identification of acoustical musical instruments", in which he applied machine-learning techniques to classify 21 different musical instruments. After, he worked for two years as a software developer for FaroLatino, the leading entertainment & music multiplatform network in Latin America. Since September 2017 he is part of the Acoustic Research Institute's workgroup "Mathematics and Signal Processing in Acoustics", where he works on sound generation and restoration for the project "Modern methods for the restoration of lost information in digital signals" (MERLIN).
Nathanaël Perraudin (Swiss Data Science Center)
Nicki Holighaus (Acoustics Research Institute)
Piotr Majdak (Acoustics Research Institute)
Related Events (a corresponding poster, oral, or spotlight)
2019 Oral: Adversarial Generation of Time-Frequency Features with application in audio synthesis »
Tue Jun 11th 02:25 -- 02:30 PM Room Grand Ballroom