Query-Based Asymmetric Modeling with Decoupled Input–Output Rates for Speech Restoration

Speech restoration aims to recover clean speech from degraded recordings affected by noise, reverberation, bandwidth reduction, or other distortions, where input and output sampling rates may differ. Existing approaches typically assume matched input--output rates and apply redundant resampling, limiting native multi-rate processing. We formulate this gap as the extended sampling-frequency-independent (xSFI) setting, where a model must operate under decoupled input--output rates, and propose TF-Restormer, a query-based xSFI modeling framework. The model encodes only the observed input band and synthesizes the unobserved high-frequency band through extension queries with band-partitioned cross-attention, yielding an asymmetric encoder--decoder that allocates capacity to analysis while keeping synthesis lightweight. Trained with a perceptual loss, a scaled log-spectral loss, and adversarial supervision via an SFI-STFT discriminator, TF-Restormer attains balanced fidelity--perceptual quality as a single unified model, without redundant resampling across denoising, dereverberation, bandwidth extension, and combined-distortion benchmarks under multiple sampling rates.

Almost every recording of human speech is imperfect in some way. On a phone or video call, in a voice message, or in an old archived clip, the sound is usually damaged along the way: background noise creeps in, rooms add echo, and fine acoustic detail is thrown away whenever audio is compressed or captured by a low-quality microphone. These problems often happen at the same time, making speech harder to understand and less pleasant to hear. The goal of speech restoration is to undo this damage and recover clean, natural-sounding speech. A subtle but important part of the problem is that recordings arrive at many different levels of audio quality, and the quality we want at the end is not always the quality we started with — a low-quality telephone call, for instance, may need to be rebuilt into rich, full-bandwidth sound. Most existing systems sidestep this by first forcing every recording into one fixed quality level before processing it. This shortcut wastes computation, discards useful information, and ties each system to a single setting, so different situations often demand different specialized systems. We developed TF-Restormer, a single system that takes audio at whatever quality it was recorded in and rebuilds it at whatever quality is desired, without that wasteful conversion step. The key idea is to stay focused: the system carefully analyzes only the sound that is actually present in the recording, instead of spending effort on parts that carry no real information, and then intelligently fills in the missing high-quality detail that was never captured in the first place. Because of this design, one model handles a wide range of damage types and quality levels at once, producing speech that stays faithful to the original signal while also sounding clear and natural to the human ear. This makes high-quality speech restoration more flexible and practical for everyday uses such as phone and video calls, voice messaging, hearing aids, voice assistants, and the clean-up and preservation of old audio archives.