Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformer

Real-time and accurate spatial audio generation is pivotal for delivering an immersive experience. However, existing spatial audio synthesis technologies are often encumbered by a tradeoff between generation quality and high inference latency, as well as difficulty in capturing precise spatial information from multimodal inputs. To address these challenges, we propose SwanSphere, a unified streaming framework for high-fidelity spatial audio generation from panoramic videos and text prompts. SwanSphere mainly makes the following contributions: 1) We introduce a causal autoregressive diffusion transformer architecture that enables streaming high-quality spatial audio generation. 2) We design a Spatial Video–Audio Contrastive (SVAC) learning strategy to align the video encoder with the acoustic domain, and further employ a multi-objective online direct preference optimization~(ODPO) scheme, resulting in strong spatial perception and robust multimodal spatial audio synthesis. 3) To alleviate the current scarcity of spatial audio datasets, we also develop an automated annotation pipeline for generating detailed spatial captions. Experimental results demonstrate that SwanSphere achieves superior performance in both video-to-spatial and text-to-spatial audio generation tasks. Codes can be found at: https://github.com/MM-Speech/SwanSphere

Imagine wearing a pair of smart glasses or stepping into a virtual reality world: when you turn your head, sound not only comes from the direction of the screen but also realistically from your left, right, or even behind you. This is "spatial audio" — the key to an immersive experience. However, existing spatial audio generation technologies face three major bottlenecks: either they produce high-quality sound but suffer from huge latency and cannot respond in real time, or they are fast but deliver rough, unnatural sound quality. Even worse, it is very difficult to accurately extract spatial information about where a sound source should be located from panoramic videos and text descriptions. To address these challenges, we developed a framework called SwanSphere, which generates high-fidelity spatial audio from panoramic videos and text prompts. SwanSphere has three main innovations: A neural network architecture that decodes sequentially during generation (similar to continuously synthesizing matching sound as new video frames are seen), enabling real‑time, high‑quality spatial audio generation. A "spatial video‑audio contrastive learning" strategy that allows the video encoder to understand where a sound is located in space. We further introduced a multi‑objective online direct preference optimization scheme, which significantly enhances the model's ability to perceive the spatial acoustic field while robustly fusing information from both video and text modalities. To solve the problem of scarce spatial audio datasets, we built an automated annotation pipeline that generates detailed text descriptions for spatial audio. Experiments show that SwanSphere outperforms existing methods in spatial audio generation tasks.