PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs

Current multimodal large language models (LLMs) process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Conversely, existing spatial audio models are constrained to fixed microphone geometries, preventing their deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is inherently agnostic to microphone geometry. By taking raw multichannel audio and microphone coordinates as inputs, PhaseCoder performs accurate localization and produces robust spatial embeddings. We demonstrate that the Gemma 3n LLM can be fine-tuned to process and reason over the "Spatial Audio Tokens" produced by our encoder. PhaseCoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

For humans and many other animals, knowing exactly where a sound comes from is crucial for understanding the world. This spatial awareness is equally important for machine hearing in technologies like autonomous robots and smart home devices. Just as humans rely on the slight time delay between a sound reaching our left and right ears, devices use arrays of microphones to measure these tiny delays and pinpoint a sound's origin. However, getting this system to work in the real world is quite challenging. Because every device has a unique microphone array, traditional algorithms must be custom-built for each specific piece of hardware. As a result, most modern voice assistants (like Alexa or Google Home) can successfully transcribe what we say, but they largely remain blind to where the sound is actually coming from. In this work, we developed a spatial audio model—PhaseCoder—that can locate speech regardless of the device's microphone geometry. We achieved this by training our model on millions of randomized microphone layouts. Our main contribution is showing that we can transform these raw, variable hardware signals into a universal, microphone-agnostic representation—which we call "Spatial Audio Tokens." We then demonstrate how an existing Large Language Model (LLM) that already understands speech can be augmented with these tokens. This creates a new AI system that not only understands the words spoken but also understands and can reason about the spatial context of the speech. Finally, we prove that this approach generalizes across different physical devices and environments by validating its performance on commercial microphone arrays.