EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs

Audio-Visual Large Language Models (AV-LLMs) face prohibitive computational costs of processing massive, redundant audio-visual tokens. Existing unimodal compression techniques fail to capture the heterogeneous and mutually influential information density of joint audio-visual signals. Furthermore, we identify a fundamental and overlooked theoretical bottleneck in sparse token reduction: positional aliasing. We demonstrate that aggressive sparse sampling on standard position-encoded sequences violates the Nyquist limit relative to the effective token interval, causing phase-wrapping collisions that corrupt temporal monotonicity. To address this, we introduce EchoingPixels, a framework for aliasing-resistant joint token reduction. Our Cross-Modal Semantic Sieve performs extractive selection on the synergistic audio-visual stream, dynamically allocating budgets based on joint-modality saliency rather than fixed per-modality ratios. To resolve positional aliasing, we derive Sync-RoPE, a spectral low-pass filter for Rotary Positional Embeddings that adapts encoding bandwidth to the sparse sampling rate, preserving monotonic temporal relationships in the reduced stream. Experiments show that EchoingPixels achieves performance comparable to full models using only 5-20% of original tokens, validating theoretically grounded sparse learning as a robust solution for efficient AV-LLMs. Code is available at https://github.com/CharlesGong12/EchoingPixels.

Advanced AI models that can understand both video and audio are incredibly powerful, but they require massive computational power to analyze every single frame of video and snippet of sound. To make these models faster and more efficient, researchers try to "compress" the data by throwing away redundant information. However, doing this creates two major problems. First, current methods compress sight and sound separately, missing the crucial context of how they interact. Second, aggressively skipping data chunks scrambles the AI's "sense of time," making it confuse the order of events when distant frames are mistakenly perceived as adjacent—a critical flaw we identify as "positional aliasing." To solve this, we introduce EchoingPixels. Instead of looking at audio and video in isolation, our method acts as a smart cross-modal filter that evaluates them together. It dynamically shifts the computing budget to whichever sense is most informative at any given moment (e.g., focusing on audio when the video is a static background). Furthermore, we introduce a mathematical fix that adjusts the AI's internal clock, preventing it from losing track of time when large gaps of data are removed. Ultimately, our method allows these heavy AI models to run almost 3 times faster and use significantly less memory, maintaining top-tier performance while processing only 5% to 20% of the original data.