V-LynX: Token Interface Alignment for Video+X LLMs

This study introduces an intriguing phenomenon in Video LLMs: rather than merely translating frames into textual embeddings, Video LLMs establish a continuous manifold, token interface, allowing visual tokens to operate as standalone entities within the architecture. Exploiting this discovery, we propose V-LynX, a scalable framework that integrates novel modalities into Video LLMs by repurposing the internalized interface. Departing from conventional paradigms that necessitate heavy modality-specific encoders or paired supervision, V-LynX employs a lightweight auxiliary pathway in parallel with the frozen vision encoder. Our method integrates new sensory inputs with intrinsic video priors by aligning both attention responses and statistical distributions using unpaired unimodal data sets. This ensures manifold compatibility while preserving the integrity of the Video LLMs. Extensive benchmarks demonstrate that V-LynX achieves SOTA and efficiency across audio-visual QA, 3D reasoning, high-frame-rate, and multi-view video understanding. The code is available at https://github.com/park-jungin/lynx.

This study shows that AI models designed to understand videos can also be adapted to process other types of information, such as audio and 3D data. Based on this finding, we propose V-LynX, a lightweight method that adds new capabilities to an existing video AI model without rebuilding or heavily retraining it. Experiments show that V-LynX efficiently improves performance across tasks involving audio, 3D scenes, fast-moving videos, and videos captured from multiple viewpoints.