AD-BTS: Adaptive Dual-Branch Token Sparsification via Spatial Information Density

High-resolution visual encoders in multimodal large language models (MLLMs) substantially improve fine-grained perception, yet incur prohibitive computational costs.Existing token pruning methods are effective on natural images but struggle with spatially sparse structured inputs (e.g., charts), where critical high-frequency information is sparse, localized, and structurally essential. To address this challenge, we propose Adaptive Dual-Branch Token Sparsification (AD-BTS), a density-aware framework that dynamically allocates computation according to input signal characteristics. Specifically, AD-BTS introduces a Gradient-based Routing Gate (GRG) that uses lightweight pixel-level gradient statistics to estimate structural flatness and guide routing. Then, AD-BTS activates either a Redundancy Selection Branch (RSB) for aggressive token pruning with a frozen encoder, or a Structural Fusion Branch (SFB) with conditional LoRA and context fusion to preserve sparse structural information.Extensive experiments on Qwen2.5-VL demonstrate that AD-BTS establishes a new Pareto frontier between efficiency and accuracy. Under extreme compression (20% token retention), AD-BTS outperforms the strongest baseline by 12.1% on ChartQA while achieving a 1.8× prefill speedup, effectively reconciling computational efficiency with structural robustness.

Modern AI models can analyze high-resolution images, but processing every tiny detail requires massive amounts of computing power and memory. To speed things up, current methods try to delete "unimportant" background pieces of an image. However, this often fails on documents or charts, where accidentally deleting a thin line or small number can ruin the AI's understanding. To fix this, we created a smart, adaptive system called AD-BTS. Instead of treating all images the same, our system quickly scans the picture's layout. If it sees a regular photo with lots of background space, it aggressively cuts out the useless parts to save time. But if it detects a complex chart or document, it automatically switches to a "careful" mode. In this mode, instead of just deleting pieces, it blends and adapts the information to ensure fragile details are preserved. Our approach significantly speeds up AI image processing and reduces memory usage without sacrificing accuracy. This makes it much easier and cheaper to run advanced, trustworthy AI systems, especially when analyzing complex, real-world visual data.