MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs

Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs). However, existing approaches often struggle with fine-grained visual grounding due to semantic entanglement in visual patch representations, where individual patches blend multiple distinct visual elements, making it difficult for models to focus on instruction-relevant details. To address this challenge, we propose MoDA (Modulation Adapter), a lightweight module that enhances visual grounding through instruction-guided channel-wise modulation. Unlike token-level methods such as Q-Former that perform additive feature selection, MoDA operates at the channel level through multiplicative modulation on already-aligned features, enabling fine-grained control over which embedding dimensions are relevant for each instruction. Following the standard LLaVA training protocol, MoDA applies cross-attention between language instructions and pre-aligned visual features, generating dynamic modulation masks without architectural modifications or additional supervision. We evaluate MoDA across 12 benchmarks spanning visual question answering, vision-centric reasoning, and hallucination detection, including recent 2024 benchmarks (MMVP, CV-Bench, MMStar, RealWorldQA), on three distinct MLLM architectures: LLaVA-1.5, LLaVA-MoRE (2025), and Qwen3-VL (2025). MoDA delivers consistent gains across all three families, with +12.0 points on MMVP for the LLaVA-1.5 family and +4.8 points on ScienceQA for the LLaVA-MoRE family, and +4.9 ScienceQA, +4.1 RealWorldQA, and +3.8 GQA on Qwen3-VL, confirming that the gains generalize beyond CLIP-based encoders with minimal overhead (<1% FLOPs). Code is available at https://github.com/waybarrios/MoDA.

AI assistants that look at pictures and answer questions about them have gotten very good in the last few years. But they still trip up on small details. Ask what color something is in a corner of the picture, or whether one object is touching another, and the answer is often wrong, even when the rest sounds fine. The reason is in how the model sees the image. It cuts the picture into a grid of small squares and compresses each square into a short summary. Most squares end up holding several things at once: the edge of one object, the corner of another, a bit of background. Once that gets mashed together, the model has a hard time pulling it apart, and ends up reasoning over a blurry mix instead of the actual detail. That is also where many of its hallucinations come from. We added a small piece to the model that helps it focus on the parts of the image that matter for the current question. Before the model answers, this piece reads the question and dials some parts of that summary up and others down. Nothing else changes, and nothing is retrained from scratch. We tested it on 12 standard benchmarks and on three families of these models. Accuracy went up in every case, with the biggest gains on detail-heavy tasks and on tests built to catch the model making things up. The extra cost is under one percent. Because the piece plugs in without changing anything else, it is a practical way to make these systems more reliable for things like apps that describe images for blind users, tutors that help students read diagrams, and any setting where the answer really needs to match the picture.