DenseMLLM: Standard Multimodal LLMs for Dense Prediction

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in high-level visual understanding. However, extending these models to fine-grained dense prediction tasks, such as semantic segmentation and depth estimation, typically necessitates the incorporation of complex, task-specific decoders and other customizations. This architectural fragmentation increases model complexity and deviates from the generalist design of MLLMs, ultimately limiting their practicality. In this work, we challenge this paradigm by accommodating standard MLLMs to perform dense predictions without requiring additional task-specific decoders. The proposed model is called DenseMLLM, grounded in the standard architecture with a novel vision token supervision strategy for multiple labels and tasks. Despite its minimalist design, our model achieves highly competitive performance across a wide range of dense prediction and vision-language benchmarks, demonstrating that a standard, general-purpose MLLM can effectively support dense perception without architectural specialization. This project is available at github.com/Eli-YiLi/DenseMLLM.

Multimodal Large Language Models (MLLMs) are powerful at high-level visual understanding, yet extending them to fine-grained dense prediction tasks like semantic segmentation and depth estimation typically forces researchers to attach complex task-specific decoders. This leads to architectural fragmentation and moves away from the generalist design of MLLMs. We challenge this pattern with DenseMLLM. Preserving the standard MLLM architecture, we introduce a novel vision token supervision strategy for multiple labels and tasks, eliminating the need for any task-specific decoders. This minimalist design achieves highly competitive performance across a broad set of dense prediction and vision-language benchmarks. Our results demonstrate that a single general-purpose MLLM can effectively support dense perception in the standard architecture.