Task-Aware Mechanism: Hybrid MoE Vision Tower Towards Holistic Video Understanding

Should an LVLM use the same video processing pipeline for \emph{comprehending the main idea of a 2-hour movie} and \emph{counting the birds appearing in a 15-second clip}? We present Task-Aware Mechanism (TAM), a hybrid-gated Mixture-of-Experts (MoE) vision tower that adapts frame count and resolution to the user query and video length. TAM introduces a compact 0.1B text-only \emph{Inductor} trained on our TA-116K dataset to infer task types, enabling task-aware visual budgeting and routing: a soft-gated MoE vision encoder for stability, and hard-gated resolution-specific projectors/pipelines for efficient specialization. The explicit budget also separates task classification from downstream visual-language reasoning, making routing decisions easier to inspect and helping diagnose when fixed visual pipelines under-sample temporal or spatial evidence. Built on Qwen2-7B, TallVA-8B-A7B achieves state-of-the-art performance among models with comparable LLMs on most video benchmarks and remains competitive against stronger-LLM baselines, showing that task-aware visual budgeting makes video understanding more holistic.

Should a video AI system process a two-hour movie summary question the same way it processes a short question about counting small objects? This paper proposes Task-Aware Mechanism (TAM), a method that lets a video-language model adjust how much visual information it uses based on the user's question and the video length. TAM uses a small text-only helper called the Inductor to first understand what kind of visual evidence the question needs. It then chooses a suitable visual budget, such as using more frames for long or time-related questions, or using higher detail for questions about small objects, text, or fine visual clues. This makes the model's choices easier to inspect, because the system explicitly decides how much visual information to collect before answering. Our experiments show that the resulting model, TallVA, performs strongly across video understanding benchmarks while remaining competitive with models that use larger language backbones. Overall, TAM shows that matching visual processing to the task can make video AI systems more flexible, interpretable, and effective.