Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatiotemporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.

OneTrackerV2 is a unified AI system for visual object tracking that can work with many different types of inputs, including standard videos, thermal images, depth sensors, event cameras, and language descriptions. Unlike previous methods that require separate models or extra retraining for different sensor types, OneTrackerV2 uses a single shared model trained once for all tasks. The system introduces new mechanisms to better combine information from different sensors and to separate motion understanding from modality-specific features. This makes the model both more accurate and more robust when some sensor inputs are missing. Experiments on 12 benchmark datasets show that OneTrackerV2 achieves state-of-the-art performance across multiple tracking tasks.