PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization

Unifying the complementary strengths of diverse Vision Foundation Models (VFMs) into a single efficient model is highly desirable but challenged by the negative transfer inherent in monolithic distillation. To address these feature conflicts, we introduce \textbf{PRISM}, a novel dual-stream Mixture-of-Experts (MoE) framework that synergizes VFMs via modular specialization. We propose a two-stage paradigm: (1) expertise deconstruction, where a teacher-conditional router guides experts to specialize in distinct representational subspaces to mitigate interference, followed by (2) dynamic recomposition, where the router learns to assemble these experts into tailored computational pathways for downstream tasks. Experiments on PASCAL-Context and NYUD-v2 show that \textbf{PRISM} establishes a new state of the art, validating that sparse, emergent specialization is a scalable approach for integrating diverse visual knowledge.

Vision foundation models are powerful, but different models often focus on different kinds of visual information, such as object meaning, shape, boundaries, or fine details. Using many of these large models together is costly, while simply compressing them into one model can cause their knowledge to interfere. We introduce PRISM, a method that combines several vision foundation models into one efficient model. PRISM lets different parts of the model specialize in different visual skills, while still sharing information that is useful across models. It first learns to separate the knowledge from different models, then learns how to recombine that knowledge for downstream vision tasks. Experiments show that PRISM achieves strong performance across multiple vision tasks, suggesting that specialization and recombination are effective for building compact general-purpose vision systems.