Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts

We propose S3 (Specialization, Selection, Sparsification), a framework that rethinks multimodal learning through a structural perspective. Instead of encoding all signals into a fixed embedding, S3 decomposes multimodal inputs into semantic experts and selectively routes them for each task. Specialization forms concept-level experts in a shared latent space, Selection adapts routing for task-specific needs, and Sparsification prunes low-utility paths to yield compact, information-minimal representations. Across four MultiBench benchmarks, S3 improves accuracy and exhibits consistent sparsity-performance dynamics, exhibiting a reverse U-shaped trend, with performance peaking at intermediate sparsity. These results suggest that structuring multimodal representations as selectable semantic components provides a practical and principled alternative to contrastive learning or InfoMax-driven approaches.

Our senses constantly receive many different types of information at once, such as images, sounds, and language. Modern AI systems try to learn from these multiple sources together, but existing approaches often compress all information into a single representation. As a result, useful and irrelevant information can become mixed together, making the learned representations harder to interpret and less efficient. In this work, we propose S3, a new framework that organizes multimodal information into specialized semantic components. Instead of treating all information equally, the model learns to select only the components that are most useful for a given task and gradually remove unnecessary ones. This allows the system to build more compact and task-focused representations. We evaluate S3 on several multimodal benchmarks and show that it consistently improves performance while using information more efficiently. Our findings suggest that structuring multimodal representations into selectable semantic building blocks may help future AI systems become more reliable, efficient, and adaptable.