SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and a mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i) a Parallel LIF with Multistage Learnable Parameters (PLMP) neuron and a tailored P-STBP algorithm to stabilize training, (ii) a spike-driven attention approximation (SMSA) with a lightweight self-compensation branch, and (iii) a spiking mixture-of-experts (SMoE) module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines while reducing the estimated compute energy of the attention module by up to 97\% under a standard MAC/AC cost model.

Modern AI systems that connect images and language can support applications such as image search, automatic captioning, and visual understanding, but they often require substantial computation. This work studies how to make such systems more efficient by drawing inspiration from the way biological neurons communicate only when needed. We introduce SMM Transformer, a model that processes visual and textual information through sparse activity patterns and uses specialized components for images, language, and their interaction. Across several image and image-text tasks, the model achieves accuracy comparable to conventional neural networks while reducing estimated computational cost. These results suggest that brain-inspired sparse computation can be a promising direction for building more efficient multimodal AI systems.