Efficient Transformer Attention for SNNs via Hadamard Simplification

Spiking Neural Networks (SNNs) enable low-power, event-driven computation, but Transformer-based SNNs remain difficult to deploy on neuromorphic hardware due to dense operations and communication overhead. We propose two simplified attention mechanisms, \textbf{Simplified Spiking Attention (SSA)} and \textbf{Ultra-Simplified Spiking Attention (USSA)}, which replace matrix multiplication with Hadamard products and eliminate hardware-unfriendly components such as multi-head attention and scaling. We show that consecutive masking is redundant and analyze a spiking-order effect in which early spiking contributes more temporal information to attention modulation. On CIFAR-10, CIFAR-100, and DVS-Gesture, SSA achieves accuracies of 96.38\%, 79.45\%, and 97.56\%, respectively, while reducing computational complexity from $\mathcal{O}(N^2D)$ to $\mathcal{O}(ND)$ and communication complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(ND)$. USSA further reduces communication complexity to $\mathcal{O}(N)$ with only marginal accuracy degradation. On ImageNet-1K, SSA and USSA achieve 76.91\% and 77.27\% accuracy, respectively, demonstrating scalability to large-scale classification.

Artificial intelligence (AI) is becoming more powerful, but many AI models consume too much energy to run on small devices like phones or robots. This is a challenge for applications that need to work locally on battery power. Inspired by the brain, a class of AI models called "spiking neural networks" uses short electrical pulses to process information, making them naturally energy-efficient. However, the most accurate AI models today, called Transformers, are too complex to run efficiently on this brain-inspired hardware. In this paper, we simplify the Transformer design to work better with spiking neural networks. We replace heavy computations with much lighter operations, reducing the amount of data that needs to move around inside the chip. Our new design matches the accuracy of existing models while using significantly less energy, making it more suitable for real-world applications on low-power devices.