Rethinking Attention in Spiking Transformers: Overcoming Density Bias with Set Similarity

Recent Spiking Transformer models have explored a variety of attention mechanisms beyond standard dot-product formulations. However, many existing similarity-based spiking attention formulations remain inherently sensitive to firing density, causing neurons with high spike rates to dominate attention scores regardless of semantic relevance. This density bias is particularly problematic in event-driven spiking representations, where sparse spike patterns often carry critical information. To address this limitation, we rethink spiking attention from a set-theoretic perspective. We propose DiceFormer, a novel Spiking Transformer architecture driven by Spike Dice Attention (SDA). Unlike traditional approaches, SDA replaces density-sensitive measures with a set similarity function derived from the Dice coefficient. By explicitly normalizing for firing density, SDA focuses on spike co-occurrence rather than high firing rates. We primarily evaluate DiceFormer on the challenging audio domain, where spike sparsity varies substantially across inputs. On AudioSet-20k, DiceFormer achieves a SOTA mAP of 0.161 with 54.3M parameters, outperforming prior SNN-based approaches and substantially narrowing the performance gap with ANN-based models. We also introduce Lin-SDA, a linearized version for computation efficiency, while achieving performance comparable to SDA. Beyond audio, we evaluate the effectiveness of SDA on CIFAR-100 to verify its applicability to the vision domain.

Modern AI models can recognize sounds and images very well, but they often require a large amount of computation and energy. Brain-inspired spiking neural networks offer a more energy-efficient alternative because they process information as short on/off signals. However, these models can struggle when deciding which signals are truly important. In particular, signals that fire more often can receive too much attention, even when they are not the most meaningful. This paper introduces DiceFormer, a new spiking model designed mainly for audio recognition. DiceFormer uses a fairer way to compare spike patterns, focusing on how well signals match rather than how frequently they fire. This helps the model better handle sparse and uneven audio signals. Experiments show that DiceFormer improves the performance of spiking audio models while keeping energy use low. We also introduce a simpler version designed for efficient computation, and we test the method on an image dataset as an initial check of its broader usefulness.