Positional Encoding for Spiking Transformers

Transformer-based Spiking Neural Networks (SNNs) have recently emerged as a promising paradigm to sequential modeling, combining the strong representational capabilities of Transformers with the sparse spike-driven computation of SNNs. Within such position-agnostic architectures, positional encoding is critical for injecting order information, allowing the model to distinguish token positions, capture sequential dependencies, and represent relative relationships among tokens. However, existing positional encoding methods for SNNs are largely inherited from ANNs and, in doing so, undermine the spike-driven computational properties that are central to spiking transformers. To address this limitation, we propose the Spiking Positional Encoding (SPE), a method designed specifically for Spiking Transformers, aimed at encoding relative positional information while preserving both spike-driven computation and the linear complexity of spiking self-attention. The core component of SPE is the Positional Encoding Leaky Integrate-and-Fire (PE-LIF) neuron, which incorporates position-dependent signals into neuronal thresholds and implicitly propagates this information through spike trains via continuous firing and membrane potential reset dynamics. Extensive experiments on thirteen NLP benchmarks demonstrate that SPE consistently outperforms existing SNN positional encoding methods, strengthens the sequence modeling capability, and improves energy efficiency without introducing additional trainable parameters. Code is available at https://github.com/CayleyZ/SPE.

Transformers are powerful models for language, but they usually require many calculations. Spiking neural networks offer a more energy-efficient alternative because they communicate with short binary signals, similar to brief pulses. Recent work combines these two ideas into spiking transformers, but these models still need a reliable way to know the order of words in a sentence. This order matters: “dog bites man” and “man bites dog” mean very different things. Existing ways to add word-order information were mostly designed for standard neural networks, and they can reduce the efficiency advantages of spiking models. We introduce Spiking Positional Encoding, a method that teaches spiking transformers about word positions by changing when neurons fire, rather than adding extra learned components. This lets the model represent how far words are from one another while keeping the efficient computation style of spiking attention. Across thirteen language benchmarks, the method improves sequence understanding and reduces energy use without adding trainable parameters. These results suggest a practical step toward language models that are both capable and more efficient for resource-limited devices.