SpikingLM: Towards Fully Spiking Language Model

Leveraging event-driven computation mechanism, Spiking Neural Networks (SNNs) have emerged as a representative paradigm for energy-efficient edge intelligence. However, extending SNNs to modern deep language models still faces two fundamental challenges. First, dead neurons in deep SNNs lead to degraded gradients, limiting the training effectiveness of spiking language models. Second, removing Softmax for energy efficiency weakens token-wise competition, reducing the model’s ability to select salient tokens. To address these challenges, we propose Spiking Language Model (SpikingLM) to bridge the efficiency of SNNs and the capability of modern language models through two key innovations. First, we propose Distribution-aware Scaling method, which rescales linear outputs into an activation-friendly range to alleviate dead neurons and stabilize gradient propagation. Notably, its scaling parameters can be fused into the preceding linear layers, incurring no additional inference overhead. Second, we introduce Spike2Max to restore winner-takes-all mechanism via base-2 exponentiation and max-subtraction. Compared with Softmax, Spike2Max reduces energy consumption by over 95\% using hardware-efficient bit-shift operations. Experiments show that SpikingLM reduces energy consumption by 57.9\% and achieves state-of-the-art performance on GLUE, laying a promising foundation for energy-efficient language modeling. Code is available at~\url{https://github.com/hamings1/SpikingLM.git}.

Modern language models are powerful, but running them can consume a large amount of energy, which makes them harder to deploy on small devices and raises sustainability concerns. Spiking neural networks offer a promising alternative: they are inspired by the brain and compute with short on/off signals, so inactive parts can consume less energy. Yet applying this idea to language models is difficult. In deep spiking models, many units may stop firing and stop learning, while efficient spiking attention often fails to focus on the most important words. We propose SpikingLM, a fully spiking language model that keeps the energy benefits of spiking computation while preserving language understanding ability. Its first component, Distribution-aware Scaling, keeps signals in a range where spiking units remain active during training and can be folded into existing weights at inference. Its second component, Spike2Max, helps the model choose important words using cheap bit-shift operations instead of the expensive calculations normally used in attention. On standard language-understanding benchmarks, SpikingLM outperforms prior spiking language models and reduces estimated energy use by 57.9% compared with a recent spiking baseline. This work points toward practical, lower-energy language models for edge and low-power devices.