Plug-and-Play Spiking Operators: Breaking the Nonlinearity Bottleneck in Spiking Transformers

ANN-to-SNN conversion offers a practical, training-free route to spiking large language models. However, current pipelines primarily focus on spike-driven realizations for Transformer linear-algebra operations, while providing limited support for key nonlinear operators. This gap limits compatibility with neuromorphic-style execution constraints, where such nonlinearities typically require division, exponentiation, or norm computations that are not naturally supported by standard leaky integrate-and-fire dynamics. To solve this problem, we propose a plug-and-play framework that implements spike-friendly approximations for Transformer nonlinearities and integrates into existing ANN-to-SNN pipelines. Our method decomposes these nonlinear computations into three recurring primitives---division, exponentiation, and $\ell_2$ norms---and realizes them via population computation using LIF neuron groups, combined with lightweight bit-shift scaling to avoid floating-point arithmetic. By composing these primitives as modular operator blocks, our framework supports common Transformer nonlinearities (e.g., Softmax, SiLU, and normalization) without any fine-tuning. Experiments on a range of LLMs Transformers show that selectively replacing the targeted nonlinear operators incurs less than a $1\%$ accuracy drop across all evaluated tasks.

Large language models are powerful but require substantial computing resources and energy, making them difficult to deploy on energy-efficient neuromorphic hardware inspired by the human brain. Spiking neural networks (SNNs) provide a promising alternative because they communicate using sparse electrical spikes, potentially enabling much lower power consumption. A practical way to build spiking language models is to convert existing neural networks into SNNs without retraining them. However, current conversion methods mainly support simple mathematical operations and struggle to implement important nonlinear functions used in Transformer models, such as Softmax and normalization. In this work, we propose a plug-and-play framework that enables these nonlinear functions to operate efficiently in spiking neural networks. Our approach breaks complex computations into a small set of simple operations that can be naturally implemented using groups of spiking neurons and lightweight bit-shift operations, avoiding expensive floating-point computation. The framework can be integrated into existing ANN-to-SNN conversion pipelines without additional fine-tuning. Experiments across multiple Transformer-based language models show that our method preserves model performance very well, with less than a 1% accuracy drop across all evaluated tasks. These results demonstrate a practical step toward efficient neuromorphic large language models.