AdaS: Adaptive Gradient Descent for Spiking Transformers

Transformer-based Spiking Neural Networks (SNNs) combine Transformer performance with SNN energy efficiency through an event-driven self-attention mechanism. However, Spiking Transformers still lag behind their Artificial Neural Network (ANN) counterparts. Most existing studies address this issue through new architectural designs, yet few have explored optimization algorithms tailored to Spiking Transformers. We substantiate the excessive noise problem in Spiking Transformer training by quantitatively defining parameter-update noise and, based on this definition, providing theoretical analysis and experimental validation. To address this problem, we propose AdaS, an adaptive gradient descent method for Spiking Transformers. AdaS reduces excessive noise by adaptively incorporating a gradient update component into adaptive optimization. Instead of simply removing noise, AdaS maintains it at an appropriate level to preserve its generalization benefits, thereby improving the performance of Spiking Transformers. We conduct extensive experiments on various Spiking Transformer architectures and datasets from both computer vision and natural language processing. The results demonstrate that the proposed AdaS consistently enhances performance across different Spiking Transformers, validating its effectiveness and generalizability. This work is among the first systematic studies of optimizer design specifically for Spiking Transformers, offering a practical tool to narrow the accuracy gap with ANNs while preserving the energy advantages of spike-based computation. Code is available at https://github.com/CayleyZ/AdaS.

Spiking neural networks are a promising way to build more energy-efficient AI because they communicate using short, sparse signals, similar to spikes in the brain. Recent Spiking Transformers try to combine this efficiency with the strong performance of Transformer models, but they still often fall behind standard neural networks. This paper shows that one reason is the way these models are trained. Because spikes are not smoothly differentiable, training must rely on an approximation, which can add too much noise to the model’s parameter updates. We introduce AdaS, an optimizer that keeps this training noise at a more useful level rather than simply removing it. AdaS does this by adaptively balancing the update from a standard adaptive optimizer with a more stable momentum-based gradient update. Experiments on language understanding, event-based tracking, semantic segmentation, and neuromorphic vision show that AdaS consistently improves several Spiking Transformer models. These results suggest that better optimizer design can help close the accuracy gap between energy-efficient spiking models and conventional neural networks.