TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers

In recent years, Spiking Neural Networks (SNNs) have achieved remarkable progress, with Spiking Transformers emerging as a promising architecture for energy-efficient sequence modeling. However, existing Spiking Transformers still lack a principled mechanism for effective temporal fusion, limiting their ability to fully exploit spatiotemporal dependencies. Inspired by feedforward–feedback modulation in the human visual pathway, we propose TEFormer, the first Spiking Transformer framework that achieves bidirectional temporal fusion by decoupling temporal modeling across its core components. Specifically, TEFormer employs a lightweight and hyperparameter-free forward temporal fusion mechanism in the attention module, enabling fully parallel computation, while incorporating a backward gated recurrent structure in the MLP to aggregate temporal information in reverse order and reinforce temporal consistency. Extensive experiments across a wide range of benchmarks demonstrate that TEFormer consistently and significantly outperforms strong SNN and Spiking Transformer baselines under diverse datasets. Moreover, through the first systematic evaluation of Spiking Transformers under different neural encoding schemes, we show that the performance gains of TEFormer remain stable across encoding choices, indicating that the improved temporal modeling directly translates into reliable accuracy improvements across varied spiking representations. These results collectively establish TEFormer as an effective and general framework for temporal modeling in Spiking Transformers. Code is available here.

Spiking Neural Networks are a type of artificial intelligence model inspired by the way biological neurons communicate through short electrical signals. Compared with conventional neural networks, they have the potential to process information more efficiently and with lower energy consumption, which makes them promising for future intelligent systems and neuromorphic hardware. However, current spiking Transformer models still have difficulty making full use of information that changes over time. This limits their ability to understand visual data where timing is important, such as event streams from dynamic vision sensors or long temporal sequences. In this work, we propose TEFormer, a new spiking Transformer designed to better combine information across time. Inspired by the forward and feedback processing pathways in the human visual system, TEFormer allows the model to use temporal information in both directions while keeping the computation efficient. We evaluate TEFormer on a broad range of image, event-based, and temporally challenging benchmarks. The results show that TEFormer consistently improves over previous spiking neural network and spiking Transformer methods. This suggests that better temporal modeling can make spiking Transformers more accurate, robust, and practical for energy-efficient visual intelligence.